Multithreading a process

Question

I was wondering if this code could be better optimized for multithreading. What it does is open a process and loops through the data, there could be any range of files to open (so I would like to have say 2 or 3 processes at once):

 For i As Integer = 0 To Fileq.Filename.Count - 1
            If i = 6 Then
                Exit For
            End If
            Fileq.file(i) = New VideoFile
            Fileq.file(i).videotrack = New VideoTrack
            MediaInfoProcess(Fileq.file, i)
        Next i

Mediainfoprocess is where the process is created:

Public Sub MediaInfoProcess(ByVal vfile() As VideoFile, ByVal interval As Integer)

        process = New Process
        Dim MediaInfoPath As String = Application.StartupPath & "\Tools\MediaInfo\MediaInfo.exe"

        With process.StartInfo
            .FileName = MediaInfoPath
            .Arguments = """" & Fileq.Location(0) & "\" & Fileq.Filename(interval) & """"
            .UseShellExecute = False
            .CreateNoWindow = True
            .RedirectStandardOutput = True
        End With

        If System.IO.File.Exists(MediaInfoPath) = True Then
            process.Start()
            outputThread = New Threading.Thread(DirectCast(Sub() ToolOutputCapture(vfile(interval), interval), Threading.ThreadStart))
            outputThread.IsBackground = True
            outputThread.Start()
        Else
        End If

        StartNextFile = New Threading.AutoResetEvent(False)
        StartNextFile.WaitOne()

    End Sub

and I loop through the data:

'Private Sub ToolOutputCapture(ByRef Vfile As VideoFile, ByVal interval As Integer)

'Do work

    process.StandardOutput.ReadToEnd()
    process.Close()
    StartNextFile.Set()

    'End Sub

I have been looking at different ways to go about doing it (like thread pool) but the process variable always gets overwritten.

Thanks.

Answer 1

One thing to note is that multi-threading does not mean several processes, but several threads on the same process. I would look into MethodInvoker from MSDN to try :)

---

If a post helps you in any way or solves your particular issue, please remember to use the Propose As Answer option or Vote As Helpful
~ "The universe is an intelligence test." - Timothy Leary ~

Answer 2

You would need each thread to start its own Process.

You should also get a count of available processors on the computer... this gives you the maximum number of threads you can run (you can run as many as you want, but should not run more than the number of processor cores or you will be slowing down the program rather than speeding it up).

This also looks like it might be a good place to use System.Threading.Tasks.Parallel since it appears that the code can run concurrently against an unknown number of files.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 3

One thing to note is that multi-threading does not mean several processes, but several threads on the same process. I would look into MethodInvoker from MSDN to try :)

---

If a post helps you in any way or solves your particular issue, please remember to use the Propose As Answer option or Vote As Helpful
~ "The universe is an intelligence test." - Timothy Leary ~

I think he meant "process" as in System.Diagnostics.Process, not just the loose sense of the term.  But a good point nonetheless.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 4

I looked at methodInvoker, however I couldn't find out much about it from any site (Msdn didn't help). What I was trying to say was I wanted to run a process on each thread, therefore I would have several processes running at the same time.

Thanks reed kimble, I'll look into the System.Threading.Tasks.Parallel and look into the processors count.

Answer 5

Maybe I got something wrong, but IMO the point is that "process.StandardOutput.ReadToEnd()" is a blocking call best executed in it's own thread (as not the built-in async reading pattern with Process.BeginOutputReadLine is employed). Starting other processes doesn't play the main role here.

One error source I see is that "StartNextFile.Set()" could possibly be executed before "StartNextFile = New Threading.AutoResetEvent(False)".
The other issue is that waiting for a thread to finish makes the multi threading approach pointless.

---

Armin

Answer 6

Could anyone see what I'm doing wrong with this code:

So whats happening is if I try and process multi files say 6, non of them return a text value but the last one and it's of the first file that was passed.

Main loop

 For i As Integer = 0 To length
            MediaInfoProcess(Fileq.file, i, length)
 Next i

  Public Sub MediaInfoProcess(ByVal vfile() As VideoFile, ByVal interval As Integer, ByVal count As integer)

        process(interval) = New Process
        Dim MediaInfoPath As String = Application.StartupPath & "\Tools\MediaInfo\MediaInfo.exe"

        With process(interval).StartInfo
            .FileName = MediaInfoPath
            .Arguments = """" & Fileq.Location(0) & "\" & Fileq.Filename(interval) & """"
            .UseShellExecute = False
            .CreateNoWindow = True
            .RedirectStandardOutput = True
        End With

        ToolOutputCaptureq.interval = interval
        ToolOutputCaptureq.Vfile = vfile(interval)
        'ToolOutputCaptureq.process = process(interval)
        outputThread(interval) = New Threading.Thread(AddressOf ToolOutputCaptureq.ToolOutputCapture)
        outputThread(interval).IsBackground = True

        outputThread(interval).Start()

    End Sub

      Sub ToolOutputCapture()

process(interval).Start()
        Dim text As String
        Try
            'Dim myStreamReader As StreamReader = process.StandardOutput
            ' text = myStreamReader.ReadLine
            ' text = myStreamReader.ReadLine
            Dim sortStreamWriter As StreamReader = process(interval).StandardOutput
            text = sortStreamWriter.ReadLine
            text = sortStreamWriter.ReadLine
            RaiseEvent ThreadDone(text, interval) 'returns the text value that i'm looking for
        Catch ex As Exception

        End Try
        process(interval).Close()
    End Sub

note I also had the process.start in media function which causes the same issue.

Answer 7

In Sub ToolOutputCapture, where is the "interval" variable declared?

---

Armin

Answer 8

Oh sorry about that, the Sub ToolOutputCapture is in a class so that I can pass variables to it:

Public Class ToolOutputCapture

    Public Vfile As VideoFile
    Public interval As Integer
    Public process As Process
    Public Event ThreadDone(ByVal CompleteText As String, ByVal interval As Integer)

    Sub ToolOutputCapture()

process(interval).Start()
        Dim text As String
        Try
            'Dim myStreamReader As StreamReader = process.StandardOutput
            ' text = myStreamReader.ReadLine
            ' text = myStreamReader.ReadLine
            Dim sortStreamWriter As StreamReader = process(interval).StandardOutput
            text = sortStreamWriter.ReadLine
            text = sortStreamWriter.ReadLine
            RaiseEvent ThreadDone(text, interval) 'returns the text value that i'm looking for
        Catch ex As Exception

        End Try
        process(interval).Close()

    End Sub



End Class

which is this line:

        ToolOutputCaptureq.interval = interval
        ToolOutputCaptureq.Vfile = vfile(interval)
        'ToolOutputCaptureq.process = process(interval)

Answer 9

Is it true that interval never changes? Then you're always using index zero.

---

Armin

Answer 10

Well the main loop is:

  For i As Integer = 0 To length
            MediaInfoProcess(Fileq.file, i, length)
        Next i

so the i is the interval, which is used in mediainfoprocess.

  Public Sub MediaInfoProcess(ByVal vfile() As VideoFile, ByVal interval As Integer, ByVal count As integer)

I just checked to make sure, but the interval does change.

Edit: I think I know whats going on, I have to see if i'm right

Answer 11

The problem seem to lie with the interval variable. I assume it was getting updated to fast, when the interval should have been 3 it was 6.

So i made a couple changes:

  Private ToolOutputCaptureq() As ToolOutputCapture

This seemed to fix the issue, I made a couple more changes but I guess because I was using the same ToolOutputCaptureq it was messing with the interval.

Thanks for the help everyone, I'll be testing each method you guys suggested to see which is better.

Answer 12

I was trying to make you an example that closely matched what you have, but ran into an issue using an event to handle notifications... Tasks.Parallel did not like that at all... I'm still not completely sure why - something I'll have to look into.

Anyway, here is a real simple example which should match pretty closely with what you are doing:

Public Class Form1
    Friend WithEvents RunExampleButton As New Button With {.Text = "Run Example", .Dock = DockStyle.Top}
    Friend WithEvents OutputTextbox As New TextBox With {.ReadOnly = True, .Dock = DockStyle.Fill, .Multiline = True, .ScrollBars = ScrollBars.Vertical}

    Private _JobList As New List(Of String)
    Private _Results As New System.Collections.Concurrent.ConcurrentQueue(Of String)

    Private Sub Form1_Load(sender As System.Object, e As System.EventArgs) Handles MyBase.Load
        Controls.Add(OutputTextbox)
        Controls.Add(RunExampleButton)
        For count As Integer = 1 To 10
            _JobList.Add(String.Format("""C:\SomeFolder\media {0}.med""", count))
        Next
    End Sub

    Private Sub RunExampleButton_Click(sender As Object, e As System.EventArgs) Handles RunExampleButton.Click
        Dim timer As New Stopwatch
        timer.Start()
        System.Threading.Tasks.Parallel.ForEach(Of String)(_JobList, AddressOf ProcessMediaJob)
        timer.Stop()
        While _Results.Count > 0
            Dim resultLine As String
            If _Results.TryDequeue(resultLine) Then
                AppendOutput(resultLine)
            End If
        End While
        AppendOutput(String.Format("Executed all in {0:ss\.ff}", timer.Elapsed))
    End Sub

    Private Delegate Sub AppendOutputDelegate(appendText As String)
    Private Sub AppendOutput(appendText As String)
        If OutputTextbox.TextLength > 0 Then OutputTextbox.AppendText(ControlChars.NewLine)
        OutputTextbox.AppendText(appendText)
        If Not appendText.EndsWith(ControlChars.NewLine) Then OutputTextbox.AppendText(ControlChars.NewLine)
    End Sub

    Private Sub ProcessMediaJob(job As String)
        Dim mediaInfoPath As String = System.IO.Path.Combine(My.Application.Info.DirectoryPath, "Tools\MediaInfo\MediaInfo.exe")
        Dim startInfo As New ProcessStartInfo(mediaInfoPath, job)
        startInfo.UseShellExecute = False
        startInfo.RedirectStandardOutput = True
        startInfo.CreateNoWindow = True
        Dim mediaProcess As Process = Process.Start(startInfo)
        _Results.Enqueue(mediaProcess.StandardOutput.ReadLine)
        mediaProcess.Dispose()
    End Sub
End Class

And here is the psuedo-"MediaInfo" console application I used with the test:

Module Module1
    Sub Main()
        If My.Application.CommandLineArgs.Count > 0 Then
            Dim filePath As String = My.Application.CommandLineArgs(0)
            Dim randomizer As New Random
            Dim deltaTime As Integer
            If randomizer.NextDouble > 0.65 Then
                deltaTime = 5
            Else
                deltaTime = 3
            End If
            deltaTime = randomizer.Next(1, deltaTime + 1) * 1000
            System.Threading.Thread.Sleep(deltaTime)
            Console.WriteLine(String.Format("{0} processed in {1} seconds.", filePath, deltaTime / 1000))
        End If
    End Sub
End Module

Hopefully you can build upon this simple example to feed it your list of file paths and read back your results.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 13

Oh wow I'll definitely look into it, Thanks. From just taking a quick peek at it I can already see some code that I would like to use.

Answer 14

Oh sorry, ignore that Delegate sub -that was a left over from trying to use events.  This code doesn't use that delegate so it can be deleted.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 15

After applying your code, the Process.StandardOutput.ReadLine keeps coming out as either "nothing" or "". It works when i have 1 file.

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click

        Dim timer As New Stopwatch
        timer.Start()
        System.Threading.Tasks.Parallel.ForEach(Of String)(_JobList, AddressOf ToolOutputCapture)
        timer.Stop()

    End Sub

  Private Sub ParseFile()

        For count As Integer = 0 To Fileq.Filename.Count - 1
            _JobList.Add(String.Format("""" & Fileq.Location(0) & Fileq.Filename(count) & """", count))
        Next
end sub

 Sub ToolOutputCapture(ByVal job As String)

        Dim MediaInfoPath As String = Application.StartupPath & "\Tools\MediaInfo\MediaInfo.exe"
        Dim startInfo As New ProcessStartInfo(MediaInfoPath, job)
        startInfo.UseShellExecute = False
        startInfo.RedirectStandardOutput = True
        startInfo.CreateNoWindow = True
        Dim Process As Process = Process.Start(startInfo)

        _Results.Enqueue(Process.StandardOutput.ReadLine)
        _Results.Enqueue(Process.StandardOutput.ReadLine)
        _Results.Enqueue(Process.StandardOutput.ReadLine)

        Process.Dispose()

        'With process.StartInfo
        '    .FileName = MediaInfoPath
        '    .Arguments = """" & Fileq.Location(0) & "\" & Fileq.Filename(i) & """"
        '    .UseShellExecute = False
        '    .CreateNoWindow = True
        '    .RedirectStandardOutput = True
        'End With


    End Sub

    Public _JobList As New List(Of String)
    Private _Results As New System.Collections.Concurrent.ConcurrentQueue(Of String)

Answer 16

Well, now you have me suspecting that MediaInfo.exe cannot run multiple instances of itself... have you ever manually tried?  Open two command prompts, type out each command without hitting enter, then quickly hit enter on both windows.  Does it work?

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 17

Also, just FWIW, you should read all of the console lines into a single result string (or array of strings if you want to change the code to use an array) and only Enqueue one result per job.  You cannot be sure what order each result will get enqueued in... one thread could insert a result in-between two of the three Enqueue operations of another thread.  In this case I would just use a System.Text.StringBuilder to AppendLine each console.ReadLine result and then Enqueue the StringBuilder.ToString.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 18

I feel really dumb now, the way I was using _JobList.Add(), I forgot to use "\" in the middle!

For count As Integer = 0 To Fileq.Filename.Count - 1
            _JobList.Add(String.Format("""" & Fileq.Location(0) & "\" & Fileq.Filename(count) & """", count))
Next

Thank you very much for all the help.

Answer 19

LOL oh, that's ok, we all make silly mistakes.

To avoid that mistake in the future though, always use System.IO.Path.Combine() to construct a path.

If your issues are resolved, would you mind marking one or more posts as "answer" if they helped get you to a solution?  Thanks!

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"