none
Cannot reliably get all of a process's output RRS feed

  • Question

  • This feels like I'm missing something obvious, but from reading some other posts in this forum, maybe not.  Why does .net make dealing with the output from console apps like pulling teeth?

    What I'm trying to do is get all of the output from ffmpeg.exe, both everything written to stdout and stderr, upon its exit.  This sounds easy but I'm finding it impossible to do either synchronously OR asynchronously.

    Synchronous problem:
    I .Start() the process, then use .StandardOutput.ReadToEnd(), or try and look at .StandardOutput.EndOfStream - my process just hangs.  ffmpeg might not write anything to standardoutput (i think it writes everything to standarderror) but I'd like to gather all output from here anyway, just it case it wrote anything.

    Asynchronous problem:
    This kinda works, but the problem is that I need all my .OutputDataReceived and .ErrorDataReceived events to fire BEFORE I run the .Exited event's code; that code needs to read the string buffers I've built up with the stdout and stderr data in.  However, because it's asynchronous, this order of events cannot be guaranteed, and indeed, the .OutputDataReceived and .ErrorDataReceived events are repeatedly firing AFTER the .Exited event does.  .Exited therefore does not have any way of getting all of the output/error data, and there doesn't seem to be any method like Process.WaitForFlushedData() or something that would let the .Exited code wait for all .OutputDataReceived and .ErrorDataReceived events to fire before continuing.

    Could anyone help me solve this problem either synchronously or asynchronously?  I'd prefer async because it seems easier, but either would do.  I'm at my wits' end at this point.  Tongue Tied
    Monday, September 10, 2007 10:40 AM

Answers

  • Hi Jez9999,

    I wrote an code example to show how you can read output from another command line process:

    Code Snippet

    static void Main ( string [] args )

    {

        String exeFileName = "output.exe";

        ProcessStartInfo info = new ProcessStartInfo(exeFileName);

     

        // Redirect the standard output of the process.

        info.RedirectStandardOutput = true;

        info.RedirectStandardError = true;

       

        // Set UseShellExecute to false for redirection

        info.UseShellExecute = false;

     

        Process proc = new Process();

        proc.StartInfo = info;

     

        // Set our event handler to asynchronously read the sort output.

        proc.OutputDataReceived += new DataReceivedEventHandler(proc_OutputDataReceived);

        proc.ErrorDataReceived += new DataReceivedEventHandler(proc_ErrorDataReceived);

       

        proc.Start();

       

        // Start the asynchronous read of the sort output stream. Note this line!

        proc.BeginOutputReadLine();

     

        proc.WaitForExit();

    }

     

    static void proc_ErrorDataReceived(object sender, DataReceivedEventArgs e)

    {

        Console.WriteLine("Error: {0}", e.Data);

    }

     

    static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)

    {

        Console.WriteLine("Output data: {0}", e.Data);

    }

     

     

    Hope this helps!

    Thanks!

    Wednesday, September 12, 2007 5:11 AM

All replies

  • I think that i has this problem some months ago, see this:

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1711728&SiteID=1

    If you don't need interactuate with the console app, see this:

    Obtener la salida de un comando de consola

    Is in spanish, but you can see the code and get the idea Wink

    Regards.
    Monday, September 10, 2007 11:10 AM
  • Sorry, but I don't see how that addresses my problem.  It reads the using the process's .StandardOutput property, which just hangs my application when I try to do it (because that waits until something is written before returning, which is really annoying).  I want to read it just in case anything is written there, but if not, I just want it to return with nothing, not hang.
    Monday, September 10, 2007 11:17 AM
  • Hi Jez9999,

    To troubleshoot this issue, we really need the source code to reproduce the problem, so that we can investigate the issue in house. It is not necessary that you send out the complete source of your project. We just need a simplest sample to reproduce the problem. You can remove any confidential information or business logic from it.

    Thanks!

    Tuesday, September 11, 2007 7:02 AM
  • Reading from StandardOutput is really tricky. The problem is that you also need to read from the Error stream at the same time. If you don't constantly read from both, one of the buffers could get full and the process will appear to hang as it waits for you to read from it. This is why you can't call ReadToEnd on the streams.

     

    I actually solved this once for the general case and I kick myself every time I think about it because I didn't think to keep a copy of that code for myself. I ended up having two reader threads processing the output and error streams which would pass the results back to the main thread.

     

    If you still need help, I can try tackling this tomorrow.

     

    Jonathan

    Wednesday, September 12, 2007 5:00 AM
  • Hi Jez9999,

    I wrote an code example to show how you can read output from another command line process:

    Code Snippet

    static void Main ( string [] args )

    {

        String exeFileName = "output.exe";

        ProcessStartInfo info = new ProcessStartInfo(exeFileName);

     

        // Redirect the standard output of the process.

        info.RedirectStandardOutput = true;

        info.RedirectStandardError = true;

       

        // Set UseShellExecute to false for redirection

        info.UseShellExecute = false;

     

        Process proc = new Process();

        proc.StartInfo = info;

     

        // Set our event handler to asynchronously read the sort output.

        proc.OutputDataReceived += new DataReceivedEventHandler(proc_OutputDataReceived);

        proc.ErrorDataReceived += new DataReceivedEventHandler(proc_ErrorDataReceived);

       

        proc.Start();

       

        // Start the asynchronous read of the sort output stream. Note this line!

        proc.BeginOutputReadLine();

     

        proc.WaitForExit();

    }

     

    static void proc_ErrorDataReceived(object sender, DataReceivedEventArgs e)

    {

        Console.WriteLine("Error: {0}", e.Data);

    }

     

    static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)

    {

        Console.WriteLine("Output data: {0}", e.Data);

    }

     

     

    Hope this helps!

    Thanks!

    Wednesday, September 12, 2007 5:11 AM
  • Well that's certainly easier than the way I used to do it.

     

    Thanks,

    Jonathan

    Wednesday, September 12, 2007 8:12 AM
  • I actually solved this once for the general case and I kick myself every time I think about it because I didn't think to keep a copy of that code for myself. I ended up having two reader threads processing the output and error streams which would pass the results back to the main thread.

    The trouble with this is that, according to the MSDN documentation (and my own tests, which backed this up), "Any instance members [of the Process class] are not guaranteed to be thread safe."  Therefore, your multi-threading solution should, in theory, not work.  Smile  It's certainly not advised by MS.  Thanks for the suggestion though.

     

    Hi Jez9999,

    I wrote an code example to show how you can read output from another command line process:

    Thanks for that.  This actually seems to work (yay!)... but only in a synchronous way.  You can use this method to reliably get all stdout and stderr output of a process, but the code has to wait until the process has exited before returning.  Fine if you want to deal with just one process at a time, so this is a good start.  Unfortunately, my ultimate ambition was to be able to deal with several at once, which would mean creating a new thread that called this code for each process.  The trouble is that, as mentioned above, the Process class instance members are not thread safe, so this can't be done.

     

    So thanks for everyone's help, and we now know how to tackle this problem in a synchronous way.  However, if there's a way to do it asynchronously (ie. fire off a few threads that deal with some different processes, then return immediately in order to keep the UI responsive), I'd be very keen to hear about how to do that.  In the meantime, though, I'll just have to live with doing this in a synchronous way.

    Wednesday, September 12, 2007 8:48 AM
  • Hi Jez9999,

    Actually, the two event handler functions were invoked by CLR’s background threads asynchronously. And, if you want to listen to several processes, why not use multiple Process class objects? You can use a event handler function to handle events from mulitiple event sources.

    Thanks!

    Wednesday, September 12, 2007 8:58 AM
  • Feng, this is a good solution if you only need the output "after" you have called proc.WaitForExit().

     

    I need that out put "during" my ftp process.

     

    See below, after I send the OPEN command, I need to then check to see what the

    return code from the server was

     

    If it was a 220, then I submit my username and password and then I need to see if

    I got a 200 code back.

     

    What if am finding is that the DataReceivedEventHandler(FTPProcess_OutputDataReceived)

    event does not fire until after the WaitForExit, which is way to late for my needs.

     

    Any idea on how to get real time data from the StandardOutput?

    Thanks.

    Terrence

     

    //...setup process code...

    private bool setup()

    {

    ...

    FTPProcess.OutputDataReceived += new DataReceivedEventHandler(FTPProcess_OutputDataReceived);

    FTPProcess.ErrorDataReceived += new DataReceivedEventHandler(FTPProcess_ErrorDataReceived);

    ...

    }

     

    private bool LoginToServer()

    {

    //Sending commands to FTP-Process

    FTPStreamWriter.WriteLine("OPEN " + this._FtpServerAddress);

    bool success = false;

     

    //if we have successfully connected wtih the ftp server

    if (FTPOutput.Contains("220"))

    {

    //try to log in

    FTPStreamWriter.WriteLine("USER " + this._UserName + " " + this._Password);

    //if we have a successful login

    if (FTPOutput.Contains("200"))

    success = true;

    }

    return success;

    }

    private void FTPProcess_OutputDataReceived(object sendingProcess, DataReceivedEventArgs output)

    {

    if (!string.IsNullOrEmpty(output.Data))

    FTPOutput.Add(output.Data);

    }

    private void FTPProcess_ErrorDataReceived(object sendingProcess, DataReceivedEventArgs error)

    {

    if (!string.IsNullOrEmpty(error.Data))

    FTPError.Add(error.Data);

    }

    Friday, March 28, 2008 8:32 PM
  • Hi,

    I'm clearly a bit late to this party, but I have the same issue.

    Refresh of what I understand to be the problem (After reading "Asynchronous problem" from the initial post by Jez9999):
    Exited event fires before the last OuputDataReceived event.  As a result you don't know when the last OutputDataReceived event has arrived (I assume this is also true for ErrorDataReceived).

    I believe the following example solves this issue.

    <START CODE SNIPIT>

    private
    static bool Processing { get; set; }

    static void Main(string[] args)
    {
        Processing =
    true;
        string command = "output.exe";
        string arguments = " whatever";

        ProcessStartInfo info = new ProcessStartInfo(command, arguments);

        
    // Redirect the standard output of the process. 
        
    info.RedirectStandardOutput = true;
        info.RedirectStandardError = true;
        
        
    // Set UseShellExecute to false for redirection
        info.UseShellExecute = false;
        
        
    Process proc = new Process();
        proc.StartInfo = info;
        proc.EnableRaisingEvents =
    true;

        
    // Set our event handler to asynchronously read the sort output.
        
    proc.OutputDataReceived += new DataReceivedEventHandler(proc_OutputDataReceived);
        proc.ErrorDataReceived +=
    new DataReceivedEventHandler(proc_ErrorDataReceived);
        proc.Exited +=
    new EventHandler(proc_Exited);

        proc.Start();

        
    // Start the asynchronous read of the sort output stream. Note this line!
        
    proc.BeginOutputReadLine();
        proc.BeginErrorReadLine();

        
    while (Processing == true)
        {
            
    Thread.Sleep(0);
        }

        
    Console.WriteLine("Exited (Main)");

    }

    static void proc_Exited(object sender, EventArgs e)
    {
        ((
    Process)sender).WaitForExit();
        
    Console.WriteLine("Exited (Event)");
        Processing =
    false;
    }

    static void proc_ErrorDataReceived(object sender, DataReceivedEventArgs e)
    {
        
    Console.WriteLine("Error: {0}", e.Data);
    }

    static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)
    {
        
    Console.WriteLine("Output data: {0}", e.Data);
    }

    <END CODE SNIPIT>

    Monday, September 15, 2008 12:50 PM
  • There is a better way to handle this.

    Simply check if the string in your DataReceivedEventHandler is null or empty. If it is, the stream has completed and you know your process has completely finished IO activity even after the exit event is raised. e.g.
            private void OnOutputDataReceived(object sender, DataReceivedEventArgs e) 
            { 
                if(!String.IsNullOrEmpty(e.Data)) 
                    log.Info(e.Data); 
                else 
                    log.Info("Output Stream finished"); 
            } 

    So if you capture that the streams are closed and trap the exit event you can accurately detect exit + IO complete.
    Friday, October 3, 2008 10:16 AM
  • <Yes I know this is an old thread>

    Checking if the data is null or empty isnt the best way to go. It is possible that the program you are running is outputting empty strings.

    As far as I can tell there is no sure fire way to know when the IO events have completed. Even "elggarc" answer above doesnt guarantee this.

    The only way I have found is to sleep your main thread for a second before returning. Example taking the answer from above:

    static void Main ( string [] args )
    
    {
    
     String exeFileName = "output.exe";
    
     ProcessStartInfo info = new ProcessStartInfo(exeFileName);
    
     
    
     // Redirect the standard output of the process. 
    
     info.RedirectStandardOutput = true;
    
     info.RedirectStandardError = true;
    
     
    
     // Set UseShellExecute to false for redirection
    
     info.UseShellExecute = false;
    
     
    
     Process proc = new Process();
    
     proc.StartInfo = info;
    
     
    
     // Set our event handler to asynchronously read the sort output.
    
     proc.OutputDataReceived += new DataReceivedEventHandler(proc_OutputDataReceived);
    
     proc.ErrorDataReceived += new DataReceivedEventHandler(proc_ErrorDataReceived);
    
     
    
     proc.Start();
    
     
    
     // Start the asynchronous read of the sort output stream. Note this line!
    
     proc.BeginOutputReadLine();
    
    
     proc.WaitForExit();
     // Give time to let all the IO events fire 
     System.Threading.Thread.Sleep(1000);
    }
    
     
    
    static void proc_ErrorDataReceived(object sender, DataReceivedEventArgs e)
    
    {
    
     Console.WriteLine("Error: {0}", e.Data);
    
    }
    
     
    
    static void proc_OutputDataReceived(object sender, DataReceivedEventArgs e)
    
    {
    
     Console.WriteLine("Output data: {0}", e.Data);
    
    }
    
    

     

    • Edited by pbyrne Thursday, March 3, 2011 10:34 PM fixing code example
    Thursday, March 3, 2011 10:32 PM
  • The documentation says about BeginOutputReadLine: 

    Begins asynchronous read operations on the redirected StandardOutput stream of the application.

    However, it seems you are trying to redirect both stderr and stdout. Will calling only BeginOutputReadLine() also read message from stderr?

    Wednesday, May 11, 2011 10:08 PM
  • This is the best one, after a big digging.

    public static void Test()
            {
                System.Diagnostics.Process winscp = new System.Diagnostics.Process();
                winscp.StartInfo.FileName = @"C:\Windows\System32\cmd.exe";
                winscp.StartInfo.UseShellExecute = false;
                winscp.StartInfo.RedirectStandardInput = true;
                winscp.StartInfo.RedirectStandardOutput = true;
                winscp.StartInfo.CreateNoWindow = false;
                winscp.StartInfo.CreateNoWindow = true;
                winscp.Start();
                winscp.EnableRaisingEvents = true;
                string result = RunCommand(winscp, "Dir");
                result = RunCommand(winscp, "Dir");
                result = RunCommand(winscp, "Dir");
                result = RunCommand(winscp, "Dir");
                result = RunCommand(winscp, "Dir");
            }

            public static string RunCommand(System.Diagnostics.Process winscp, string command)
            {
                int timout = 15;
                winscp.StandardInput.WriteLine(command);
                Thread threadToKill = null;
                Func<System.IO.StreamReader, StringBuilder, string> wrappedAction = (reader, sb) =>
                {
                    threadToKill = Thread.CurrentThread;
                    int cnt = reader.Peek();
                    while (cnt > 0)
                    {
                        char[] buffer = new char[cnt];
                        int rcnt = reader.Read(buffer, 0, cnt);
                        Console.WriteLine(cnt.ToString() + " : " + rcnt.ToString());
                        lock (sb)
                        {
                            sb.Append(buffer);
                        }
                        if (rcnt != cnt)
                            break;
                    }
                    return sb.ToString();
                };
                StringBuilder sbresult = new StringBuilder();
                IAsyncResult result = wrappedAction.BeginInvoke(winscp.StandardOutput, sbresult, null, null);
                string results = "";
                if (result.AsyncWaitHandle.WaitOne(new TimeSpan(0, 0, timout)))
                {

                    results = wrappedAction.EndInvoke(result);
                }
                else
                {
                    lock (sbresult)
                    {
                        results = sbresult.ToString();
                    }
                    threadToKill.Abort();
                }
                winscp.StandardOutput.DiscardBufferedData();
                return results;
            }

    Monday, November 23, 2015 2:40 PM
  • I was using the async pattern and was getting a truncated output every one and then.

    The fix was to increase the buffer size. Lost data with 1k, no losses with 10k buffer. 

        class Program
        {
            static async Task Main(string[] args)
            {
                var process = new Process();
                process.StartInfo = new ProcessStartInfo("cmd.exe")
                {
                    Arguments = "/c dir %windir%",
                    CreateNoWindow = true,
                    UseShellExecute = false,
                    RedirectStandardOutput = true,
                    RedirectStandardError = true,
                    RedirectStandardInput = true,
                };
                process.Start();
    
                var task1 = WriteStreamToConsole(process.StandardOutput);
                
                // method1 to wait for process end
                while (process.WaitForExit(10))
                {
    
                }
    
                // method2 to wait for process end
                while (!process.HasExited)
                {
                    await Task.Delay(10); // let other threads process stuff
                }
    
                // method3 to wait for output end
                await task1;
            }
    
            public async static Task WriteStreamToConsole(StreamReader reader)
            {
                char[] buffer = new char[10240];
                int cch;
    
                while ((cch = await reader.ReadAsync(buffer, 0, buffer.Length)) > 0)
                {
                    // I usually do other stuff here. Writing to console for demostration purposes
                    Console.Write(new string(buffer, 0, cch));
                    await Console.Out.FlushAsync();
                }
            }
        }



    Gerardo

    Friday, December 6, 2019 3:54 PM