Allocating/Attaching a console from a WinForms app - it just doesn't work

Question

I've done this many times in years past from Win32 GUI applications, but this is C# Winforms, .NET 3.5 SP2 on Windows 2008 R2 X64.  My app is targeting .NET 3.5, but I'm running on .NET 4.0 (current SP/patches) using VS2010 SP1 Ultimate.

I've read this article, and this one and lots of other ones, but none of them quite work.  In an attempt to track down what's failing, I've completely re-implemented the __ConsoleStream class that the framework uses so that I can step through the code.  What I'm seeing is:

• AttachConsole(-1) returns true.
• GetStdHandle(-11) returns the value 7.
• Calling WriteFile with that handle value sets last error to 6 - ERROR_INVALID_HANDLE

Some code:

    [System.Runtime.InteropServices.DllImport("kernel32.dll")]
    private static extern bool AllocConsole();

    [System.Runtime.InteropServices.DllImport("kernel32.dll")]
    private static extern bool AttachConsole(int pid);

    [System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)]
    private static extern IntPtr GetStdHandle(int h);

    private static bool RunAsConsole()
    {
      var args = Environment.GetCommandLineArgs();
      foreach (var arg in args)
      {
        if (arg.ToLowerInvariant() == "/console")
        {
          if (!AttachConsole(-1))
            AllocConsole();

          var ih = GetStdHandle(-11);
          var h = new Microsoft.Win32.SafeHandles.SafeFileHandle(ih, false);
          using (var os = new ConsoleStream(h, System.IO.FileAccess.Write))
          using (var wrtr = new System.IO.StreamWriter(os))
          {
            wrtr.WriteLine("Hello, console");
            wrtr.Flush();
          }

          return true;
        }
      }

      return false;
    }

This results in an error on wrtr.Flush, since that's the first time the code actually tries to write to the stream.

If I just use Console.WriteLine, output is silently eaten.

If I call Console.OpenStandardOutput, I always get back an instance of System.IO.NullStream (which explains why the output is eaten).  Looking at OpenStandardOutput in reflector, I believe that it's returning Stream.Null because it's getting the same error I'm getting - attempting to call WriteFile is returning FALSE with last error set to 6.

Any ideas?  This should "just work" without all this failing around, but something's evading my attempts!

---

-cd Mark the best replies as answers!

Answer 1

I see you don't verify that AllocConsole() is succeeding.  I recommend that you do that.  If it is failing, collect the error number.

---

Jose R. MCP

Answer 2

Agree with webJose.

You could add a try catch block and get more information from the caught exception.

Answer 3

I see you don't verify that AllocConsole() is succeeding.  I recommend that you do that.  If it is failing, collect the error number.

---

Jose R. MCP

It's not failing. In fact, it's not even being called, because AttachConsole is succeeding.

---

-cd Mark the best replies as answers!

Answer 4

Agree with webJose.

You could add a try catch block and get more information from the caught exception.

Since it's my own code that's throwing the exception when one occurs, I already know exactly what the exception says.  As this is just experimental code I didn't bother to clutter it with exception handling.

---

-cd Mark the best replies as answers!

Answer 5

Worked OK for me:

Imports System.Runtime.InteropServices
Imports Microsoft.Win32.SafeHandles
Imports System.IO
Public Class Form1
  Declare Function AllocConsole Lib "kernel32" () As Boolean
  Declare Function AttachConsole Lib "kernel32" (ProcessId As Integer) As Boolean
  Declare Function GetStdHandle Lib "kernel32" (StdHandle As Integer) As IntPtr
  Function RunAsConsole() As Boolean
    If Not AttachConsole(-1) Then AllocConsole()
    Dim pConsole As IntPtr = GetStdHandle(-11)
    Dim hConsole As New SafeFileHandle(pConsole, True)
    Using Wrtr As New StreamWriter(New FileStream(hConsole, FileAccess.Write))
      Wrtr.WriteLine("Hello console")
    End Using
    Return True
  End Function
  Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    RunAsConsole()
  End Sub
End Class

Answer 6

I also tried using FileStream instead of my re-engineered ConsoleStream - it too failed with ERROR_INVALID_HANDLE.  I implemented console stream specifically so I could debug into the stream code to see exactly where the error was occuring.

It seems that somehow AttachConsole is returning true but actually failing, or that GetStdHandle is not returning the correct value.  Neither of those makes any sense though.

@John - note that in your VB code, you've reversed the condition in the if statement - we only want to call AllocConsole if AttachConsole fails, not if it succeeds.

---

-cd Mark the best replies as answers!

Answer 7

"@John - note that in your VB code, you've reversed the condition in the if statement - we only want to call AllocConsole if AttachConsole fails, not if it succeeds."

The C# punctuation marks are easy to miss.  I changed the code.  It still works OK.  Starts a new window and writes the message.

And the C# code (converted in SharpDevelop 4.1) works the same:

using System;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;
using System.IO;
namespace WindowsFormsApplication1
{
  public partial class Form1 : Form
  {
    public Form1()
    {
      InitializeComponent();
    }
    [DllImport("kernel32", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern bool AllocConsole();
    [DllImport("kernel32", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern bool AttachConsole(int ProcessId);
    [DllImport("kernel32", CharSet = CharSet.Ansi, SetLastError = true, ExactSpelling = true)]
    public static extern IntPtr GetStdHandle(int StdHandle);
    public bool RunAsConsole()
    {
      if (!AttachConsole(-1)) AllocConsole();
      IntPtr pConsole = GetStdHandle(-11);
      SafeFileHandle hConsole = new SafeFileHandle(pConsole, true);
      using (StreamWriter Wrtr = new StreamWriter(new FileStream(hConsole, FileAccess.Write)))
      {
        Wrtr.WriteLine("Hello console");
      }
      return true;
    }
    private void button1_Click(object sender, EventArgs e)
    {
      RunAsConsole();
    }
  }
}

Answer 8

@John -

How are you running the program?

The scenario I'm testing is launching the Winforms EXE from a console program, expecting the EXE to pick up the same console - I specifically don't want a new console window to be allocated. 

I'm guessing that you're running from within VS, in which case AttachConsole(-1) will return false and you'll call AllocConsole which creates a new console window.

Indeed, if I change my code to not call AttachConsole and simply call AllocConsole, I do get a new console window as expected and it does work.  Time to dig deeper into AttachConsole.

---

-cd Mark the best replies as answers!

Answer 9

There's some key text in the description for GetStdHandle:

"The standard handles of a process may be redirected by a call to SetStdHandle, in which case GetStdHandle returns the redirected handle. If the standard handles have been redirected, you can specify the CONIN$ value in a call to the CreateFile function to get a handle to a console's input buffer. Similarly, you can specify the CONOUT$ value to get a handle to a console's active screen buffer."

Taking that direction, the following code works for me:

          int r = AttachConsole(-1);
          if (r == 0)
            AllocConsole();

          var ih = CreateFile("CONOUT$", 1 << 30, 0, IntPtr.Zero, 4, 128, IntPtr.Zero);
          var h = new Microsoft.Win32.SafeHandles.SafeFileHandle(ih, true);
          if (h.IsInvalid)
          {
            var error = System.Runtime.InteropServices.Marshal.GetLastWin32Error();
            // React to the error...
          }
          else
          {
            using (var os = new System.IO.FileStream(h, System.IO.FileAccess.Write))
            using (var wrtr = new System.IO.StreamWriter(os))
            {
              wrtr.WriteLine("Hello, console");
              wrtr.Flush();
            }
          }

What's really interesting is that CreateFile is returning the same handle (7) as was returned by GetStdHandle, but getting the handle from CreateFile works.  My only theory is that for some reason, the handle has been closed when I use GetStdHandle, but calling CreateFile re-opens it.

Unfortunately, it's not possible to pass "CONOUT$" to the FileStream constructor - internally it tries to convert it to a full file path which, of course, fails.

I've still got no idea what's special about this program that's making this so difficult, but this approach does appear to work at least.

Sadly, while this fixes one problem (invalid handle), it causes another - it's not possible to re-direct the console output to a file anymore :(

---

-cd Mark the best replies as answers!

Answer 10

Hi Carl,

I modified your code to this:

        private static bool RunAsConsole()
        {
            var args = Environment.GetCommandLineArgs();
            foreach (var arg in args)
            {
                if (arg.ToLowerInvariant() == "/console")
                {
                    //if (!AttachConsole(-1))
                        //AllocConsole();
                        MessageBox.Show(AttachConsole(-1).ToString());
                    Console.WriteLine("Winform app!");
                    //var ih = GetStdHandle(-11);
                    //var h = new Microsoft.Win32.SafeHandles.SafeFileHandle(ih, false);
                    //using (var os = new ConsoleStream(h, System.IO.FileAccess.Write))
                    //using (var wrtr = new System.IO.StreamWriter(os))
                    //{
                    //    wrtr.WriteLine("Hello, console");
                    //    wrtr.Flush();
                    //}

                    return true;
                }
            }

            return false;
        }

And the application is a Winform application, and this is the test result, it seems that it fits to your situation:

Best regards,

---

Mike Feng
MSDN Community Support | Feedback to us
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 11

@Mike -  Yes, it should be that easy.  Not surprisingly, that's where I started - all the rest of this exploration was an attempt to figure out why the most basic approach doesn't work in my app.  I'm still not to the bottom of it - in fact, I just gave up.

Now, a helpful bit of information that no one's replied with:  Are you Mike, or you John, running in the environment that I described in the original post?  .NET 3.5-targeted app built with VS 2010 running on .NET 4.0 latest SP and patches on 64 bit Windows Server 2008 R2?

---

-cd Mark the best replies as answers!

Answer 12

Hi Carl,

Yes, I test it on win server 2008 R2 64bit, VS 2010 sp1. .net framework 3.5.

Best regards,

---

Mike Feng
MSDN Community Support | Feedback to us
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 13

Hi Carl,

Yes, I test it on win server 2008 R2 64bit, VS 2010 sp1. .net framework 3.5.

Interesting.  So whatever's making mine not work is apparently something unique to the application, which is odd, because I'm running the code above as the very first statements in Main.  Wierd.

---

-cd Mark the best replies as answers!

Answer 14

Hi Carl,

What do you mean " I'm running the code above as the very first statements in Main" ?

Is this code Not all your test code?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Forms;

namespace WinFormApp_RunAsConsole
{
    static class Program
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        [STAThread]
        static void Main()
        {
            RunAsConsole();
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            Application.Run(new Form1());
            
        }

        [System.Runtime.InteropServices.DllImport("kernel32.dll")]
        private static extern bool AllocConsole();

        [System.Runtime.InteropServices.DllImport("kernel32.dll")]
        private static extern int AttachConsole(int pid);

        [System.Runtime.InteropServices.DllImport("kernel32.dll", SetLastError = true)]
        private static extern IntPtr GetStdHandle(int h);

        private static bool RunAsConsole()
        {
            var args = Environment.GetCommandLineArgs();
            foreach (var arg in args)
            {
                if (arg.ToLowerInvariant() == "/console")
                {
                    MessageBox.Show(AttachConsole(-1).ToString());
                    Console.WriteLine("Winform app!");
                    return true;
                }
            }

            return false;
        }
    }
}

What is the rest code?

Best regards,

---

Mike Feng
MSDN Community Support | Feedback to us
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 15

The rest of the code is a huge application, none of which should be running since the very first thing I'm doing in Main is calling RunAsConsole and exiting the application.  The only thing that could possible run before that is static constructors in the entry assembly.  I haven't yet looked to see if there's anything in a static constructor that could explain what I'm seeing, but I don't think there is - this is 100% code that I wrote (other than the framework itself).  In the end it may be that there's something subtely broken about my environment - I need to try doing this on another computer to find out.

---

-cd Mark the best replies as answers!

Answer 16

Hi Carl,

Does it work on anther computer?

Best regards,

---

Mike Feng
MSDN Community Support | Feedback to us
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 17

Hi Carl,

Does it work on anther computer?

Haven't had a chance to try yet.   I'll post back when I do.

---

-cd Mark the best replies as answers!

Answer 18

Hi Carl,

Any update?

Best regards,

---

Mike Feng
MSDN Community Support | Feedback to us
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 19

Hi Carl,

Any update?

I did a bit more experimenting which led to me believe that there's something about my environment that's responsible, but I still haven't tracked down the ultimate culprit.  I changed this thread to a discussion thread so you can stop worring about whether it's been answered or not.

---

-cd Mark the best replies as answers!