locked
How to get Unicode output from cl.exe? RRS feed

  • Question

  • Hello,

    I have a code that runs "cl.exe" by CreateProcess. Then I parse the output from "cl.exe". All works fine with English characters. But if output contains non-English characters, I get "???????" instead of non-English characters.

    As I know, "cl.exe" can create two named pipes and write Unicode output to them, instead of using stderr and stdout.
    Also, I've read about an environment variable "VS_UNICODE_OUTPUT". As I've understand that variable causes "cl.exe" to create these two named pipes.

    The questions are:
    How can I run "cl.exe" in such mode (to get Unicode output)?
    What names should I use for pipes?
    What value should I set to "VS_UNICODE_OUTPUT", and what does this value mean?
    • Moved by Feng Chen Tuesday, August 5, 2008 1:30 AM VC compiler issue. (Moved from Visual Studio Extensibility to Visual C++ General)
    • Edited by Yuri Plyakhin Tuesday, August 5, 2008 4:05 AM Added additional info and questions
    Monday, August 4, 2008 10:37 AM

All replies

  • Is there any news?

    Wednesday, August 6, 2008 3:43 AM
  • I found this thread when I was googling garbled unicode text in VS IDE build output. I was actually inspired by the question description related to pipes and "VS_UNICODE_OUTPUT".
    So I would like to answer this pretty old question basing on my investigation, just in case someone might need this information.

    You have to create pipe by yourself. Cl.exe doesn't create any pipes. However, it does redirect its unicode output to the pipe write port specified by VS_UNICODE_OUTPUT=xxxx. 

    For a process that runs cl.exe, if VS_UNICODE_OUTPUT=xxxx is set, cl.exe will output to the pipe write port(xxxx), then you can pick the output from the paired pipe read port.
    If VS_UNICODE_OUTPUT is not defined, CL.exe will output to its stderr and stdout normally.

    I've attached my demo source code(in C#). You can comment/uncomment the line below to check the difference.

    info.EnvironmentVariables.Add("VS_UNICODE_OUTPUT", rawHandle.ToString());

    demo source code:

    using Microsoft.Win32.SafeHandles;
    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using System.Linq;
    using System.Runtime.InteropServices;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    
    namespace ConsoleApp1
    {
        [StructLayout(LayoutKind.Sequential)]
        internal class SecurityAttributes
        {
            private uint nLength;
            public IntPtr lpSecurityDescriptor;
            public bool bInheritHandle;
            public SecurityAttributes()
            {
                this.nLength = (uint)Marshal.SizeOf(typeof(SecurityAttributes));
            }
        }
        class Program
        {
            private static SafeFileHandle unicodePipeReadHandle;
            private static SafeFileHandle unicodePipeWriteHandle;
            private static IntPtr NullIntPtr = new IntPtr(0);
    
            [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
            internal static extern bool CreatePipe(
                out SafeFileHandle hReadPipe, 
                out SafeFileHandle hWritePipe, 
                SecurityAttributes lpPipeAttributes, 
                int nSize);
    
            [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
            internal static extern bool ReadFile(SafeFileHandle hFile, 
                byte[] lpBuffer, 
                uint nNumberOfBytesToRead, 
                out uint lpNumberOfBytesRead, 
                IntPtr lpOverlapped);
    
            static void Main(string[] args)
            {
                //create unnamed pipe and wait for Unicode output in worker thread
                BeginUnicodeOutput();
    
                // your path to source
                string src1 = @"C:\source\source1.cpp";
                // your path to cl.exe
                string compilerPath = @"C:\Program Files (x86)\Microsoft Visual Studio\2017"+
                    @"\Professional\VC\Tools\MSVC\14.10.25017\bin\HostX64\x64\cl.exe";
    
                ProcessStartInfo info = new ProcessStartInfo();
                info.Arguments = $@"/c ""{src1}"" ";
                info.FileName = compilerPath;
                info.UseShellExecute = false;
                info.CreateNoWindow = true;
                info.RedirectStandardOutput = true;
                info.RedirectStandardError = true;
    
                //make sure VS_UNICODE_OUTPUT is not defined in system env variables
                //add env varible VS_UNICODE_OUTPUT=xxxx to indicate cl.exe to redirect 
                //its Unicode output to xxxx(a decimal integer, fd of pipe write port)
                IntPtr rawHandle = unicodePipeWriteHandle.DangerousGetHandle();
                //comment the line below, cl.exe will write its output to process stderr and stdout
                info.EnvironmentVariables.Add("VS_UNICODE_OUTPUT", rawHandle.ToString());
    
                Process p = new Process();
                p.StartInfo = info;
                p.OutputDataReceived += OnOutputDataReceived;
                p.ErrorDataReceived += OnErrorDataReceived;
    
                p.Start();
    
                p.BeginErrorReadLine();
                p.BeginOutputReadLine();
    
                p.WaitForExit((int)TimeSpan.FromSeconds(30).TotalMilliseconds);
                Console.WriteLine($"[DONE]: Process has exited with code {p.ExitCode}");
            }
    
            public static void BeginUnicodeOutput()
            {
                unicodePipeReadHandle = (SafeFileHandle)null;
                unicodePipeWriteHandle = (SafeFileHandle)null;
                var attributes = new SecurityAttributes() 
                { 
                    lpSecurityDescriptor = NullIntPtr, 
                    bInheritHandle = true 
                };
                if (CreatePipe(out unicodePipeReadHandle, out unicodePipeWriteHandle, attributes, 0))
                {
                    ThreadPool.QueueUserWorkItem(new WaitCallback(ReadUnicodeOutput));
                }
                else
                {
                    Console.WriteLine("Failed to create pipe");
                    Environment.Exit(-1);
                }
            }
    
            public static void ReadUnicodeOutput(object stateInfo)
            {
                byte[] numArray = new byte[1024];
                uint lpNumberOfBytesRead;
                while (ReadFile(unicodePipeReadHandle, numArray, 1024U, out lpNumberOfBytesRead, NullIntPtr) 
                    && lpNumberOfBytesRead != 0U)
                {
                    //convert bytes into Unicode string
                    string str = Encoding.Unicode.GetString(numArray, 0, (int)lpNumberOfBytesRead);
                    Console.WriteLine("[UNICODE]: " + str);
                }
            }
    
            // there will be no output in stdout and stderr if VS_UNICODE_OUTPUT=xxxx is set
            private static void OnOutputDataReceived(object sender, DataReceivedEventArgs e)
            {
                Console.WriteLine("[stdout]: " + e.Data);
            }
            private static void OnErrorDataReceived(object sender, DataReceivedEventArgs e)
            {
                Console.WriteLine("[stderr]: " + e.Data);
            }
        }
    }



    • Edited by Dele Cui Monday, August 24, 2020 3:52 AM Formatted the source code to improve readability on web page
    Monday, August 24, 2020 3:35 AM