none
PeekNamedPipe on anonymous pipes always returns 'zero bytes available'

    Question

  • Hi there, I am writing a small utility console application that spawns two other console applications.  These applications are actually chess engines, using the UCI protocol (a simple text protocol designed to allow the GUI and engine elements to be separated and the engines to be interchanged).

     

    Anyway I am having problems successfully reading the output from the spawned processes.  My main thread does something like this (following the sample code here http://support.microsoft.com/kb/190351):

     

    1) Create two anonymous pipes for each of the engines.

    2) Duplicate the engine stdout pipe to be its stderr pipe.

    3) Use SetHandleInformation set to my side of the pipes as being uninheritable (have also tried use DuplicateHandle to do this - it makes no difference).

    4) Create the process in a new console, using the new in/out/err handles.

    5) Create an I/O thread to manage the I/O between my program and the engines.  This uses simple I/O queue objects to communicate from and to the engine. It employs thread locking (critical section), and event notification (auto-reset event object) .

    6) Perform comms to the engines via the I/O queues.

     

    Now the I/O thread uses WaitForMultipleObjects to listen for the output pipes from the engines to become signalled and also for the event objects on each of the 'to engine' I/O queues to become signalled (i.e. there is something to send to the engines).

     

    The problem I am having is that using a ReadFile on the pipes from the engines often causes a block, which I understand and it seems the correct solution to this is to use PeekNamedPipe to see how many characters are available.  Once you know this you can then use ReadFile to read the correct number of characters.  However when I use PeekNamedPipe, the lpTotalBytesAvail is always returned as 0 and I/O thread spends all its time being told a pipe has data and being unable to find any data to read.  I have tried various combinatons of parameters to PeekNamedPipe, but it makes no difference.

     

    Can anyone provide some advice please?

     

    Cheers,

    Andy

    Friday, August 29, 2008 8:50 AM

All replies

  • I don't think WaitForMultipleObjects likes pipe handles.

    WaitForMultipleObjects
    http://msdn.microsoft.com/en-us/library/ms687025(VS.85).aspx

    See 'Remarks' section:

    The WaitForMultipleObjects function can specify handles of any of the following object types in the lpHandles array:

    • Change notification
    • Console input
    • Event
    • Memory resource notification
    • Mutex
    • Process
    • Semaphore
    • Thread
    • Waitable timer
    Maybe try ReadFileEx with an lpOverlapped specified so you can do an asynchronous read, then notify an event object to indicate when a read chunk is done, instead?
    Friday, August 29, 2008 3:34 PM
  •  

    OK, I have put in the new structure.  I issue a ReadFileEx on the output handle of engine child process and have made use of the overlapped.hEvent handle to create an event for WaitForMultipleObjects to use.  The completion routine then simply sets the event if the read was successful and I get information about the read using GetOverlappedResult and then re-issue the read.

     

    The ReadFileEx's are issued before the re-entry into WaitForMultipleObjects (obviously) but I am having a problem with the second ReadFileEx blocking, so the WaitForMultipleObjects never gets called.  It's not supposed to ever block is it?  It's 100% async?

     

    Are you allowed only one async read per thread, even for different handles? Doesn't seem right.

     

    Cheers,

    Andy

     

    Friday, August 29, 2008 11:10 PM
  •  

    Hi,

     

     

    I'm having the same problem, after reading a bit, it seems like ReadFile blocks, when reading from a unnamed pipe, if there is nothing to read. This behaviour is the same with ReadFileEx, and of course ReadFile with an overlapped structure.

     

    Using an unnamed pipe with PeekNamedPipe, allways returns 0 bytes to be read, so that one isn't usefull either. If someone knows a solution or workaround, that would be highly appreciated!

     

    The only two workarounds I can think of is to use a temporary file to exchange information between the two processes, or to spawn a thread that tries to call ReadFile - if it blocks, it can be killed. Both of these seem really really ugly, there must be another way!

     

    anyone with any input on this?

    Wednesday, October 08, 2008 9:19 PM
  • Yes, I have found the solution.  You need to open the anonymous pipe using this function which opens it with overlapped mode.  You then need to use WaitForMultipleObjectsEx so that the thread is alertable and the I/O completion routine is executed.

     

    Hope that helps,

    Andy

     

    Code Snippet

    static ULONG PipeSerialNumber = 1;

    static BOOL APIENTRY MyCreatePipeEx(
       OUT LPHANDLE lpReadPipe,
       OUT LPHANDLE lpWritePipe,
       IN LPSECURITY_ATTRIBUTES lpPipeAttributes,
       IN DWORD nSize,
       DWORD dwReadMode,
       DWORD dwWriteMode
    )

    /*++

    Routine Description:

    The CreatePipeEx API is used to create an anonymous pipe I/O device.
    Unlike CreatePipe FILE_FLAG_OVERLAPPED may be specified for one or
    both handles.
    Two handles to the device are created.  One handle is opened for
    reading and the other is opened for writing.  These handles may be
    used in subsequent calls to ReadFile and WriteFile to transmit data
    through the pipe.

    Arguments:

    lpReadPipe - Returns a handle to the read side of the pipe.  Data
    may be read from the pipe by specifying this handle value in a
    subsequent call to ReadFile.

    lpWritePipe - Returns a handle to the write side of the pipe.  Data
    may be written to the pipe by specifying this handle value in a
    subsequent call to WriteFile.

    lpPipeAttributes - An optional parameter that may be used to specify
    the attributes of the new pipe.  If the parameter is not
    specified, then the pipe is created without a security
    descriptor, and the resulting handles are not inherited on
    process creation.  Otherwise, the optional security attributes
    are used on the pipe, and the inherit handles flag effects both
    pipe handles.

    nSize - Supplies the requested buffer size for the pipe.  This is
    only a suggestion and is used by the operating system to
    calculate an appropriate buffering mechanism.  A value of zero
    indicates that the system is to choose the default buffering
    scheme.

    Return Value:

    TRUE - The operation was successful.

    FALSE/NULL - The operation failed. Extended error status is available
    using GetLastError.

    --*/
    {
     HANDLE ReadPipeHandle, WritePipeHandle;
     DWORD dwError;
     CHAR PipeNameBuffer[ MAX_PATH ];

     //
     // Only one valid OpenMode flag - FILE_FLAG_OVERLAPPED
     //
     if ((dwReadMode | dwWriteMode) & (~FILE_FLAG_OVERLAPPED)) {
      SetLastError(ERROR_INVALID_PARAMETER);
      return FALSE;
     }

     //
     //  Set the default timeout to 120 seconds
     //

     if (nSize == 0) {
      nSize = 4096;
     }

     sprintf( PipeNameBuffer,
      "\\\\.\\Pipe\\TruthPipe.%08x.%08x",
      GetCurrentProcessId(),
      PipeSerialNumber++
      );

     ReadPipeHandle = CreateNamedPipeA(
      PipeNameBuffer,
      PIPE_ACCESS_INBOUND | dwReadMode,
      PIPE_TYPE_BYTE | PIPE_WAIT,
      1,             // Number of pipes
      nSize,         // Out buffer size
      nSize,         // In buffer size
      120 * 1000,    // Timeout in ms
      lpPipeAttributes
      );

     if (! ReadPipeHandle) {
      return FALSE;
     }

     WritePipeHandle = CreateFileA(
      PipeNameBuffer,
      GENERIC_WRITE,
      0,                         // No sharing
      lpPipeAttributes,
      OPEN_EXISTING,
      FILE_ATTRIBUTE_NORMAL | dwWriteMode,
      NULL                       // Template file
      );

     if (INVALID_HANDLE_VALUE == WritePipeHandle) {
      dwError = GetLastError();
      CloseHandle( ReadPipeHandle );
      SetLastError(dwError);
      return FALSE;
     }

     *lpReadPipe = ReadPipeHandle;
     *lpWritePipe = WritePipeHandle;
     return( TRUE );
    }

     

     

    Wednesday, October 08, 2008 9:38 PM
  • Hi Andy,

     

     

    That was excatly what I needed, of course using named pipes is a solution that will provide the functionality I need.

     

    Thanks!

     

    Uffe

    Friday, October 10, 2008 9:39 AM
  •  

    Well I am now facing the other side of this saga now.  My utility program has been working very well and I am able to spawn chess engine processes and exchange data without problem, but now I have started writing the chess engine side of the equation. My main thread sits waiting for input from the external program or output from the chess engine search thread or for the quit-event to be signalled (set by a console control handler).

     

    This works fine if I invoke the engine process in a console, however if I invoke it as a spawned process, which will use anonymous pipes for stdin/stdout/stderr, then WaitForMulitpleObjects will always be signalled for stdin even when there is nothing to read - calling ReadFile on stdin will block, breaking the rest of the output and quit functionality.

     

    I have tried setting the pipe to non-blocking mode using SetNamedPipeHandleState but I get access denied, presumably because the handle needs GENERIC_WRITE set in order to change the flags, and it just has GENERIC_READ.  I have also tried using overlapped I/O to do the read, but that doesn't work as the pipe wasn't opened in overlapped mode.

     

    Can anyone tell me the correct/best way of waiting for stdin via a pipe as well as waiting for other handles/events?


    Cheers,

    Andy

    Thursday, November 27, 2008 9:17 PM