SChannel problem on Windows Server 2008 R2 RRS feed

  • Question

  • I am having a problem with our SChannel server implementation on Windows 2008 R2 that does not happen when running the same server on Windows 2003. Our server uses an Schannel implementation that was written about nine years ago and has been working well throughout that time. We recently discovered that when the server is run on 2008 R2, one of our clients, written in Java, is unable to complete the SSL handshake while the other one, native, can. The Java client, which uses the very simple Java SSLSocket class, can successfully connect to the server when the server runs on Windows 2003 but fails when the server is on 2008 R2. The native client can complete the handshake to either server. I wrote a simple Java application that just takes an address and port and uses Java's SSLSocket to create an SSL connection and it fails in the same manner against the server running on 2008 R2.

    The handshake process fails right at the end when the server is processing the client's handshake finished message, AcceptSecurityContext returns the error SEC_E_DECRYPT_FAILURE. Up to this point, the server has successfully processed the Client Hello, Client Key Exchange and Change Cipher Spec messages - the cipher suite and session key have been agreed upon but when the Java client sends the finished message, the first one to be encrypted using the session key, the server is unable to decrypt it and we abort the handshake. Comparing the Java client connecting to the server running on 2003 vs 2008 R2, the only real difference is that 2003 uses MD5 for the hashing algorithm whereas 2008 uses SHA1. I set the 2008 server to only use MD5 but that did not help. I have also enabled all cipher suites on the 2008 R2 server and it still fails. For some reason, Java SSLSocket prefers to use SSL v3 - I have forced Java to use TLS v1 but that also fails.

    I've debugged this on both ends and sniffed the network and don't see anything wrong with the message flow or how we deliver the messages to SChannel. It seems that the client and server think they agree on the cipher suite and session key but in fact are not using the same values, otherwise I don't see how SChannel is unable to decrypt the first message. I am able to get the session key from the Java side but have not found a way to extract it from SChannel so I have been unable to compare the two sides. The cipher suite chosen is SSL_RSA_WITH_RC4_128_SHA - again I can verify that on the Java side but not on the SChannel side.

    The SChannel code for the handshake process is pretty straightforward - I've reviewed it and don't see any problems (though I'm not an SChannel expert by any means). Below is the method we use to process the incoming handshare messages - during the handshake process, our socket code is just reading bytes from the socket and feeding them to this method:

    SECURITY_STATUS SecureCommunication::CreateServerSecurityContext(bool authenticateClient, // in
                                                                     bool newContext,         // in
                                                                     int bufferSize,          // in
                                                                     signed char* buffer)     // in
        TimeStamp        tsExpiry;
        SECURITY_STATUS  securityStatus;
        DWORD            dwSSPIFlags, dwSSPIOutFlags;

        securityStatus = SEC_E_SECPKG_NOT_FOUND; // default error if we run out of packages

                      ASC_REQ_REPLAY_DETECT   |
                      ASC_REQ_CONFIDENTIALITY |
                      ASC_REQ_EXTENDED_ERROR  |
                      ASC_REQ_ALLOCATE_MEMORY |

        if (authenticateClient)
            DEBUGDUMPSTRING("CreateServerSecurityContext : Authenticate Client");
            dwSSPIFlags |= ASC_REQ_MUTUAL_AUTH;

        if (newContext)
            DEBUGDUMPSTRING("CreateServerSecurityContext : newContext = TRUE");
            DEBUGDUMPSTRING("CreateServerSecurityContext : newContext = FALSE");

        // InBuffers[1] is for getting extra data that
        //  SSPI/SCHANNEL doesn't process on this
        //  run around the loop.
        CopyMemory(_inputBuffer,   // destination
                   buffer,         // source
                   bufferSize);    // length

        _inBuffers[0].pvBuffer   = _inputBuffer;
        _inBuffers[0].cbBuffer   = bufferSize;
        _inBuffers[0].BufferType = SECBUFFER_TOKEN;

        _inBuffers[1].pvBuffer   = NULL;
        _inBuffers[1].cbBuffer   = 0;
        _inBuffers[1].BufferType = SECBUFFER_EMPTY;

        _messageIn.cBuffers      = 2;
        _messageIn.pBuffers      = _inBuffers;
        _messageIn.ulVersion     = SECBUFFER_VERSION;
        // Initialize these so if we fail, pvBuffer contains NULL,
        // so we don't try to free random garbage at the quit
        _outBuffers[0].pvBuffer   = NULL;
        _outBuffers[0].BufferType = SECBUFFER_TOKEN;
        _outBuffers[0].cbBuffer   = 0;

        // set _messageOut for AcceptSecurityContext call
        _messageOut.cBuffers  = 1;
        _messageOut.pBuffers  = _outBuffers;
        _messageOut.ulVersion = SECBUFFER_VERSION;

        DEBUGDUMPHEX("CreateServerSecurityContext: BEFORE hContext.dwLower = ", _hContext.dwLower);
        DEBUGDUMPHEX("CreateServerSecurityContext: BEFORE hContext.dwUpper = ", _hContext.dwUpper);

        DEBUGDUMPDECIMAL("CreateServerSecurityContext: buffer with size = ", bufferSize);
        PrintHexDump(bufferSize, (unsigned char*) buffer);

        securityStatus = g_securityFunc.AcceptSecurityContext(g_pServerCredentialsHandle,
                                                              (newContext ? NULL : &_hContext),
                                                              (newContext ? &_hContext : NULL),
        DEBUGDUMPSTRING("CreateServerSecurityContext: securityStatus");

        if (FAILED(securityStatus) && (dwSSPIOutFlags & ISC_RET_EXTENDED_ERROR))
            DEBUGDUMPSTRING("CreateServerSecurityContext : WARNING: Extended error encountered.");
            _securityContextExtendedError = true;

        DEBUGDUMPHEX("CreateServerSecurityContext: AFTER hContext.dwLower = ", _hContext.dwLower);
        DEBUGDUMPHEX("CreateServerSecurityContext: AFTER hContext.dwUpper = ", _hContext.dwUpper);
        DEBUGDUMPHEX("CreateServerSecurityContext: AFTER Context out flags = ", dwSSPIOutFlags);

        for (int i=0; i < MAX_BUFFERS; i++)
            if (_outBuffers[i].BufferType != SECBUFFER_EMPTY && _outBuffers[i].pvBuffer != NULL)
                DEBUGDUMPDECIMAL("CreateServerSecurityContext: output buffer type = ", _outBuffers[i].BufferType);
                DEBUGDUMPDECIMAL("CreateServerSecurityContext: output buffer bytes = ", _outBuffers[i].cbBuffer);
                PrintHexDump(_outBuffers[i].cbBuffer, (unsigned char*) _outBuffers[i].pvBuffer);
                DEBUGDUMPDECIMAL("CreateServerSecurityContext: output buffer type = ", _outBuffers[i].BufferType);
        } // end for loop

        // If the context was successfully created, then
        // get the stream sizes: header, trailer, maximum message size
        if (securityStatus == SEC_E_OK)
            // Obtain any extra data, if it exists.

            SECURITY_STATUS status = g_securityFunc.QueryContextAttributes(&_hContext,

            if (status == SEC_E_OK)
               if ((signed) _sizes.cbMaximumMessage > _inputBufferSize)
                   // Allocate a new buffer, free up original first.
                   delete [] _inputBuffer;

                   _inputBufferSize = _sizes.cbMaximumMessage + _sizes.cbHeader + _sizes.cbTrailer;
                   _inputBuffer = new char[_inputBufferSize];

                   DEBUGDUMPDECIMAL("CreateServerSecurityContext: new _inputBufferSize=", _inputBufferSize);

               if ((signed) _sizes.cbMaximumMessage > _outputBufferSize)
                   // Allocate a new buffer, free up original first.
                   delete [] _outputBuffer;

                   _outputBufferSize = _sizes.cbMaximumMessage + _sizes.cbHeader + _sizes.cbTrailer;
                   _outputBuffer = new char[_outputBufferSize];

                   DEBUGDUMPDECIMAL("CreateServerSecurityContext: new _outputBufferSize = ", _outputBufferSize);

               DEBUGDUMPDECIMAL("CreateServerSecurityContext: _sizes.cbHeader = ", _sizes.cbHeader);
               DEBUGDUMPDECIMAL("CreateServerSecurityContext: _sizes.cbTrailer = ", _sizes.cbTrailer);
               DEBUGDUMPDECIMAL("CreateServerSecurityContext: _sizes.cbMaximumMessage = ", _sizes.cbMaximumMessage);
               DEBUGDUMPDECIMAL("CreateServerSecurityContext: _sizes.cBuffers = ", _sizes.cBuffers);
               DEBUGDUMPDECIMAL("CreateServerSecurityContext: _sizes.cbBlockSize = ", _sizes.cbBlockSize);
                DEBUGDUMPSTRING("***** CreateServerSecurityContext: Couldn't get stream sizes.");
                DEBUGDUMPHEX("CreateServerSecurityContext: Query status = ", status);
        } // securityStatus == SEC_E_OK

        // Save output message response to send to remote party - note this will
        // free the context buffer, if any.
        return securityStatus;

    Thanks in advance for any help, this one is very perplexing.


    Tuesday, March 13, 2012 11:26 PM

All replies

  • I resolved this issue. It turns out, our SChannel implementation was only enabling the SSL protocol, which on the surface seems fine as the client and server can negotiate the protocol and settle on SSL. In fact, two of our other clients, one using SChannel and the other using OpenSSL can both communicate successfully using the SSL protocol with the server on 2008 R2. Unfortunately, Java SSLSocket is unable to get through the handshake. By enabling TLS, Java SSLSocket is now able to complete the handshake.
    Monday, March 26, 2012 8:52 PM