Enveloped data verify fails with NTE_BAD_DATA error on WIN XP SP3 when data is decrypted using crypt32 api function CryptMsgControl with private key stored in RSA SID800 smart card RRS feed

  • Question

  • Problem Description:

    1.       A RSA smart card SID800 is used to verify the PKCS#7 enveloped data.

    2.       The PKCS#7 enveloped data verification is done using the Microsoft Crypto API(crypt32.dll) functions.

    3.       The verification works fine on windows 7 machine but fails  on Windows XP SP3 machine.

    4.       We have our own logging in our library that does this stuff and the messages on Windows XP XP3 shows the following information:

                 a. We are able to encode and encrypt the data using the private key stored on the smart card reader.

                 b. We are able to decode the enveloped data that is sent by a third party.

                 c. The decrypt of enveloped data using the Private key on the smart card readers fails with windows error NTE_BAD_DATA on Windows XP XP3 but works fine on Windows 7.

                d. Find below is the Microsoft Crypto API used to decrypt the enveloped data. This function fails with the error code NTE_BAD_DATA.

                             if(CryptMsgControl (

                                              hMsg,               /* Handle to the decoded message */




    5.       A test program is written to verify the envelope data that was failing on the Windows XP3 system.

                  a.       When the same enveloped data is given as input to the test program on Windows 7, the code was able to use MS Crypto API is to decrypt successfully and verify the enveloped data using the private key stored on the smart card. Whereas, the same test program with the same enveloped data fails to decrypt with error NTE_BAD_DATA on Windows XP SP3. We have tested this test program on more than one Windows XP SP3 system and got the same error in order to eliminate the case where the issue is specific to a system.

                 b.      The same decoded data when decrypted using the Microsoft Crypto API function CryptMsgControl fails on the Windows XP SP3 but works fine in the other machine(Windows 7) though the data passed to two functions are same.

                 c.       It clearly shows that there is some problem with the underlying Microsoft Crypto API libraries or the Microsoft Base Smart card crypto graphic provider or RSA smart card reader that uses the windows smart card API’s.

    6.       When a different vendor smart card reader is used,  we were able to decrypt and verify the enveloped data successfully on Windows XP SP3 and Windows 7 using certificates found on smart card reader.

    Thursday, September 29, 2011 7:28 AM