RSA/AES using CryptXXX API RRS feed

  • Question

  • I have zero experience with cryptography and am attempting to implement a reverse-engineered (which is somewhat documented) network protocol that uses it. The protocol works as follows: a 256-AES and 128-bit IV are generated by the client which is then encrypted with an RSA private key (which I have in a .PEM file) and sent to initialize the session. According to the documentation this encryption scheme is futile anyway.

    I'm interested in what the correct way to go about this would be; at present I do the following:

    1) Load the RSA private key from the PEM file using CryptStringToBinary and CryptDecodeObjectEx

    2) Create a context using CryptAquireContext (using MS_ENH_RSA_AES_PROV / PROV_RSA_AES)

    3) Import the RSA private key using CryptImportKey

    4) Generate an AES key using CryptGenKey (CALG_AES_256 / size: 0x100 flags: CRYPT_EXPORTABLE)

    5) Use CryptGenRandom to generate the IV and then associate it with the AES key using CryptSetKeyParam (KP_IV)

    6) To get the AES key I use CryptExportKey (using the priv. RSA key) then decrypt the blob key data using CryptDecrypt again with the priv. RSA key.

    7) Lastly I use CryptEncrypt with the priv. RSA key to encrypt a buffer containing the AES and the IV [and send it to the server]

    The problem is that the resulting encrypted data I'm sending is invalid according to the server (of which I don't have access to and thus I don't know why it's invalid)

    I'm aware of a working implementation (albeit in Java) which is very small source-wise if that'd help understand what exactly I'm on about.


    Specifically the data structure I need to generate is as follows:

    #pragma pack(1)

    struct gsp_init { DWORD Len; // Message length. WORD Magic1; // 0x00AD // Following encrypted with the RSA private key. BYTE AesKey[32]; // Randomly generated AES-256 key. BYTE AesIV[16]; // Randomly generated 128-bit IV WORD Magic2; // 0xF00F };

    #pragma pack()

    Since these fields are randomly generated presumably it doesn't matter what AesKey/AesIV's value is (thus ruling out them out as being interpreted as invalid). Therefor I'm confident the problem is with the RSA encryption.

    • Edited by Matt.J Friday, August 30, 2013 5:48 AM
    Thursday, August 29, 2013 6:34 PM

All replies

  • Okay so I've managed to decrypt a sample encrypted block (ie., a gsp_init structure) which I ripped off the wire from a legitimate client and the resulting plaintext is correct however this was done using OpenSSL. When attempting to do the same using CryptDecrypt and the same key I receive the error BAD_DATA (specifying CRYPT_DECRYPT_RSA_NO_PADDING_CHECK enables the function to succeed however the output appears garbage)

    Why would this be?

    I've tried reversing the endianness of the encrypted data prior to CryptDecrypting() it but to no avail. The padding scheme I'm confident is also the same (PKCS1 1.5).

    I suspect the answer to this question will address my original problem.

    • Edited by Matt.J Friday, August 30, 2013 5:59 AM
    Friday, August 30, 2013 5:28 AM