Use RSACryptoServiceProvider for encryption and let openssl decrypt RRS feed

  • Question

  • I want to write a .NET client that sends an encrypted message to a server written in C with openssl crypto library, and decrypt the message there.
    I use RSA with PKCS1 padding. I have managed to get that to work for C clients using crypto library, Java clients using standard 1.5 library classes.

    The .NET client uses a public key generated by the server that uses openssl crypto library.
    The client encrypts without error.
    But the server cannot decrypt the sequence from the .NET client:
    error:2006E079:BIO routines:BIO_puts:unsupported method
    I see 3 possible sources of error in my code.
    1. How big is the chance that the public key encoded by the server for the .NET client is not correct even so it is accepted by the .NET client without error?
    2. The way how I encode the encrypted sequence in the .NET client is wrong? That would be an obvious reason for the error during decryption.
    3. There are 2 different versions of PKCS1 padding used in .NET and the C crypto library? From documentation I got the impression that it fits.

    To test the scenario  I use the J# code at the end of this mail. First I made sure that encryption / decryption works in J# alone.
    To get the public key generated with openssl into a format that can be used in .NET I use the following crypto library calls:

        BIO *b64, *out, *wbio;
        b64 = BIO_new(BIO_f_base64());
        out = BIO_new(BIO_s_file());

        int val[1] = {65537};
        char *buf = (char*) val;
        BIO_write(wbio, val, 3);  /* exponent, only 3 bytes written */
        BIO_write(wbio, rsa_public->n->d, 128); /* modulus */

    The result I put into the XML format that .NET expects.

    The server decrypts the password with the same private key used to generate the public key:
            retval = RSA_private_decrypt(RSA_size(rsa_private), encryptedPasswordCharArrayfromNetClient, cleartext-output, rsa_private, RSA_PKCS1_PADDING);

    I tested my code so that encryption /  decryption works in the C code using openssl crypto library, latest version.
    I tested my J# code so that encryption /  decryption works using standard .NET class  for RSA.
    I have trouble to get the two together. If I use RSA_private_decrypt with a byte sequence created from the .NET client I get the error:
    Error: 537321593
    error:2006E079:BIO routines:BIO_puts:unsupported method

    Debugging shows that the error is returned from crypto library when it checks the used padding:
        case RSA_PKCS1_PADDING:

    The .NET J# client code:

    package RSAEncryption.java;

    import System.*;
    import System.Security.Cryptography.*;
    import System.Text.*;

    class RSASample
        public static void main(String[] args)
            try {
                // Create a UnicodeEncoder to convert between byte array and string.
                UnicodeEncoding byteConverter =  new UnicodeEncoding();
                ubyte unicodeDataToEncrypt[] = byteConverter.GetBytes("myPassword!");

                Encoding ascii = Encoding.get_ASCII();
                Encoding unicode = Encoding.get_Unicode();
                ubyte dataToEncrypt[] = Encoding.Convert(unicode, ascii, unicodeDataToEncrypt);
                RSACryptoServiceProvider rsa = new RSACryptoServiceProvider();

                String publicRSAKeyXML = "<RSAKeyValue><Modulus>h...deleted...0=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>";

                RSAParameters rsaParameter = rsa.ExportParameters(false);

                // 3. parameter: a boolean flag specifying no OAEP padding.
                ubyte [] encryptedData = RSAEncrypt(dataToEncrypt, rsaParameter, false);

        // Display the encrypted sequence to the console.
        System.out.print("static signed char encryptedPasswordStringfromNetClient[] = {");
        for (int i = 0; i < encryptedData.length; i++)
            if (i % 16 == 0)
                System.out.print("    ");
            System.out.print((byte)encryptedData[i] + ", ");
            catch (ArgumentNullException exp) {
                //Catch this exception in case the encryption did
                //not succeed.
                Console.WriteLine("Encryption failed.");
        } //main

        public static ubyte[] RSAEncrypt(ubyte dataToEncrypt[],
            RSAParameters rsaKeyInfo, boolean doOaepPadding)
            try {
                // Create a new instance of RSACryptoServiceProvider.
                RSACryptoServiceProvider rsa =  new RSACryptoServiceProvider();
                // Import the rsa Key information. This only needs
                // to include the public key information.
                // Encrypt the passed byte array and specify OAEP padding. 
                // OAEP padding is only available on Microsoft Windows XP or
                // later. 
                return rsa.Encrypt(dataToEncrypt, doOaepPadding);
            // Catch and display a CryptographicException 
            // to the console.
            catch (CryptographicException e) {
                return null ;
        } //RSAEncrypt

    } //RSASample

    Friday, December 5, 2008 1:28 PM