How can I encrypt a message with .NET RSACryptoServiceProvider for a server written with OpenSSL?


  • I want to write a .NET client that sends a message encrypted  with a public key to a server written in C with OpenSSL, and decrypt the message there.
    I use RSA with PKCS1 padding. I have managed to get that to work for C clients using crypto library, Java clients using standard 1.5 library classes.

    The .NET client uses a public key generated by the server that uses OpenSSL crypto library.
    The client encrypts without error.
    But the server cannot decrypt the sequence from the .NET client:
    error:2006E079:BIO routines:BIO_puts:unsupported method
    I see 3 possible sources of error in my code.
    1. How big is the chance that the public key encoded by the server for the .NET client is not correct even so it is accepted by the .NET client without error?
    2. The way how I encode the encrypted sequence in the .NET client is wrong? That would be an obvious reason for the error during decryption.
    3. There are 2 different versions of PKCS1 padding used in .NET and the C crypto library? From documentation I got the impression that it fits.

    To test the scenario  I use the J# code at the end of this mail. First I made sure that encryption / decryption works in J# alone.
    To get the public key generated with openssl into a format that can be used in .NET I use the following crypto library calls:

        BIO *b64, *out, *wbio;
        b64 = BIO_new(BIO_f_base64());
        out = BIO_new(BIO_s_file());

        int val[1] = {65537};
        char *buf = (char*) val;
        BIO_write(wbio, val, 3);  /* exponent, only 3 bytes written */
        BIO_write(wbio, rsa_public->n->d, 128); /* modulus */

    The result I put into the XML format that .NET expects.

    The server decrypts the password with the same private key used to generate the public key:
            retval = RSA_private_decrypt(RSA_size(rsa_private), encryptedPasswordCharArrayfromNetClient, cleartext-output, rsa_private, RSA_PKCS1_PADDING);

    I tested my code so that encryption /  decryption works in the C code using openssl crypto library, latest version.
    I tested my J# code so that encryption /  decryption works using standard .NET class  for RSA.
    I have trouble to get the two together. If I use RSA_private_decrypt with a byte sequence created from the .NET client I get the error:
    Error: 537321593
    error:2006E079:BIO routines:BIO_puts:unsupported method

    Debugging shows that the error is returned from crypto library when it checks the used padding:
        case RSA_PKCS1_PADDING:

    The .NET J# client code:


    import System.*;
    import System.Security.Cryptography.*;
    import System.Text.*;

    class RSASample
        public static void main(String[] args)
            try {
                // Create a UnicodeEncoder to convert between byte array and string.
                UnicodeEncoding byteConverter =  new UnicodeEncoding();
                ubyte unicodeDataToEncrypt[] = byteConverter.GetBytes("myPassword!");

                Encoding ascii = Encoding.get_ASCII();
                Encoding unicode = Encoding.get_Unicode();
                ubyte dataToEncrypt[] = Encoding.Convert(unicode, ascii, unicodeDataToEncrypt);
                RSACryptoServiceProvider rsa = new RSACryptoServiceProvider();

                String publicRSAKeyXML = "<RSAKeyValue><Modulus>h...deleted...0=</Modulus><Exponent>AQAB</Exponent></RSAKeyValue>";

                RSAParameters rsaParameter = rsa.ExportParameters(false);

                // 3. parameter: a boolean flag specifying no OAEP padding.
                ubyte [] encryptedData = RSAEncrypt(dataToEncrypt, rsaParameter, false);

        // Display the encrypted sequence to the console.
        System.out.print("static signed char encryptedPasswordStringfromNetClient[] = {");
        for (int i = 0; i < encryptedData.length; i++)
            if (i % 16 == 0)
                System.out.print("    ");
            System.out.print((byte)encryptedData[i] + ", ");
            catch (ArgumentNullException exp) {
                //Catch this exception in case the encryption did
                //not succeed.
                Console.WriteLine("Encryption failed.");
        } //main

        public static ubyte[] RSAEncrypt(ubyte dataToEncrypt[],
            RSAParameters rsaKeyInfo, boolean doOaepPadding)
            try {
                // Create a new instance of RSACryptoServiceProvider.
                RSACryptoServiceProvider rsa =  new RSACryptoServiceProvider();
                // Import the rsa Key information. This only needs
                // to include the public key information.
                // Encrypt the passed byte array and specify OAEP padding. 
                // OAEP padding is only available on Microsoft Windows XP or
                // later. 
                return rsa.Encrypt(dataToEncrypt, doOaepPadding);
            // Catch and display a CryptographicException 
            // to the console.
            catch (CryptographicException e) {
                return null ;
        } //RSAEncrypt

    } //RSASample 

    Thursday, December 18, 2008 2:41 PM

All replies

  • I asked this question already in the past but got no replies. I thought it is because the topic might not be of interest for most people. But now I tried to find my own message by searching in this forum. I tried different key words like  RSACryptoServiceProvider, RSA_private_decrypt, opensll, my alias ... but I cannot retrieve my message other then by sequential manual scan through the entire list. Does anyone have an idea how this happens?

    Friday, December 19, 2008 9:01 AM