none
RC2/AES CBC compatibility issues between Microsoft .NET and Crypto++/OpenSSL RRS feed

  • Question

  • Hi,

    I'm trying to decrypt data that was encrypted using code implemented in Microsoft .NET, that is then decoded in C++ using Crypto++. Unfortunately when decoding, the data will not be successfully decoded. Around 30%-50% of the beginning of the data is correctly decoded, after that, the decoded data becomes different from the original. How much that is possible to decrypt correctly seems to depend on the input data that was given to the .NET encryption. Different input, gives different success rate, same input gives consistent results.

    The .NET implementation looks something like this,
            public static void Encrypt(byte[] plainData, string outputFile, byte[] key, byte[] iv)
            {
                // Create an configure the crypto service provider
                SymmetricAlgorithm rc2 = CreateCryptoServiceProvider(key, iv);
                // Transform the input data through the crypto and save in output file
                using (ICryptoTransform encryptor = rc2.CreateEncryptor())
                {
                    using (FileStream outputStream = new FileStream(outputFile, FileMode.OpenOrCreate))
                    {
                        using (CryptoStream cryptoStream = new CryptoStream(outputStream, encryptor, CryptoStreamMode.Write))
                        {
                            cryptoStream.Write(plainData, 0, plainData.Length);
                            cryptoStream.FlushFinalBlock();
                        }
                    }
                }
            }
            private static SymmetricAlgorithm CreateCryptoServiceProvider(byte[] key, byte[] iv)
            {
                    // Configure crypto service provider
                    RC2CryptoServiceProvider rc2 = new RC2CryptoServiceProvider()
                    {
                        Key = key,
                        IV = iv,
                        Mode = CipherMode.CBC,
                        EffectiveKeySize = 128,
                        Padding = PaddingMode.PKCS7,
                        //BlockSize = 64, -- If inspecting the rc2 instance, the block size property is 64. If we set it again here, the first 8 bytes in the decoded data will be corrupt when decoding in .NET!!!
                        FeedbackSize = 8,
                        UseSalt = false
                    };
                    return rc2;
            }

    And the decoding using Crypto++ looks like this,

      byte *key, *iv;
      size_t keySize, ivSize;
      int effectiveKeyLen = 128;
      int feedbackSize = 8;
    .... the key and iv is set.
      // Load the encrypted test file
      byte* cryptData;
      int cryptDataLength;
      loadFile("random.crypt", cryptData, cryptDataLength);
      
      // Setup output buffer
      byte* decryptData = new byte[cryptDataLength];
      ArraySink* decryptSink = new ArraySink(decryptData, cryptDataLength);
      int decryptDataLength;
        // Setup CryptoPP RC2 decoder
        CryptoPP::RC2::Decryption rc2Decryption(key, keySize, effectiveKeyLen);
        CryptoPP::CBC_Mode_ExternalCipher::Decryption cbcDecryption(rc2Decryption, iv, feedbackSize);
        // Transform the encrypted input through the cipher (since decryption fails some way through, the padding will not be found)
        CryptoPP::StreamTransformationFilter stf(cbcDecryption, decryptSink, BlockPaddingSchemeDef::NO_PADDING /*PKCS_PADDING*/, false);
        stf.Put(cryptData, cryptDataLength);
        stf.MessageEnd();
        decryptDataLength = decryptSink->TotalPutLength();

    The KEY and IV byte arrays is set from base64 decoded strings, I have verified that the byte array of the key and IV is the same in both .NET and C++. 

    I have also tried this with AES instead of RC2 and get the same unsuccessful result. I have also tried to decode RC2/CBC using OpenSSL and still get the same unsuccessful results. 

    If I reverse the process and encode using Crypto++ and then decode using .NET, I just get garbage and nothing is successfully decoded. If I encode and decode only using C++, its successful, same if I only encode and decode using .NET.

    The funny(?) thing is that it works for a while, it successfully decodes a number of bytes and then it turns into garbage, I have not seen any correlation with how many bytes is correctly decoded and for example the block size.

    What's wrong? And how come there is a partial success in decoding the data? (And a complete fail when the .NET code tries to decode the C++ encoded data?)

    Best regards,
    Hans

    • Moved by Mike FengModerator Thursday, October 25, 2012 8:15 AM CLR (From:.NET Base Class Library)
    Wednesday, October 24, 2012 10:20 AM

Answers

  • Hi UncleRedz,

    I run your test project.

    In the C# test project, I get expected result: both the plain file and the crypo file are 1181 bytes. As you said, it is correct.

    After test the UnmanagedTest.exe, I noticed that, the file size is changed to 1184, so I think this is what you didn't decrypt the file correctly.

    I think when you find why it read the crypt data araise 3 bits, the answer show up.

    Have a nice day.


    Ghost,
    Call me ghost for short, Thanks
    To get the better answer, it should be a better question.

    Monday, October 29, 2012 11:54 AM

All replies

  • I'm not sure if the my  answer is correct, but I think so (or close).  The encryption is 16 bit words.  You are probably only decoding half the message.   The algorithm takes two bytes and combines then into a word then encrypts the word.  When decrypting the encrypt function reads two bytes from the file and decrypts, then puts the data back into a buffer.  So the buffer size of the results is twice the excryption buffer.  The feedback size may need to be changed from 8 to 16.

     From : byte* decryptData = new byte[cryptDataLength];

     To : byte* decryptData = new byte[2 * cryptDataLength];

    It is possible you are decrypting the entire message and just getting garbage at the end of the message.


    jdweng

    Wednesday, October 24, 2012 1:22 PM
  • My understanding of RC2 and CBC is that the block size is 64 bits (or 8 bytes) and the data is encrypted and decrypted in chunks of 8 bytes. The size of the encrypted data only increases as a result of padding the last block to an even 8 bytes. With the exception of that padding, the size of the data before and after encryption is the same. (And the CBC part of it uses the previous block of encrypted data as input to the encryption of the next block of data.)

    In any case, just to make sure, I tried your suggestions and it doesn't change the result. Since I can compare the original plain data with the decrypted data, I can verify that I did not get all of the data decrypted.

    Wednesday, October 24, 2012 2:40 PM
  • Hi UncleRedz,

    May be you posted the test result will be helpful.

    Before the test, are the orginal bytes array are the same?

    Best regards,


    Ghost,
    Call me ghost for short, Thanks
    To get the better answer, it should be a better question.

    Thursday, October 25, 2012 8:36 AM
  • Hi Ghost,

    I've posted example code here which shows the problem, EncryptionTest.zip.

    For testing the input data a file containing 1181 bytes of the letter "A" is used, after encryption in .NET, that becomes 1184 bytes, as a few bytes of padding is needed for the last block.

    So basically the original data looks like this, "AAAAAAAAAAAA" and the decrypted data also looks like "AAAAAAAAAAAA", but only the first 175 bytes, the remaining bytes is garbage. 

    That is what really surprises me, I would think that either the encryption/decryption works, or it doesn't, it can not work for a while and then stop working. which is basically what it is doing now.

    If anyone would like to test the example code, you first have to run the C# EncryptionTest.exe which does encryption and decryption, but also generates some test files. Then run the UnmanagedTest.exe, which is the C++ code that tries to decrypt the file generated from the C# exe.

    Thursday, October 25, 2012 12:23 PM
  • Hi UncleRedz,

    I run your test project.

    In the C# test project, I get expected result: both the plain file and the crypo file are 1181 bytes. As you said, it is correct.

    After test the UnmanagedTest.exe, I noticed that, the file size is changed to 1184, so I think this is what you didn't decrypt the file correctly.

    I think when you find why it read the crypt data araise 3 bits, the answer show up.

    Have a nice day.


    Ghost,
    Call me ghost for short, Thanks
    To get the better answer, it should be a better question.

    Monday, October 29, 2012 11:54 AM