none
Problem while trying to encrypt Large File using any Symmetric Algorithm RRS feed

  • Question

  • Hi,

    Can anybody please help me debug this issue?

    I am trying to encrypt a big file (around 300mb) using Symmetric Algorithm (RC2 for example). I am encrypting using MemoryStream object but for some reason at the point when it calls CryptoStream.FlushFinalBlock(), it keeps throwing OutofMemoryException even though I have 3GB RAM in my machine:

    Encrypting through RC2
    Error occurred while uploading file C:\Test\AmazonS3Testing\Testing\Upload\data2
    50.zip. # of try: 1
    
    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was
     thrown.
       at System.IO.MemoryStream.set_Capacity(Int32 value)
       at System.IO.MemoryStream.EnsureCapacity(Int32 value)
       at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
       at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
       at RC2Encrypt.RC2StreamEncryptor.EncryptStream(Stream inputStream, Stream out
    putStream, String password, String salt, Int64 maxInputStreamSize) in C:\My Docu
    ments\Amazon_S3_Examples\S3Browser\RC2Encrypt\RC2StreamEncryptor.cs:line 33
       at S3Browser.Program.GetRC2EncryptFileStream(String cFilename) in C:\My Docum
    ents\Amazon_S3_Examples\S3Browser\Program.cs:line 761
       at S3Browser.Program.GetEncryptFileStream(String cFilename, EncryptMode oMode
    ) in C:\My Documents\Amazon_S3_Examples\S3Browser\Program.cs:line 662
       at S3Browser.Program.UploadFile(String bucketname, String cFilename, String k
    ey, Int64 inputLength, MetadataEntry[] metadata, Grant[] grant) in C:\My Documen
    ts\Amazon_S3_Examples\S3Browser\Program.cs:line 975
    



    Here is my code:
            private static Stream GetRC2EncryptFileStream(string cFilename)
    {
    DateTime dtStart = DateTime.Now;
    RC2StreamEncryptor oEncryptor = new RC2StreamEncryptor();
    MemoryStream memStream = new MemoryStream((int)new FileInfo(cFilename).Length);
    oEncryptor.EncryptStream(new StreamReader(cFilename).BaseStream, memStream, cPassword, cSalt, nMaxAllowFileSize);
    memStream.Position = 0;
    tsEncDecDuration += DateTime.Now.Subtract(dtStart);
    return memStream;
    }

    ....
    ....
    ....

    using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Security.Cryptography; using System.Text; namespace RC2Encrypt { public class RC2StreamEncryptor { private readonly SymmetricAlgorithm _cryptoProvider = new RC2CryptoServiceProvider(); private const int _bufferSize = 32768; public void EncryptStream(Stream inputStream, Stream outputStream, string password, string salt, long maxInputStreamSize) { byte[] key; byte[] iv; _GetKeyAndInitialVector(password, salt, out key, out iv); CryptoStream cryptoStream = new CryptoStream(outputStream, _cryptoProvider.CreateEncryptor(key, iv), CryptoStreamMode.Write); long totalBytesRead = 0; int bytesRead; byte[] buffer = new byte[_bufferSize]; do { bytesRead = inputStream.Read(buffer, 0, _bufferSize); totalBytesRead += bytesRead; if (maxInputStreamSize > 0 && totalBytesRead > maxInputStreamSize) throw new Exception("Input stream has exceeded maximum allowed size of " + maxInputStreamSize); cryptoStream.Write(buffer, 0, bytesRead); } while (bytesRead > 0); cryptoStream.FlushFinalBlock(); } public delegate CryptoStream CryptoStreamFactory(Stream inputStream, ICryptoTransform transform, CryptoStreamMode mode); public Stream DecryptStream(Stream inputStream, string password, string salt, CryptoStreamFactory factory) { byte[] key; byte[] iv; _GetKeyAndInitialVector(password, salt, out key, out iv); if (factory == null) return new CryptoStream(inputStream, _cryptoProvider.CreateDecryptor(key, iv), CryptoStreamMode.Read); return factory(inputStream, _cryptoProvider.CreateDecryptor(key, iv), CryptoStreamMode.Read); } private void _GetKeyAndInitialVector(string password, string salt, out byte[] key, out byte[] iv) { byte[] saltBytes = Encoding.Unicode.GetBytes(salt); Rfc2898DeriveBytes pdb = new Rfc2898DeriveBytes(password, saltBytes); key = pdb.GetBytes(_cryptoProvider.LegalKeySizes[0].MaxSize / 8); iv = pdb.GetBytes(_cryptoProvider.BlockSize / 8); } } }

    Please let me know if you need more details.

    Thanks,
    Amar
    amar
    Wednesday, May 13, 2009 1:36 AM

All replies



  • cryptoStream.Write(buffer, 0, bytesRead);
    cryptoStream.Flush();


    Try adding cryptoStream.Flush(); , seems to me that he FlushFinalBlock is flushing all the data ( 300mb in one time ) and its not liking it...
    Wednesday, May 13, 2009 7:34 AM
  • Yes, this won't work on a 32-bit operating system.  The most memory in one clean contiguous chunk you can hope to allocate is around 500 MB.  It doesn't have anything to do with the amount of RAM you've got, it is caused by fragmentation of the virtual memory space for your program.

    There is never a need to read the full content of the file, using chunks of 4096 bytes is enough.  You are already doing that with the _bufferSize member.  Pass a FileStream to your EncryptStream method.

    Hans Passant.
    Wednesday, May 13, 2009 10:19 AM
    Moderator
  • Thanks Hans.

    MemoryStream memStream = new MemoryStream((int)new FileInfo(cFilename).Length);

    Couple of questions:

    In this case, I am apparently allocating 300 MB (file size). Do you expect it to still not work? If yes, then why is it only failing at the point of FlushFinalBlock() and why not at the point where the actual encryption happens in chunks?

    We are implementing a tool for uploading\downloading our Data over the Cloud (Amazon S3) and we really need to use MemoryStream as we will be dealing with zillions of files with all different type of file size. FileStream won't be idealistic solution in this case. Is there anything else we can do in order to support Symmetric Encryption through MemoryStream for any file size? Please note, Asymmetric Algorithm is not supported in this case.

    Thanks,
    Amar
    amar
    Wednesday, May 13, 2009 5:17 PM
  • I did read your code wrong, it is the output stream that is causing the OOM.  The MemoryStream in your first snippet is never used, it just takes up needless space.

    Cold hard fact is that you are pushing what's possible on a 32-bit operating system.  The problem is not the amount of RAM you may have, it is the need for a chunk of contiguous memory.  That's hard to come by, address space is fragmented into chunks of code and data.  A 32-bit program usually OOMs around 500 MB.  Which is relevant to your file size, once the buffer fills up to 256MB it will make a new allocation, double the size, to grow the memory buffer.  That goes kaboom.

    It is otherwise a very solved problem, 64-bit operating systems don't have this problem.

    Hans Passant.
    • Proposed as answer by Marco Zhou Thursday, May 14, 2009 3:52 AM
    Wednesday, May 13, 2009 6:38 PM
    Moderator
  • Hi Dirksma,

    I don't think cryptoStream.Flush() will encrypt the data correctly. Also FlushFinalBlock() can be called only once.

    Thanks,
    Amar
    amar
    Wednesday, May 13, 2009 6:39 PM
  • Hello Amar,

    I'm encoutering exactly the same problem as you.

    Has the problem be solved ?
    If yes, how did you do ?

    Thanks a lot for answering.

    Sylvain
    Wednesday, March 17, 2010 2:12 PM
  • Hi,

     

    Will you please provide solution for this issue, I am too trying to Insert 10,000 Char's in Sql server 2005 having NVARCHAR(MAX) as datatype.

    This is running for 3000 CHAR's but when I incress the legnth insert's only 38k cHAR'S WITH ENCRIPTION AND when retriew it geting exceoption as -- System.Security.Cryptography.CryptographicException: Bad Data.

     

    if (String.IsNullOrEmpty(inputValue))

     

    return null;

     

    string result = inputValue;

     

    TripleDESCryptoServiceProvider provider = null;

     

    if (iv != null && key != null)

    provider = GetCryptoProvider(iv, key);

     

    else
     

    provider = GetCryptoProvider();

     

    //Convert to base64
     

     

    byte[] inputEquivalent = null;

     

    try
     

    {

    inputEquivalent =

    Convert.FromBase64String(inputValue);

    }

     

    catch (Exception ex)

    {

     

    LogHelper.LogDbError("CCS FromBase64String", null, null, ex.Message);

     

    throw;

    }

     

    if (inputEquivalent != null && inputEquivalent.Length > 0) // Converted OK
     

    {

     

    try
     

    {

     

    // Create a new MemoryStream.
     

     

    MemoryStream msDecrypt = new MemoryStream();

     

    // Create a CryptoStream using the MemoryStream
     

     

    // and the passed key and initialization vector (IV).
     

     

    CryptoStream csDecrypt = new CryptoStream(msDecrypt,

    provider.CreateDecryptor(),

     

    CryptoStreamMode.Write);

    csDecrypt.Write(inputEquivalent, 0, inputEquivalent.Length);

    csDecrypt.FlushFinalBlock();

    csDecrypt.Close();

    result =

    new UTF8Encoding().GetString(msDecrypt.ToArray());

    }

     

    catch (Exception ex)

    {

     

    LogHelper.LogDbError("CCS CryptoStream", null, null, ex.Message);

     

    throw;

    }

    }

     

    //Convert the buffer into a string and return it.
     

     

    return result;

      

    ///

     

     

     

     

     

    ///

     

     

     

    ///

     

     

     

    /// <param name="inputValue">The input value.

     

     

     

    ///

     

     

     

    public static string GetEncryptedValue(string inputValue, string iv, string

    key)

     

    if (String

    .IsNullOrEmpty(inputValue))

     

    return null

    ;

     

    TripleDESCryptoServiceProvider provider = null

    ;

     

    if (iv != null && key != null

    )

     

     

     

    provider = GetCryptoProvider();

     

     

     

     

    MemoryStream mStream = new MemoryStream

    ();

     

     

     

     

     

     

     

    CryptoStream cStream = new CryptoStream(mStream, provider.CreateEncryptor(), CryptoStreamMode

    .Write);

     

     

     

     

    byte[] toEncrypt = new ASCIIEncoding

    ().GetBytes(inputValue);

     

     

     

    cStream.Write(toEncrypt, 0, toEncrypt.Length);

    cStream.FlushFinalBlock();

     

     

     

     

     

     

     

     

     

     

    byte

    [] ret = mStream.ToArray();

     

     

     

    cStream.Close();

    mStream.Close();

     

     

     

     

    return Convert

    .ToBase64String(ret);

     

     

    }

    

     

     

     <summary>

     

    ///

     

     

     

    ///

     

     

     

    ///

     

     

     

    private static TripleDESCryptoServiceProvider

    GetCryptoProvider()

     

    string iv = "SuFjcEmp/TE="

    ;

     

    if (ConfigHelper.AppSettingExists("Crypto_IV"

    ))

     

    ConfigHelper.GetAppSetting("IV"

    );

     

    string key = "KIPSToILGp6fl+3gXJvMsN4IajizYBBT"

    ;

     

    if (ConfigHelper.AppSettingExists("Crypto_Key"

    ))

     

    ConfigHelper.GetAppSetting("Crypto_Key"

    );

     

    TripleDESCryptoServiceProvider provider = new TripleDESCryptoServiceProvider

    ();

     

    Convert

    .FromBase64String(iv);

     

    Convert

    .FromBase64String(key);

     

    return

    provider;

     

     

     

    }

    

     

    Appreciate you help for quick response.

    Thanks and Regard's

    Ali

     

     

     

     

     

     

     

     

     

     

     

    {

     

     

    <returns></returns>

     

    </summary>

     

    Gets the crypto provider.

     

    // Return the encrypted buffer.

     

    // Close the streams.

     

     

    // encrypted data.

     

    // MemoryStream that holds the

     

    // Get an array of bytes from the

     

    // Write the byte array to the crypto stream and flush it.

     

     

    // Convert the passed string to a byte array.

     

     

    // and the passed key and initialization vector (IV).

     

    // Create a CryptoStream using the MemoryStream

     

     

    // Create a MemoryStream.

     

    else

     

    provider = GetCryptoProvider(iv, key);

     

     

     

     

     

    {

     

     

    <returns></returns>

     

    </param>

     

    </summary>

     

    Gets the encrypted value.

     

    <summary>

     

    Bad Data.

    Description:

    Line 162:                {
    Line 163:                    LogHelper.LogDbError("CCS CryptoStream", null, null, ex.Message);
    Line 164:                    throw;
    Line 165:                }
    Line 166:            }
    An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

    Exception Details: System.Security.Cryptography.CryptographicException: Bad Data.


    Source Error:


    Source File: C:\Baseline Code\CCS\Contigo.Jazz.Core\Helper\EncryptHelper.cs Line: 164

    Stack Trace:

     

    [CryptographicException: Bad Data.
    ]
       System.Security.Cryptography.CryptographicException.ThrowCryptogaphicException(Int32 hr) +33
       System.Security.Cryptography.Utils._DecryptData(SafeKeyHandle hKey, Byte[] data, Int32 ib, Int32 cb, Byte[]& outputBuffer, Int32 outputOffset, PaddingMode PaddingMode, Boolean fDone) +0
       System.Security.Cryptography.CryptoAPITransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount) +313
       System.Security.Cryptography.CryptoStream.FlushFinalBlock() +33
       System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing) +124
       System.IO.Stream.Close() +17
       Contigo.Jazz.Core.EncryptHelper.GetDecryptedValue(String inputValue, String iv, String key) in C:\Baseline Code\CCS\Contigo.Jazz.Core\Helper\EncryptHelper.cs:164
       Contigo.Jazz.Core.EncryptHelper.GetDecryptedValue(String inputValue) in C:\Baseline Code\CCS\Contigo.Jazz.Core\Helper\EncryptHelper.cs:111
       Contigo.CCS.BLL.ActivityDAL.GetByFilterActivity(Int32 caseId, Int32 customerId, String activityType, String activitySubtype, String assignedType, Int32 assignedToId, Nullable`1 startActionRequiredBy, Nullable`1 endActionRequiredBy, Nullable`1 startActivityAt, Nullable`1 endActivityAt, String searchFor) in C:\Baseline Code\CCS\Contigo.CCS.BLL\DAL\ActivityDAL.cs:105
       Contigo.CCS.BLL.ActivityMgr.GetByFilterActivity(Int32 caseId, Int32 customerId, String activityType, String activitySubtype, String assignedType, Int32 assignedToId, Nullable`1 startActionRequiredBy, Nullable`1 endActionRequiredBy, Nullable`1 startActivityAt, Nullable`1 endActivityAt, String searchFor) in C:\Baseline Code\CCS\Contigo.CCS.BLL\Mgr\ActivityMgr.cs:69
       Contigo.CCS.UI.CaseDetail.PopulateActivity(Int32 caseId) in C:\Baseline Code\CCS\CCS\Case\CaseDetail.aspx.cs:172
       Contigo.CCS.UI.CaseDetail.PopulateForm(Int32 id) in C:\Baseline Code\CCS\CCS\Case\CaseDetail.aspx.cs:232
       Contigo.CCS.UI.CaseDetail.Page_Load(Object sender, EventArgs e) in C:\Baseline Code\CCS\CCS\Case\CaseDetail.aspx.cs:52
       System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +14
       System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +35
       System.Web.UI.Control.OnLoad(EventArgs e) +99
       System.Web.UI.Control.LoadRecursive() +50
       System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +627

    OS windoes server 2003 SE R2 with 32-bit

     

     

    provider.Key =

    provider.IV =

     

    key =

     

     

    iv =

     

     

    ///

     

     

     

     

     

     

     



    Please find below error message for ref - 
    • Edited by ALBSHAH Wednesday, September 7, 2011 9:14 AM
    Tuesday, September 6, 2011 1:42 PM