none
Bitmap Memory Issues RRS feed

  • Question

  • My application reads a file (normally plain text), then compresses it, then creates a bitmap from the compressed data. Up to a file size of about 30 MB of text (approximately 15 MB compressed) it works just as expected. At 40 MB of text (20 MB compressed) I get an exception when creating the new bitmap.

    Source is 30 MB Text File, compressed to approximately 15 MB
    Image Height 2943
    Image Width  5232
    pixels = 15397776
    No Error
    
    
    Source is 40MB Text file, compressed to approximately 20 MB
    ImageHeight	3402	Integer
    ImageWidth	6048	Integer
    pixels = 20575296
    Offending Line - MainBitmap = New Bitmap(ImageWidth, ImageHeight)
    System.ArgumentException was unhandled
    HResult=-2147024809
    Message=Parameter is not valid.
    Source=System.Drawing
    
    I found this comment from NoBugz here

    Yeah, you're pushing the envelope a bit with this bitmap.  You'll need 10000 x 9000 x 4 = 360 MB of contiguous virtual memory space for it.  I had no problem allocating it myself within the debugger, 1 GB Ram, 2.5 GB swap.  But you're probably allocating some other large buffer too to process the PDF.  Or are bringing in some DLL whose preferred load address fragments the virtual memory space.  DependencyWalker can help you diagnose that.
    Note that you can limit the allocated size of the bitmap by choosing a more conservative pixel format.  Format24bppRgb will need 270 MB.

    I am nowhere that 360 MB so I am wondering what I am doing wrong, if anything, and how to resolve it. I am using 32 bbp and it would be inconvenient but possible to switch to 24 bbp

    Tuesday, October 24, 2017 5:04 PM

All replies

  • I don't have a problem with this. It saves a bmp ok. 13 mb

    Dim bmp As New Bitmap(6048, 3402)

    Using img As Image = Image.FromFile("C:\bitmaps\rusty.jpg"), g As Graphics = Graphics.FromImage(bmp) g.DrawImage(img, New Rectangle(0, 0, bmp.Width, bmp.Height)) End Using bmp.Save("c:\bitmaps\testsize.bmp") bmp.Dispose()


    Tuesday, October 24, 2017 5:40 PM
  • I don't have a problem with this. It saves a bmp ok. 13 mb

    Dim bmp As New Bitmap(6048, 3402)

    Using img As Image = Image.FromFile("C:\bitmaps\rusty.jpg"), g As Graphics = Graphics.FromImage(bmp) g.DrawImage(img, New Rectangle(0, 0, bmp.Width, bmp.Height)) End Using bmp.Save("c:\bitmaps\testsize.bmp") bmp.Dispose()


    I tried to somewhat reproduce what happens in my code, but it does not error. 

    Imports System.Security.Cryptography
    Public Class Form1
        Dim Rnd As New Random
        Dim Arr((50 * 1024 * 1024) - 1) As Byte ' this simulates the 50 MB text file being read 
        Dim Cols As New List(Of Color) ' I actually create this list AFTER creating the bitmap
        Private Sub Form1_Load(sender As System.Object, e As System.EventArgs) Handles MyBase.Load
            Using rng As New RNGCryptoServiceProvider
                rng.GetBytes(Arr)
            End Using
            For index As Integer = 0 To 2560 * 3 Step 3
                Cols.Add(Color.FromArgb(255, Arr(index), Arr(index + 1), Arr(index + 2)))
            Next
        End Sub
    
        Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click
            Dim MYBMap As New Bitmap(6048, 3402)
            For r As Integer = 0 To 3401
                For c As Integer = 0 To 6047
                    MYBMap.SetPixel(c, r, Cols(Rnd.Next(0, Cols.Count)))
                Next
            Next
            MYBMap.Save("d:\MYBmap.png", System.Drawing.Imaging.ImageFormat.Png)
            MessageBox.Show("Done")
        End Sub
    End Class
    

    Tuesday, October 24, 2017 9:05 PM
  • LOL. Wild guess, are you doing this in a loop and not disposing the bitmaps? Maybe you are running out of memory.

    Tuesday, October 24, 2017 9:49 PM
  • LOL. Wild guess, are you doing this in a loop and not disposing the bitmaps? Maybe you are running out of memory.

    No, the user starts by either opening a text file or creating / typing a password. Once both are done, the user selects "Encrypt" and the process begins:
    1 - The text file contents (stored as an array of bytes) is compressed to another byte array using either GZIP or Deflate compression.
    2 - The compressed array size is translated to a Bitmap Width and Height with approximate Aspect Ratios as the user chooses (16:9, 4:3, 1:1), and an empty bitmap is created**
    3 - For each possible byte in the array (0..255), multiple unique colors are generated and added to a list (of color), using the SHA Hash of the character and the password concatenated in a fancy manner.
    4 - for each byte in the text file contents, the bitmap pixel is set to one of the colors that corresponds to that byte.
    5 - When finished, the user can save the bitmap as a .png file (lossless)

    Step two is where the exception occurs

    There are several for-next loops, determining the Image size needed, setting Pixels, creating the List of Colors but none involving bitmaps. It is created, filled and saved. I do not dispose of it but this error happens on the first run and is size related only as far as I can tell. I can encrypt, save, and decrypt as many smaller files one after the other as I want without closing the program.

    Here's the Memory Profile annotated.

    • Edited by Devon_Nullman Tuesday, October 24, 2017 10:45 PM Added Memory Profile
    Tuesday, October 24, 2017 10:30 PM
  •  The way i understand it,  the largest bitmap you can create in your app will depend on the largest continuous block of virtual memory that is available as Hans Passant said in This Thread.  Keep in mind that the memory can get a little fragmented as you create and release different variables throughout your code too,  similar to a hard drive getting fragmented from creating and deleting files.

     Also, as nobugz said,  "... But you're probably allocating some other large buffer too ..." such as your large byte arrays.  As far as creating the List(Of Color) after creating the bitmap as you commented in your code,  i am not positive but,  i believe that a List of colors probably would not require a continuous block of memory as a Bitmap would.

     Creating a new Bitmap without disposing it can raise hell with it too if the GC does not collect them fast enough.


    If you say it can`t be done then i`ll try it

    • Edited by IronRazerz Tuesday, October 24, 2017 10:38 PM
    Tuesday, October 24, 2017 10:36 PM
  • The Bitmap is declared at Form level just as a Bitmap, then set as a new bitmap once the user chooses to encrypt a bunch of text. At that point The Memory Profile details show the 50 MB array as 52 MB and the compressed as 25. Memory usage spikes very high (900 MB) when the text is loaded, then settles back down to 80 MB or so and stays there. It rises a small amount just before the exception. Creating the List Of Color makes no visible spike in memory usage. 

    The profile image I posted above includes creating those lists after the text is loaded.


    Tuesday, October 24, 2017 11:01 PM
  • The Bitmap is declared at Form level just as a Bitmap, then set as a new bitmap once the user chooses to encrypt a bunch of text. At that point The Memory Profile details show the 50 MB array as 52 MB and the compressed as 25. Memory usage spikes very high (900 MB) when the text is loaded, then settles back down to 80 MB or so and stays there. It rises a small amount just before the exception. Creating the List Of Color makes no visible spike in memory usage. 

    The profile image I posted above includes creating those lists after the text is loaded.


    So you start the app, save one file and get the error? Or after 5 files? Can you duplicate the pattern using the same files?

    What is the big bump and cliff in your graph on the left? Load test the what? Bitmap? Text?

    Tuesday, October 24, 2017 11:42 PM
  • Deven,

     Not sure if you where replying to me but,  what i am saying is that you might actually have more memory than is needed but,  by the time you try creating the New Instance of the bitmap,  the memory may very well be fragmented and there is not a continuous block of memory (all together in a row) that is large enough for a bitmap of that size.  That would require a continuous block of memory that is (6048 * 3402 * 4) = 82301184 bytes (78.49 Mb) in length.


    If you say it can`t be done then i`ll try it

    Wednesday, October 25, 2017 12:42 AM
  • Deven,

     Not sure if you where replying to me but,  what i am saying is that you might actually have more memory than is needed but,  by the time you try creating the New Instance of the bitmap,  the memory may very well be fragmented and there is not a continuous block of memory (all together in a row) that is large enough for a bitmap of that size.  That would require a continuous block of memory that is (6048 * 3402 * 4) = 82301184 bytes (78.49 Mb) in length.


    If you say it can`t be done then i`ll try it


    Hi Razerz!

    Can one check and or allocate the memory chunk and then write and save with lockbits or something? Just to see if that is the problem maybe?

    Not hard to check just make bitmaps increasing in size until it blows?

    Wednesday, October 25, 2017 1:16 AM
  • Hi Tom  8)

     I have not tried it but,  maybe it would work.  However,  you would need to know the size of the bitmap beforehand.  I did read something in one of the links earlier where they suggested trying to create all the larger buffers and what not at startup,  then creating the bitmap.  Sounded like basically the same idea but,  in revers.  Still would need to know the sizes of all them beforehand too though.  I'm off to ZZZzzzz land now but,  maybe i will test it a little tomorrow if one of you two don't test it by then.

     I also see that Hans suggested in the link from my second post back to try using the Windows Imaging Component to overcome this problem with large images.  Don't know if that would fit in with the recent things i have seen Devon experimenting with in his last few questions though.  8)  


    If you say it can`t be done then i`ll try it

    • Edited by IronRazerz Wednesday, October 25, 2017 2:07 AM
    Wednesday, October 25, 2017 2:04 AM
  • So you start the app, save one file and get the error? Or after 5 files? Can you duplicate the pattern using the same files?

    What is the big bump and cliff in your graph on the left? Load test the what? Bitmap? Text?

    I can start the app and process a seemingly arbitrary number of text files as long as they are less than 25 MB, it never fails. I can mix different sizes, I can try 50 MB files that fail then back to smaller files.

    I can start the app and try a 30, 40 or 50 MB file and it fails. A 30 MB file will rarely succeed, 40 and 50, never.

    The bump and cliff is "Load Text", not "Load Test", why it spikes so high and why it settles back down I have no idea, that was from loading a 50MB test file.

    Wednesday, October 25, 2017 3:50 AM
  • So you start the app, save one file and get the error? Or after 5 files? Can you duplicate the pattern using the same files?

    What is the big bump and cliff in your graph on the left? Load test the what? Bitmap? Text?

    I can start the app and process a seemingly arbitrary number of text files as long as they are less than 25 MB, it never fails. I can mix different sizes, I can try 50 MB files that fail then back to smaller files.

    I can start the app and try a 30, 40 or 50 MB file and it fails. A 30 MB file will rarely succeed, 40 and 50, never.

    The bump and cliff is "Load Text", not "Load Test", why it spikes so high and why it settles back down I have no idea, that was from loading a 50MB test file.

    When I use this example in a loop I went to 10,000 x 20,000 bitmap size and at 20 it made a 147 mb file but no error.

    When I remove the dispose I get to 10 or 11 and then throw the error.

    Are you disposing or not? You did not in your example?

    PS The No. 10 (10000 x 10000) file size where it errs is about 80mb when it is made when disposing.

    PS2: maybe you need to dispose your lists/arrays you make in the Load Text then declare the bitmap. Maybe the memory you use in your calc is still hanging and the total you are using is over the limit somehow. Try disposing everything you make then declare the bitmap???

    How big do your arrays get? Or whatever you are using.

    PS3 how hard is it to change your app to just do the same file in a loop like this until it errs? You say when you hit 30 mb it errs. So do a 10mb in a loop similar to this until it errs ie 10 times.

    Public Class Form5
        Private Sub Form5_Load(sender As Object, e As EventArgs) Handles Me.Load
            Using img As Image = Image.FromFile("C:\bitmaps\rusty.jpg")
    
                For i As Integer = 1 To 20
                    Dim bmp As New Bitmap(10000, i * 1000)
    
                    Using g As Graphics = Graphics.FromImage(bmp)
                        g.DrawImage(img, New Rectangle(0, 0, bmp.Width, bmp.Height))
                    End Using
    
                    bmp.Save("c:\test\test" & i.ToString & ".bmp")
    
                    bmp.Dispose()
    
                Next
            End Using
        End Sub
    End Class




    Wednesday, October 25, 2017 10:27 AM
  • Are you disposing or not? You did not in your example?
    No, I do not dispose of anything, I suppose I could dispose of the bitmap once it is saved, but certainly not before, but to me that doesn't explain why I can compress, encrypt and save 5 or more 5,10,15,20 MB files one after another with no issues.

    PS2: maybe you need to dispose your lists/arrays you make in the Load Text then declare the bitmap. Maybe the memory you use in your calc is still hanging and the total you are using is over the limit somehow. Try disposing everything you make then declare the bitmap???
    When the text is loaded, one array is created and it is the size of the text file.
    When a password is typed or generated, two color dictionaries are made, each is 3000 colors, one is for encrypting (Integer,Color), the other for decrypting(Color, Integer). That could be changed to only make the dictionary needed for the pending operation. When the "Encrypt" button is clicked, the original array from the text file is compressed to another array, typically about half the size of the original. That compressed array is used to calculate the smallest bitmap dimensions that will hold at least every byte in the compressed array, the bitmap size is always slightly more capacity than the compressed array. When I actually fill the bitmap, I create another array of random numbers exactly the correct length for the bitmap and use Array.copy to insert the compressed data into it at index 0. So again, I suppose that once the compressed array is created, the original array is no longer needed until another file is loaded, or another password is used to encrypt the same data again. Also, once the array to fill the bitmap is created and filled, the compressed array is no longer needed. I have nothing in place to prevent re-compressing the original even though it already may have been compressed by a previous operation. I'm not even sure how to dispose of an array and still leave it useable later.

    How big do your arrays get? Or whatever you are using.
    The "original" array is the size of the text file, using File.ReadAllBytes.
    The compressed array is around half the size of the "original" array.
    The two Dictionaries are 3000 entries each

    PS3 how hard is it to change your app to just do the same file in a loop like this until it errs? You say when you hit 30 mb it errs. So do a 10mb in a loop similar to this until it errs ie 10 times.

    If you mean to create a test that automates loading a smaller file over and over, I could do that. Last night I manually ran through around 20 cycles, 1MB, 30, 5, 10, 40, 20, 50, back to smaller files at random and it only failed on the 30, 40 and 50, and after a fail, a smaller file succeeded. I created another program that just makes text files of whatever size you want using several lists of words gathered from dictionaries, Crossword Puzzle sites, Scrabble sites, etc.

    I have probably run this through 500 files in testing and I have seen two times where after trying a big file that fails, and then trying a file that should be fine, an anomaly occurs - the process finishes in just a few milliseconds (It generally takes 700 milliseconds per megabyte), and the resulting image file is way way too small (like 30 KB for a 5 MB text file). That I have no explanation for at all.

    Wednesday, October 25, 2017 2:18 PM
  • Devon,

    "but to me that doesn't explain why I can compress, encrypt and save 5 or more 5,10,15,20 MB files one after another with no issues."

    Yes I suppose. I dont really know. It takes time for the memory to move sometimes maybe. GC is not instant maybe 30 secs for graphics I have seen.

    It maybe one of those do what it takes to get it to run things.

    If you get it to work then perhaps what you did will point to what the problem is. For example if you dispose your dag-nabit bitmap and it starts to work then you know what book to look in for your answer at least.

    Still looking at the rest of your nice response.

    :)

    PS In my example I create 5, 10, ... and at 80 mb it throws the error when I don't dispose. When I dispose I got to 150mb or so no problems. So one  little .dispose may solve your problem!

    Or not...

    Wednesday, October 25, 2017 4:41 PM
  • Devon,

    "but to me that doesn't explain why I can compress, encrypt and save 5 or more 5,10,15,20 MB files one after another with no issues."

    Yes I suppose. I dont really know. It takes time for the memory to move sometimes maybe. GC is not instant maybe 30 secs for graphics I have seen.

    It maybe one of those do what it takes to get it to run things.

    If you get it to work then perhaps what you did will point to what the problem is. For example if you dispose your dag-nabit bitmap and it starts to work then you know what book to look in for your answer at least.

    Still looking at the rest of your nice response.

    :)

    PS In my example I create 5, 10, ... and at 80 mb it throws the error when I don't dispose. When I dispose I got to 150mb or so no problems. So one  little .dispose may solve your problem!

    Or not...

    On the encrypt side, I added a .dispose just after saving the bitmap, as well as on the decrypt side, just after taking the pixels from the bitmap and converting them to an array, but it made no difference. I also copied the entire Solution to my 64 bit computer and found a whole new issue. On the x64 box I can encrypt a file as big as 150MB, but when I encrypt anything over 50 MB and decrypt , the result is missing approximately 5% of the original file (at the end). This is crazy.

    Here are the Form-Level Variables:

        Const ColorsPerCharacter As Integer = 4
        Const BytesPerColor As Integer = 3
        Private LastError As String = ""
        Private Misses As Integer = 0
        Private PWStrength As Integer = 0
        Private RNG As New Random
        Private SWatch As New Stopwatch
        Private Delay As Integer = 0
        Private EncryptColorDict As New Dictionary(Of Integer, Color) 'has 256 * ColorsPerCharacter Entries
        Private DecryptColorDict As New Dictionary(Of Color, Integer) 'has 256 * ColorsPerCharacter Entries
        Private MainBitmap As Bitmap
        Private EncryptKey As String '8 to 32 chars
        Private DecryptKey As String '8 to 32 chars
        Private PTData() As Byte 'Depends on size of text file loaded
        Private GZData() As Byte 'Usually 50% of PTData
        Private DecryptedData() As Byte 'Should wind up as same size as PTData
        Private ImageWidth As Integer
        Private ImageHeight As Integer
        Private WMult As New List(Of Integer) '4 entries
        Private HMult As New List(Of Integer) '4 entries
        Private KeyLength As Integer = 0
        Private RSAPublicKey As String 'can be several hundred chars
        Private RSAPrivateKey As String 'bigger than Public Key but under 1K
        Private KeySizes As New List(Of Integer) '8 entries
        Private EncodingToUse As Encoding
        Private EncNames As New List(Of String) ' 4 entries
        Private EncTypes As New List(Of Encoding) '4 entries

    Could any of these be an issue? Most just have to be Accessible from anywhere. I also have a few of these:

        Private Function DeflateCompress(ByVal raw() As Byte) As Byte()
            Using memory As MemoryStream = New MemoryStream()
                Using Deflate As DeflateStream = New DeflateStream(memory, CompressionMode.Compress, True)
                    Deflate.Write(raw, 0, raw.Length)
                End Using
                Return memory.ToArray()
            End Using
        End Function
    The 'End Using' never gets executed ????

    Friday, October 27, 2017 12:16 AM
  • Quick Update - I converted the Forms App to a Console App and it breezes through a 100MB source file with no issues at all.

    Remaining is to create a somewhat complex parser for Command line Args.

    Update - 200 MB OK also, took 3 minutes
    Sunday, October 29, 2017 2:14 AM
  • Hi Devon_Nullman,

    Have you solved your issue now? If yes, please remember to close your thread by marking the helpful post as answer, it is beneficial to other community members who face the same issue.

    Thanks for your understanding.

    Best Regards,

    Cherry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, October 31, 2017 6:37 AM
    Moderator
  • Hi Devon_Nullman,

    Have you solved your issue now? If yes, please remember to close your thread by marking the helpful post as answer, it is beneficial to other community members who face the same issue.

    Thanks for your understanding.

    Best Regards,

    Cherry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Cherry - No, I can't really say that the issue has been solved, I still would like to be able to accomplish my goal using a Forms application or at least know why I can't. Using the exact same code converted to a Console App makes me think it's something in how much memory forms use or how it's managed.
    Thursday, November 2, 2017 2:49 PM



  • Cherry - No, I can't really say that the issue has been solved, I still would like to be able to accomplish my goal using a Forms application or at least know why I can't. Using the exact same code converted to a Console App makes me think it's something in how much memory forms use or how it's managed.

    Devon,

    There is a big difference between a Console and a Windows Forms application. A good Forms application is based on OOP, a console application is static. 

    It means that the Windows Forms application needs to release its not yet released memory more. Like more things there are more ways to do that. 

    I tested your kind of problem a long time ago. I recognized that if I forced a windows paint (in fact giving the process to the video adapter) in long processes without painting the GC got time to clean up the still not released space. Be aware, calling the method dispose of components does not much in this, although it helps the process to earlier find memory which can be released. 


    Success
    Cor




    Friday, November 3, 2017 6:38 AM



  • Cherry - No, I can't really say that the issue has been solved, I still would like to be able to accomplish my goal using a Forms application or at least know why I can't. Using the exact same code converted to a Console App makes me think it's something in how much memory forms use or how it's managed.

    Devon,

    There is a big difference between a Console and a Windows Forms application. A good Forms application is based on OOP, a console application is static. 

    It means that the Windows Forms application needs to release its not yet released memory more. Like more things there are more ways to do that. 

    I tested your kind of problem a long time ago. I recognized that if I forced a windows paint (in fact giving the process to the video adapter) in long processes without painting the GC got time to clean up the still not released space. Be aware, calling the method dispose of components does not much in this, although it helps the process to earlier find memory which can be released. 


    Success
    Cor




    Cor - I don't I ever do anything that forces a Paint. Here's basically the order of things that happen:
    1 - The user selects a text file and a password, the order does not matter but both must be selected to proceed.
    2 - When the encrypting starts, a dictionary of Integer,Color is created and filled with 1024 integers and 1024 Colors.
    3 - The text file is read into an array of bytes.
    4 - That array is compressed using .NET Deflate Compression.
    5 - The smallest bitmap dimensions are found that has more pixels than the compressed array
    6 - That bitmap is created using those dimensions
    7 - each pixel in the bitmap is set to a color based or the value of the byte in the compressed array
    8 - The bitmap is saved

    Saturday, November 4, 2017 11:02 PM
  • Devon,

    That paint is only to give the process to another processor, in this case the video processor. 

    Now the computer processor has time to  starts its garbage collection process. 

    It could have been any other processor you've in your computer. 

    This was a solution some 14 years ago, I don't know if the GC process is now optimized that there is no need anymore for it. I thought about it when you wrote about the difference between Console en Form app.


    Success
    Cor



    Sunday, November 5, 2017 9:33 AM
  • Devon,

    That paint is only to give the process to another processor, in this case the video processor. 

    Now the computer processor has time to  starts its garbage collection process. 

    It could have been any other processor you've in your computer. 

    This was a solution some 14 years ago, I don't know if the GC process is now optimized that there is no need anymore for it. I thought about it when you wrote about the difference between Console en Form app.


    Success
    Cor



    What about a specific bit of code like this:

        Private Function DeflateCompress(ByVal raw() As Byte) As Byte()
            Using memory As MemoryStream = New MemoryStream()
                Using Deflate As DeflateStream = New DeflateStream(memory, CompressionMode.Compress, True)
                    Deflate.Write(raw, 0, raw.Length)
                End Using
                Return memory.ToArray()
            End Using
        End Function
        
        CompressedData = DeflateCompress(UncompressedData)
    
    

    In this line (The BOLD) - once it executes, the UncompressedData Byte Array is no longer needed, for a 50 MB file, the UncompressedData is 50MB and the CompressedData is around half that size. Should I call UncompressedData = Nothing and possibly GC.Collect ? I cannot call dispose, Byte Array does not support that.

    A similar thing happens when I make a Temporary array that is the exact size for the Bitmap, fill it with random Numbers and copy the CompressedData to it.

            Dim TotalPixels As Integer = MainBitmap.Width * MainBitmap.Height
            Dim TempArray(TotalPixels - 1) As Byte
            Using CRNG As New RNGCryptoServiceProvider
                CRNG.GetNonZeroBytes(TempArray)
            End Using
            Array.Copy(CompressedData, 0, TempArray, 0, CompressedData.Length)
    

    Now the CompressedData Array is no longer needed.

    Monday, November 6, 2017 2:56 AM

  • What about a specific bit of code like this:

        Private Function DeflateCompress(ByVal raw() As Byte) As Byte()
            Using memory As MemoryStream = New MemoryStream()
                Using Deflate As DeflateStream = New DeflateStream(memory, CompressionMode.Compress, True)
                    Deflate.Write(raw, 0, raw.Length)
                End Using
                Return memory.ToArray()
            End Using
        End Function
        
        CompressedData = DeflateCompress(UncompressedData)
    

    In this line (The BOLD) - once it executes, the UncompressedData Byte Array is no longer needed, for a 50 MB file, the UncompressedData is 50MB and the CompressedData is around half that size. Should I call UncompressedData = Nothing and possibly GC.Collect ? I cannot call dispose, Byte Array does not support that.

    Now the CompressedData Array is no longer needed.

    I personally would first try to use the getBuffer method. You create currently an extra immutble array. Which exist as long as CompressedData exist. 

    https://msdn.microsoft.com/en-us/library/system.io.memorystream.getbuffer(v=vs.110).aspx


    Success
    Cor


    Monday, November 6, 2017 12:17 PM
  •     CompressedData = DeflateCompress(UncompressedData)

    In this line (The BOLD) - once it executes, the UncompressedData Byte Array is no longer needed, for a 50 MB file, the UncompressedData is 50MB and the CompressedData is around half that size. Should I call UncompressedData = Nothing and possibly GC.Collect ? I cannot call dispose, Byte Array does not support that.

    How about:

        UncompressedData= DeflateCompress(UncompressedData)

    or:

       DeflateCompress(UncompressedData)

        Private Sub DeflateCompress(Byref raw() As Byte)

                  raw = deflate

    Monday, November 6, 2017 1:29 PM
  • No, the user starts by either opening a text file or creating / typing a password. Once both are done, the user selects "Encrypt" and the process begins:
    1 - The text file contents (stored as an array of bytes) is compressed to another byte array using either GZIP or Deflate compression.
    2 - The compressed array size is translated to a Bitmap Width and Height with approximate Aspect Ratios as the user chooses (16:9, 4:3, 1:1), and an empty bitmap is created**
    3 - For each possible byte in the array (0..255), multiple unique colors are generated and added to a list (of color), using the SHA Hash of the character and the password concatenated in a fancy manner.
    4 - for each byte in the text file contents, the bitmap pixel is set to one of the colors that corresponds to that byte.
    5 - When finished, the user can save the bitmap as a .png file (lossless)

    Step two is where the exception occurs

    There are several for-next loops, determining the Image size needed, setting Pixels, creating the List of Colors but none involving bitmaps. It is created, filled and saved. I do not dispose of it but this error happens on the first run and is size related only as far as I can tell. I can encrypt, save, and decrypt as many smaller files one after the other as I want without closing the program.

    Here's the Memory Profile annotated.

    Hi,

    in my opinion - from your picture - this has nothing to do with having enough memory or not, it seems that there is enough of Ram. Also a parameter not valid error might be caused by some other thing(s).

    Do you use eg multiple threads (in any part of the code)? Can it be that the save command is raised before the bitmap is completely written/processed? Or that any other component wants to access the bitmap while being loaded or processed? This then would raise an unvalid param error (with HResult=-2147024809)

    Some hint for things like that you could get from putting a breakpoint to the "offending" line, and see, if the code execution succeeds when just breaking and waiting a bit on that line, then continuing the code execution... Means different results on debugging with waiting and running all at once...

    Regards,

      Thorsten



    Monday, November 6, 2017 11:12 PM
  • in my opinion - from your picture - this has nothing to do with having enough memory or not, it seems that there is enough of Ram. Also a parameter not valid error might be caused by some other thing(s).

    >> I was assuming that because smaller files = smaller bitmaps and smaller files never error. I know for sure that I am not passing a negative number for Width or Height. What other conditions can cause that parameter not valid exception ?

    Do you use eg multiple threads (in any part of the code)? Can it be that the save command is raised before the bitmap is completely written/processed? Or that any other component wants to access the bitmap while being loaded or processed? This then would raise an unvalid param error (with HResult=-2147024809)

    >> The bitmap is instantiated in the Main Thread, a Background Worker fills in the pixels and the "Save" button stays disabled until the Background Worker is finished.

    Some hint for things like that you could get from putting a breakpoint to the "offending" line, and see, if the code execution succeeds when just breaking and waiting a bit on that line, then continuing the code execution... Means different results on debugging with waiting and running all at once...

    >> I will try that - the only thing that happens before setting up the bitmap is to create a compressed array from the file chosen. I cannot determine the Bitmap dimensions without the length of the compressed data. Previous versions read the file directly to an array and compressed it to another array, so I figured going direct from file to compressed would help. I suppose I could get the length of the compressed array, then dispose of it before trying to instantiate the bitmap. I'll try that also and then get the compressed array after the bitmap is set up.

    Regards,

      Thorsten

    Wednesday, November 8, 2017 12:14 AM
  • , then dispose of it 

    Devon,

    I don't know if you paid attention to what I wrote about not using the ToArray. However, this I write because you wrote dispose. I don't know if you mean calling the method dispose which is inherited from components. 

    The only thing that method does (if it is not complete overriden like with old versions of Sharepoint)  is making an object unusable, the object stays complete in memory. 


    Success
    Cor

    Wednesday, November 8, 2017 12:55 AM

  • >> The bitmap is instantiated in the Main Thread, a Background Worker fills in the pixels and the "Save" button stays disabled until the Background Worker is finished.

    Hi,

    maybe just test doing it all synchronously by commenting out the start of the bgw and call the dowork and workerfinished methods directly...

    Another question: Is that bitmap connected to any UI-control (Panel, PictureBox)? If so, test with disabling that... Probably the UI receives a paint message while the bitmap is written...

    Regards,

      Thorsten

    Wednesday, November 8, 2017 2:21 AM
  • Maybe your D drive isn't large enough for storing the file or some other issue with the D drive. Or perhaps the file is still open that you are trying to write to for some reason.

    On the other hand a .Png is a compressed format and I was under the impression you want to compress a Bitmap for storage using compression capabilities.

    This PDF [MS-ERREF]: Windows Error Codes is from 2017 but I don't remember how to convert the negative decimal (-2147024809) to hex. Maybe also see this MSDN link (same name as .PDF download but perhaps more info)  [MS-ERREF]: Windows Error Codes.


    La vida loca

    Wednesday, November 8, 2017 6:10 PM
  • Maybe your D drive isn't large enough for storing the file or some other issue with the D drive. Or perhaps the file is still open that you are trying to write to for some reason.

    On the other hand a .Png is a compressed format and I was under the impression you want to compress a Bitmap for storage using compression capabilities.

    This PDF [MS-ERREF]: Windows Error Codes is from 2017 but I don't remember how to convert the negative decimal (-2147024809) to hex. Maybe also see this MSDN link (same name as .PDF download but perhaps more info)  [MS-ERREF]: Windows Error Codes.


    La vida loca

    The Drive I have been using has 2.7 TB of free space, so I don't think that is the issue. I don't use any file operations that leave a file open. PNG is a compressed lossless format, the lossless attribute is required for successful decryption, the compression beforehand does two things:
    1 - With text files, it makes the amount of data required about 50%
    2 - With text files, it makes the character distribution much more even, which makes it impossible to try to guess the message by looking at the frequency of unique colors in the bitmap.

    HRESULT -2147024809 = 0x80070057 - E_INVALIDARG - One or more arguments are invalid. No other details seem available.

    Wednesday, November 8, 2017 11:06 PM

  • >> The bitmap is instantiated in the Main Thread, a Background Worker fills in the pixels and the "Save" button stays disabled until the Background Worker is finished.

    Hi,

    maybe just test doing it all synchronously by commenting out the start of the bgw and call the dowork and workerfinished methods directly...

    Another question: Is that bitmap connected to any UI-control (Panel, PictureBox)? If so, test with disabling that... Probably the UI receives a paint message while the bitmap is written...

    Regards,

      Thorsten

    I tried running everything in the main thread, obviously I left out the "Report Progress" part, but on a large file, it never gets there. I added some Message Boxes to show if it ever actually starts encrypting or decrypting but they never show up. Nope, the Bitmap is just a Bitmap, no picture box, no panel, it never gets displayed at all.

    On large files (40 to 50 MiB), it errors when I try to create the bitmap. On files over 100 MiB, the compress function throws an Out of memory exception. Compression always takes place before the Bitmap is set. I have not tried 60,70,80,90 MiB files to see exactly where the shift in errors occurs.

    I rewrote all this as a console app, 90% by just copy and pasting code and the largest file I have tested was 200 MiB, it encrypted, decrypted and verified with zero issues.

    Update - I rewrote the encryption part and got rid of every "large object"-
    The input file never gets read into an array or string. The input file is compressed file to file into a compressed file. If the Input File is less than 256K, it gets displayed, otherwise only the first and last lines are displayed using a streamreader to read lines, again, no big strings.
    That compressed file is directly saved to disk as a Temp File, it never gets into an array.
    When encrypting, one byte at a time is binary read from the compressed file and processed.
    Tested and confirmed working (by decrypting) on anything up to and including 40 MiB.
    At 50 MiB, the same error while setting the bitmap size (Invalid Parameter).

    • Edited by Devon_Nullman Friday, November 10, 2017 7:37 PM Added more bad news
    Thursday, November 9, 2017 6:22 PM