none
Write a string with a size > 126MB into a text file RRS feed

  • Question

  • I'm developing a C# application with .NET Framework 4.7 and SQL Server 2016.

    I have a stored procedure that return a string. That string has a size greater than 126MB and when I try to save to disk I don't get any error but the string is not saved it.

    I have tried with WriteAllText and with StreamWriter, but with the same result: no error and no file save it.

    StoredProcedures.ExportEndedProductionOrder(connectionString, out json);
    
    //File.WriteAllText(fullPath, json);
    StreamWriter file = new StreamWriter(fullPath)
    {
        AutoFlush = true
    };
    
    file.Write(json);
    file.Close();

    The value of fullPath is something like: D:\MyCompany\exportFiles\myFile.json.

    The json doesn't have new lines.

    This method works fine with smaller strings.

    How can I save very large strings?


    • Edited by VansFannel Tuesday, March 20, 2018 7:43 AM
    Monday, March 19, 2018 3:24 PM

All replies

  • I would do three things.

    First I would put it in a "try" block with a corresponding catch block of course. Just in case there are errors. That should be done even if there are no errors yet.

    I would also read the documentation. It says:

    Flushing the stream will not flush its underlying encoder unless you explicitly call Flush or Close.

    I am not sure what that means, but other things in the documentation would concern me enough that I would sure put in a Flush. Why not? I would at least try that.

    Also, you don't show us where "json" comes from or anything. I would put a breakpoint on the Write method and debug the program and look to see if there is data in json. You assume that if there is data for small amounts of data then there will be data for large amounts of data but I would verify that assumption.



    Sam Hobbs
    SimpleSamples.Info


    Tuesday, March 20, 2018 2:17 AM
  • Hello VansFannel,

    >>but with the same result: no error and no file save it.

    What do you mean "no file save it"? It's strange behavior that no file is generated because a file would be created when create a stremwriter object. Or you mean that the origin file just don't append josn data?

    And you could try the below code. I test the json size could be larger than 11M until it throws out-of-memory exception.

     using (FileStream fs = new FileStream("Test.txt", FileMode.OpenOrCreate))
                {
                    using (StreamWriter sw = new StreamWriter(fs)) {
                        sw.Write(json);
                    }
                }

    Best regards,

    Neil Hu


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, March 20, 2018 5:28 AM
    Moderator
  • I was going to post yet another reply trying to help you find YOUR problem, but in exploring this, it is clear there is some kind of a bug in StreamWriter.  Consider the following program:

    using System;
    using System.IO;
    using System.Text;
    
    public class Program
    {
        static public void Main()
        {
            // Create a 126MB string.
            StringBuilder xxx = new StringBuilder();
            for( int i = 0; i < 126000000/50; i++ )
            {
                xxx.Append( "12345678901234567890123456789012345678901234567890" );
            }
            StreamWriter file = new StreamWriter( "bigstring.txt" );
            file.Write( xxx.ToString() );
            file.Close();
        }
    }

    As written, that creates a file of 126,000,000 bytes.  If I use "Flush" instead, I also get 126,000,000 bytes.  But if I delete that line, the file is only 125,999,104 bytes.  That is a dangerous flaw in a fundamental class.  When a program terminates normally, buffers and files are all supposed to be flushed and closed.  That has been true on every operating system and every library that has ever supported a file system.


    Tim Roberts, Driver MVP Providenza & Boekelheide, Inc.

    Wednesday, March 21, 2018 7:02 PM
  • That has been true on every operating system and every library that has ever supported a file system.

    Yes bizarre but true. We must do a flush for .Net streams. I think Close does a flush but if we don't call either explicitly then we loose data. It is documented so if anyone were to report it as a bug then Microsoft will just say it is by design. I assume it is a symptom of garbage collection but it seems to me that there could be a solution somehow. If they want to blame garbage collection then it is a good example of a problem with garbage collection.


    Sam Hobbs
    SimpleSamples.Info

    Wednesday, March 21, 2018 7:27 PM
  •         StreamWriter file = new StreamWriter( "bigstring.txt" );
            file.Write( xxx.ToString() );
            file.Close();
        }
    }

    As written, that creates a file of 126,000,000 bytes.  If I use "Flush" instead, I also get 126,000,000 bytes.  But if I delete that line, the file is only 125,999,104 bytes.  That is a dangerous flaw in a fundamental class.  When a program terminates normally, buffers and files are all supposed to be flushed and closed.  That has been true on every operating system and every library that has ever supported a file system.

    It seems they expect everyone to use a using statement which handles
    the flushing/closing automatically:

    using (StreamWriter file = new StreamWriter("bigstring.txt"))
    { 
        file.Write(xxx.ToString());
    }
    

    Without a using, or an explicit flush/close, the object doesn't get
    appropriately disposed.

    In the docs for StreamWriter it cautions:

    Important

    "This type implements the IDisposable interface. When you have finished
    using the type, you should dispose of it either directly or indirectly.
    To dispose of the type directly, call its Dispose method in a try/catch
    block. To dispose of it indirectly, use a language construct such as
    using (in C#) or Using (in Visual Basic). For more information, see the
    “Using an Object that Implements IDisposable” section in the IDisposable
    interface topic."

    https://msdn.microsoft.com/en-us/library/system.io.streamwriter(v=vs.110).aspx

    - Wayne

    Wednesday, March 21, 2018 9:04 PM
  • Yes they want us to use a using statement. I don't like that because the using statement does not use try/catch. I think however that flushing is not related to disposing (although in real life people say they are related). A fundamental problem is the documentation; it is not clear.


    Sam Hobbs
    SimpleSamples.Info

    Wednesday, March 21, 2018 10:00 PM