none
Reading Files Sequentially in a Universal Windows App RRS feed

  • Question

  • I’m working on a universal windows app that has to read a list of files located in a variety of folders and process them sequentially. I’m using the Windows::Storage class Async file I/O functions to do the reading.  I’m calling them in the chain of continuation tasks that I generate recursively.  The number of files I’m reading has grown to 1100 and the app is running out of memory.  I commented out all code that could possibly cause a memory leak and ran tests that indicate that the app is accumulating a lot of stack memory in the large number of threads that it fires up (probably around 3000).  The amount of memory that the app uses is proportional to the number of function calls that it is making in the threads and I finally received a message indicating that the app ran out of stack memory.  Increasing the stack memory allocations didn’t seem to offer any relief.

    Can you refer me to the latest thinking, advice and code examples for doing file I/O sequentially in a loop or avoiding the memory accumulation issue.

    Thank you in advance for your help.

    Monday, July 1, 2019 8:48 PM

All replies

  • P.S.  The app is a C++ app.
    Monday, July 1, 2019 8:52 PM
  • >The number of files I’m reading has grown to 1100 and the app is running out of memory.

    Are you reading 1100 files into memory at the same time, or are there just a small number (how many typically) loaded concurrently?
    How large are those files?

    Dave

    Monday, July 1, 2019 10:44 PM
  • Thank you for helping.

    The files are text files.  The size of each one is 450 kBytes.

    I am reading the files one at a time, processing their contents and then deallocating the memory required to process each one before moving on to the next one.  I'm using FileIO::ReadLinesAsync to read the text.  The app typically consumes about 1600 MBytes before it throws and exception.

    Tuesday, July 2, 2019 12:29 AM
  • @Craig6213,

    Can you show some code about how you do it in C++/CX? We want to give it a test first. Well, actually a minimal repro would be better for us to understand the scenario.  

    Best regards,

    Barry


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, July 2, 2019 2:27 AM
    Moderator
  • While investigating the allocated memory accumulation that I reported above I discovered some unusual behavior I wanted to share.

    I isolated two functions that were called while memory was leaking.  These two functions were returning objects by value.  Each one had straight forward overloaded assignment operators that updated the class instances on the left hand sides of the expressions in which they were called.  Each of the classes also had an overloaded (char *) operator that I had added so I could get a pointer to a character string representation of the class instance that I could use in printf() calls.

    The unusual behavior I found was that the overloaded (char*) operators were being executed somewhere within the return by value and assignment sequence even though there were no explicit statements in the assignment operators or copy constructors for the classes that appeared to require them to execute.  

    I isolated the classes and function calls in a small main() program for testing and ran the debugger on it.  The debugger call stack list showed that the overloaded (char*) operators were called by the main() program.  There were no explict statements in main() containing them.

    I removed the overloaded (char*) operators from the two classes and the allocated memory accumulation stopped.  The updated app can process 1100 files with so significant accumulaltion of memory.

    Friday, July 5, 2019 9:15 PM
  • Hi,

    According to your description,we are still hard to understand what caused this issue.Can you show some code about this?

    Best regards,

    Fay

    Monday, July 8, 2019 6:17 AM
  • Hi,

    Could you please share some code snippet about your issue?

    Best regards,

    Roy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Friday, July 12, 2019 6:14 AM
    Moderator