none
How to speed up the file reading RRS feed

  • Question

  • Hi,

    working in VS2017 and C#.

    Help needed in choosing concurrent vs parallel. i have folder which will get multiple files (user enrollment file. It means file will have 100 * 1000 of user information and one row per user with pipe delimited. ) so I will have to read the file one by one and read the records and it will call the stored procedure which has the logic to make entry in user, address and other tables. Correspondingly. I wanted to make this process much faster and I am interested in using parallel for each or any other parallel programming which creates threads to handle this more faster. Please suggest me which approach should I choose?  Please advice me in this.

    loving dotnet


    Tuesday, August 20, 2019 12:55 AM

All replies

  • Hello,

    You might want to read the following post, not specific to file read but on topic for Concurrency vs. Parallelism.


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange


    Tuesday, August 20, 2019 3:23 AM
    Moderator
  • Based on my experience, file based I/O parallelism alone rarely can speed things up much. (I/O is not CPU-bound task)

    On the other hand, reading huge trunk of data in memory then break data down instead of reading one by one can help a lot. (This time the operation is very likely to be CPU-bound, which is the scene that most parallelization libraries designed for. You should be able to get more performance boost if you start from here)

    Of course it also mean a much larger memory footprint. So find a right balance yourself.

    Tuesday, August 20, 2019 3:45 AM
    Answerer
  • Hi Born2Achieve,

    Thank you for posting here.

    For your question, you want to know which will read the file faster between concurrent and  parallel.

    I think parallel is a good choice, the following describe the difference very clearly, you could have a look.

    What is the difference between concurrency and parallelism?

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, August 20, 2019 5:19 AM
    Moderator