none
Help needed in datatable RRS feed

  • Question

  • Hi, I am working on c#.net 2017. i have a requirement like i have 10k records in datatable and i need to use multi threading to pick 5 records at the time for process. Assume, each thread will do database call and returns UserId. based on google search, i could see parallel.foreach  is recommended. but there is one caveat that it will not maintain the order. my requirement is, i need to matain nother datatable which will have the entry of processed records from first datatable in the order of data in the first data table. assume, below the columns of datatables. its must to maintain the same order in the second datatable once processed from first datatable. please help me with some sample code. 

    First data talble columns :

    Id(unique),FirstName, Last Name, EmailAddress,BirthDate

    second data table columns: 

    Id(unique),FirstName, Last Name, EmailAddress,BirthDate, IdUser



    loving dotnet

    Tuesday, September 3, 2019 12:27 AM

All replies

  • Hi Born2Achieve,

    Thank you for posting here.

    >>my requirement is, i need to matain nother datatable which will have the entry of processed records from first datatable in the order of data in the first data table.

    I don't know how do you get the second datatable. I suggest that you could use Datable.copy method, which will maintain the order.

                DataTable table = new DataTable();
                table.Columns.Add("Id");
                table.Columns.Add("FirstName");
                table.Columns.Add("Last Name");
                table.Columns.Add("Email address");
                DataTable table1 = table.Copy();
                table1.Columns.Add("IdUser");

    If this could not solve your problem, please provide the related code that you have done.

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, September 3, 2019 5:52 AM
    Moderator
  • Hi Jack,

    please look at my requirement once again. i need to implement the multi threading in datatable and have to maintain the order of execution. any help please. 


    loving dotnet

    Tuesday, September 3, 2019 2:09 PM
  • Hi Born2Achieve,

    Thanks for the feedback.

    >>i need to implement the multi threading in datatable and have to maintain the order of execution.

    Could you describe it more clearly or give a code example to express your thought? Because I still don't know what you want.

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, September 4, 2019 1:32 AM
    Moderator
  • Hi Jack,

    This is user creation logic in backend through front end.

    Thanks for the reply and here is the requirement again. I have a datatable which has 10k rows. i need to loop through this datatable and pass the values of each row to stored proc and get the IdUser from the procedure as output. since i have to loop though datatable 10k times sequentially, i am trying to implement parallel. so each time it should talk 5 records(based on processor core count ). so threading is the way that can achieved from here. ideally based on processor core count, it should initiate the thread to pick the records. Since thread invlolved, it will maintain the order or execution. so what i planned, as soon the stored proc called and received the IdUser, add the row to another data table along with IdUser. incase any error occurs , in the catch blaock also have to add the data row to the second Datatable. The reason for this would be at the end, i have to write to log file that which record successfully processed and which record failed. So once the data in first data table processed, it will have the entry in the second datatable which tells success and failed details. This can be implemented in database level too. but i have told to do this on front end.

    Please let me know if my requirement is clear.


    loving dotnet

    Wednesday, September 4, 2019 10:44 AM
  • Ideally just do this in the database as that is where it'll be fastest and make most sense. However...

    "i need to loop through this datatable and pass the values of each row to stored proc and get the IdUser from the procedure as output"

    Use Parallel.ForEach to walk through the rows based upon your partition size. You'll need to use the overload accepting the parallel options. You might also need a custom partitioner.

    Alternatively create an async method that takes a single DataRow and does the processing. Then wrap X of these calls in a for loop that enumerates your rows and makes X calls for each one. After each async method is started then await the array of tasks that are running that loop before starting again.

    Yet another alternative is to forget the 5 count altogether (or use a max concurrency task scheduler) and just use async methods on each row. After you've enumerated all the rows and started each task then await them all to wait until they finish. The scheduler will then be responsible for running the tasks in parallel until they are all done.

    "Since thread invlolved, it will maintain the order or execution"

    Ordering is not guaranteed with threading unless you enforce it. The only real thing you can control is how you enumerate the original rows). How long each of those take to return back (and the ordering) is undefined after that. Enforcing ordering is a lot harder and can defeat the benefits of parallel processing.

    " as soon the stored proc called and received the IdUser, add the row to another data table along with IdUser. incase any error occurs"

    The method you write to call the sproc should handle inserting data into the target table. However order is not guaranteed. Also DataTable/DataSet is not thread safe for writes so attempting to update the same table on different threads will result in undefined behavior. Therefore you'll need to sync to that table in order to add the rows. It will slow down processing. Alternatively just have your async method (mentioned earlier) return the DataRow as a result. Then join them into a table at the end.

    Finally, you're really at the upper end of what you'd want to use a DataTable for. Ideally just stream your data from the database/source rather than wasting your time on DataTable and DataRow. It would be more efficient on memory and potentially not require locking (depending upon how you implement it). The fact that it is backed by a database table isn't relevant here unless you want to batch update the entire table in one call.


    Michael Taylor http://www.michaeltaylorp3.net

    Wednesday, September 4, 2019 7:51 PM
    Moderator