none
How to pause ConcurrentBag processing while uploading data to db RRS feed

  • Question

  • Hi,

    We have code were we are using Parallel.ForEach() method to process different folder of directory. We have used ConcurrentBag instead of List object as it is thread safe. Now from this Concurrentbag we add row to datatable. Once all the folder is read and data is inserted into data table we upload the bulk data into SQL. 

    Now the problem is that when there is millions of data then there is memory running out problem. Below is the code to push data into datatable from concurrentBag

    lock(datatable)

    {

    foreach(var item in conBag){

    // add row to datatable

    }

    }

    How can we use BlockingCollection/ConcurrentBag to check if data = 30000 then upload data and after uploading resume the pushing of data into datatable from this collections 

    Tuesday, August 25, 2020 1:15 PM

Answers

  • Hi Md Zakir,

    Thanks for your feedback.

    You can consider using BlockingCollection instead of ConcurrentBag.

            const int MAX_BAG_SIZE = 30000;
            static BlockingCollection<string> data = new BlockingCollection<string>(MAX_BAG_SIZE);
            static void addItem()
            {
                // add item to BlockingCollection
            }
            static void addRowToDataTable()
            {
                foreach (var item in data.GetConsumingEnumerable())
                {
                    // add row To dataTable
                }
            }
    
                //...
                Task.Factory.StartNew(() => addItem());
                Task.Factory.StartNew(() => addRowToDataTable());
    
    

    When the count of members in BlockingCollection reaches 30000, BlockingCollection.Add method will be blocked until an item is removed from the collection.

    Best Regards,

    Xingyu Zhao

     

    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Md Zakir Wednesday, August 26, 2020 8:09 AM
    Wednesday, August 26, 2020 7:32 AM
    Moderator

All replies

  • Hi Md Zakir,

    Thank you for posting here.

    You can consider using BlockingCollection to provide blocking and bounding capabilities for ConcurrentBag.

        const int MAX_BAG_SIZE = 30000;
        BlockingCollection<string> log = new BlockingCollection<string>(new ConcurrentBag<string>(), MAX_BAG_SIZE);

    More details you can see this reference.

    Hope it could be helpful.

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, August 26, 2020 2:55 AM
    Moderator
  • i have used

    collBag.Add(item);

    if(collBag.count >= 30000){

    foreach(var item in collBag.GetEnumerable()){

    //add row to data table

    }}

    if(datatable.Rows.Count >= 30000){

    //Bulk insert

    }

    But above code is not working

    Wednesday, August 26, 2020 6:08 AM
  • Hi Md Zakir,

    Thanks for your feedback.

    You can consider using BlockingCollection instead of ConcurrentBag.

            const int MAX_BAG_SIZE = 30000;
            static BlockingCollection<string> data = new BlockingCollection<string>(MAX_BAG_SIZE);
            static void addItem()
            {
                // add item to BlockingCollection
            }
            static void addRowToDataTable()
            {
                foreach (var item in data.GetConsumingEnumerable())
                {
                    // add row To dataTable
                }
            }
    
                //...
                Task.Factory.StartNew(() => addItem());
                Task.Factory.StartNew(() => addRowToDataTable());
    
    

    When the count of members in BlockingCollection reaches 30000, BlockingCollection.Add method will be blocked until an item is removed from the collection.

    Best Regards,

    Xingyu Zhao

     

    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Md Zakir Wednesday, August 26, 2020 8:09 AM
    Wednesday, August 26, 2020 7:32 AM
    Moderator