none
BackgroundWorker - How to access to item on work completion RRS feed

  • Question

  • Hello folks,

    Today question is : Is it possible to access a field in the Item which complete run in BackgroundWorker?

    I'm aware of _RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) - it didn't provide access to object  passed to RunWorkerAsync(object pObjectToRun).

    Background:  

    I have few set's of T4 templates and expect a grow of amount of templates. 

    In way in which I use templates they call one another recursively, unlimited deep.

    Each instance of T4 template use a StringBuilder to store generated text.

    This StringBuilder's are relatively expensive items. In the default version of T4 base class time of StringBuilder creation took more than 30% of execution time. Second limit is that I was not able to instantiate more than 250 StringBuider in application - so unlimited deep is very much limited. 

    There are few solution for this situation. At least I can use GC or have a pull and return existing StringBuilder to the pull when it not longer used by T4. I use a pull.

    For long time I use a single thread application and was happy with that. Recently I reach a limit ot instantiated StringBuilder (even with returning them to the pull). This force me to use a multithreading. This allow to resolve problem with amount of instantiated StringBuilder by using ThreadStatic version. I even can allow thread to instantiate and drop StringBuilder, but I want to keep them reused over pull.

    Question Is : How to get return SB to the pull when thread complete t4 run?

    Thursday, May 25, 2017 12:23 PM

All replies

  • Humm... I won't say async/await is designed to replace BackgroundWorker.

    BackgroundWorker is designed to spawn task in separate thread, with ProgressChanged event handler exposed that let you update UI element while you're doing work without ever need to check whether you're on the UI thread. That's more than what async/await offers.

    ======

    The result can be accessed through the RunWorkerCompletedEventArgs.Result property. Note that freeing it does not free the StringBuilder because the background worker itself may still not be disposed and holds reference to that StringBuilder instance. Try create StringBuilder within your DoWork method and return the resulting string only.

    Friday, May 26, 2017 6:14 AM
    Answerer
  • Hello William,

    Probably, I also will stop using BackgroundWorker. And will go back to clear Thread using. At least what I wrote using Thread's - working.

    async/await - may be a good idea, but I need to study this concept. And to do a study will take a time which I didn't have at the moment.

    Any ideas how to resolve problem with BackgroundWorker will be appreciated.

    Best regards,

    Andrey

    Friday, May 26, 2017 7:50 AM
  • Hello cheong00,

    Yesterday spend some time on code modification and find that something goes completely wrong - StringBuilder are retrieved from the pull and assigned to the variable in the template, but didn't remain there.

    I would not like to have a StringBuilder created within the thread. This will increase a single pass time from 40 sec to 20-25 minutes. This 20 minutes will be spend exactly on instantiating and utilisation of StringBuilder's. Second problem - limit of 240 StringBuilder. StringBuilder need a special utilisation to be completely removed from the system. So, I will have to run GC after I release 100-150 StreamBuilder's. All this is very much unacceptable.

    So, I would be happy to use a pull for StringBuilder's store and some how manage to get it work within BackgroundWorker.

    Best regards,

    Andrey

    Friday, May 26, 2017 8:08 AM
  • It sounds as if you're using the default constructor of StringBuilder and let it reallocate more memory as you fit the data there. <strike>Try pass the initial capacity that's a little bigger than the initial template size to minimize the reallocation overhead.</strike> This is not relevant.

    Also, as of .NET v4.0 and v4.5.X, don't attempt to reuse StringBuilder in short cycle iteration as it'll induce pressure on your GC. It'd be way better to just use it, discard it, and instantiate it again. And manually calling GC.Collect() won't help either - it will just promote those memory to higher generation and make it more expensive to dispose.

    EDIT: Although the article I linked is for the problem on .Insert(), it applies to .Replace(string, string) as well because there will also be new block of memory allocated to the linked list - the inserted string may not be the same size as your replacement marker string.

    Friday, May 26, 2017 8:32 AM
    Answerer
  • Hello cheong00,

    Yes, I use a default constructor for StringBuilder.

    I do this as I do not have any idea about amount of text which should be produced. I can say that the minimum amount will be 16K (and this will demand app 2Gb memory for 120 SB's) and have the real output 100 bytes. Other way - real output easily can be 16M - in this case it's almost no difference do I specify initial 16K size or not.
    So, it's will remain a default constructor.

    Second what stop me on initiate with size - I have a sequential appending to the SB. So, I should be able to replace StringBuilder with some kind of Stream (for example StreamWriter, MemoryStream) and didn't fill any difference. Almost none - StreamWriter will created a bit faster, but it less than 0.1%... 

    Real problem is in amount of StringBuilder's or Stream's I need to created - each call of subtemplate demand another instance. And call's are recursive with unlimited depth.

    That's how I see a situation.

    Ok. I do find where was a problem.

    To reduce an amount of instantiated StringBuilder I use a single (ThreadStatic) StringBuilder for each thread I use. This works fine as each T4 produce only sequential output.

    Problem was that this ThreadStatic variable must be initialized within the thread. Other way it's remain uninitialized within thread... even if passed object have this this variable initialized.

    It looks like it would be a bit of a problem to handle this by BackgroundWorker - object are initiated in one stream and passed for execution into another stream.

    Xmmm... I do think about something like MyThreadContext to hold all required elements for a thread - need to implement a version which hold SB, Template and, probably, FileManager...

    Best regards,

    Andrey

    Friday, May 26, 2017 10:15 AM
  • See my first reply, you can access the StringBuilder if you assign it to e.Result before your .DoWork() returns.
    Friday, May 26, 2017 10:39 AM
    Answerer
  • Hello cheong00,

    This may work, but there is different problem - ThreadStatic variable which used in the template base class are initialized in DoWork() for first template and used by first template and working fine, but it is not the same variable which used for subtemplate.

    I have to say that the very strange threads management used in BackgroundWorker. I do implement what I need by using standard Thread and code works without problem, but the same code in BW - failed... 90% of BackgroundWorker purpose are.. lost!!!

    So, I decide to do not use BacgroundWorker for anything what can't be done within DoWork()...

    Best regards,

    Andrey

    Friday, May 26, 2017 1:25 PM
  • Btw, is there any reason you need that much StringBuilder?

    Normally we'll create 1 worker thread for each logical processor. Since each logical processor can only have one thread running at a time, create worker thread more than that will not give you any performance gain. Instead, the context switching between sleeping and wakeup threads will kill your performance.

    Now if your application is running on a typical 16 logical core server, even if you need 5 StringBuilder running on each thread, the total number of StringBuilder you'll have running would be less than 100.

    Honestly speaking I've not heard of the 250-StringBuilder-limit you mentioned before, but even then IMO for most of us it's not very likely to hit this limit in, say, 10 years. To create more core on die, the circuit is going to be very complex - and therefore very expensive. And given we're very close to the theatrical limit of smallest silicon transistor size, the form factor of processor would limit the number of cores on die.


    Friday, May 26, 2017 3:47 PM
    Answerer
  • > is there any reason you need that much StringBuilder?

    -----

    It is how Microsoft implement a base class for T4-templates - each time you instantiate a template - you also created StringBuilder. My templates used one another and some are recursive. So, if I have to process by the templates some information from database, for example a table with 50 fields, and I have 5 templates to process each field and call are recursive I will reach a limit of StringBuilder's in the system. This for single thread app.

    >Since each logical processor can only have one thread running at a time, create worker thread more than that will not give you any performance gain.

    -----

    This not completely true. You, I think, accept as given that the source data for each template are 100% available.  But data in the file or somewhere on the network not the same as data in instantiated object in memory. It take a time to access to those data. This time can be used by different thread to run. I assume that 5-10 threads per logical core will work fine. At least this is a parameter for system.

    The only problem I see with this an amount of StringBulder's. If it one for a thread there should be no problem.

    >Honestly speaking I've not heard of the 250-StringBuilder-limit you mentioned before

    -----

    In 2005/6 I hit this limit on Win XP system with DotNet 2.0. That was why a Pull for StringBuilder's was introduced to the system - avoid to use a GC and re-creation of StringBuilder. And I think that nothing was changed in DotNet StringBuilder since that time.

    Monday, May 29, 2017 9:00 AM
  • Note that the memory usage characteristics of StringBuilder in .NET v4+ is completely different than .NET v3.5 or below, therefore it introduced the performance issue I mentioned earlier that does not exist on .NET v3.5 or below. In a number of forum threads here, it seems any attempt to reduce it's memory footprint will make the performance worse. Therefore I suggest you to measure the memory usage again to make sure.
    Monday, May 29, 2017 12:44 PM
    Answerer
  • Hello,

    Yes, they did some work on this.

    New test - I5-2320, 8Gb RAM, Win7 SP1 - allow me to create 250K StringBuilder's before system significantly slow down.

    It's keep working even on 500K StringBuilder's, but already get partly frozen.

    I do test only creation of StringBuilder(16K, 20M) and didn't put any data into them.

    I do finish this part.

    I have my Pull for StringBuilders and 10 threads running in parallel for approx 500 templates.

    Work, which with for single thread do took 20-25 sec, now are completed within 0.9 sec.

    So, if I took 4 CPU with 4-8 cores (plus HT) system it wold be able to give first respond  within 0.3 sec for approx 2000 clients. This more then satisfied me.

    Thanks for your help.

    Thursday, June 1, 2017 4:26 PM
  • Hi Andrey Belyakov,

    I am glad to know that you resolve the issue, please mark the helpful reply as answer, it will be beneficial to other communities who have the similar issue.

    Best regards,

    Cole Wu


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, June 5, 2017 9:41 AM
    Moderator