locked
How to store the real time data efficiently into database RRS feed

  • Question

  • Hi all,

        I am developing a windows application using c# where I am getting the real time data from the server ,the maximum number of data for a particular item in one second will be 65 data and there may be chances of more thousand items .

        So , what I need is the data for all the items should be logged into database without any loss of data which the server gives.

        There is a datatable for each item to store its own data ,there is no problem in my application till the data stores into the datatable and I want to use threading efficiently to store these data from datatable into database(My SQL and Ms SQL) table.

        Please anyone help me in doing this efficiently without any performance issues like CPU usage,Memory leaks etc...

    Thanks,


    Prabhakaran.K

    Wednesday, September 12, 2012 4:32 AM

Answers

  • Ah...sounds to me a big ask in a short question and I doubt if the answer is going to be specific (unless you narrow down/tweak question's scope by providing further inputs). I would recommend studying and making use of .NET Framework's Task Parallel Library  to serve your purpose. Especially, if LINQ delights you and you have the liberty to use it in your application you may want to take deep dive of PLINQ and to get quickly started you may jump onto PLINQ Data Sample.

    HTH

    Sameer


    If this answers your question, please Mark it as Answer. If this post is helpful, please vote as helpful.

    • Proposed as answer by Jason Dot Wang Thursday, September 13, 2012 7:38 AM
    • Marked as answer by Jason Dot Wang Wednesday, September 19, 2012 9:31 AM
    Wednesday, September 12, 2012 6:30 AM
  • Read this article

    http://www.codeproject.com/Articles/193611/DotNetMQ-A-Complete-Message-Queue-System-for-NET

    You can create a message to to the database in your application.  You can view all the source code of the project online.  Even if you don't use all of the code you can take a good portion of the code you need from this webpage.


    jdweng

    • Proposed as answer by Jason Dot Wang Thursday, September 13, 2012 7:38 AM
    • Marked as answer by Jason Dot Wang Wednesday, September 19, 2012 9:31 AM
    Wednesday, September 12, 2012 6:30 AM
  • Let me see if I have understood the problem:

    You have an in-memory DataTable which is loaded with data that you want to store in a database server. You want to do this efficiently without imposing a significant load on your client, because it is simultaneously receiving live data which you don't want to lose.

    My suggested approach:

    Use one thread for receiving the live data and storing it in the DataTable. As soon you have completed loading this DataTable, start a new thread for uploading its contents into the database server, while the main thread continues to receive data into another separate DataTable. The code that uploads the datatable into the server could use a SqlBulkCopy object, which can efficiently send the whole table to the server.


    Wednesday, September 12, 2012 6:30 AM

All replies

  • Ah...sounds to me a big ask in a short question and I doubt if the answer is going to be specific (unless you narrow down/tweak question's scope by providing further inputs). I would recommend studying and making use of .NET Framework's Task Parallel Library  to serve your purpose. Especially, if LINQ delights you and you have the liberty to use it in your application you may want to take deep dive of PLINQ and to get quickly started you may jump onto PLINQ Data Sample.

    HTH

    Sameer


    If this answers your question, please Mark it as Answer. If this post is helpful, please vote as helpful.

    • Proposed as answer by Jason Dot Wang Thursday, September 13, 2012 7:38 AM
    • Marked as answer by Jason Dot Wang Wednesday, September 19, 2012 9:31 AM
    Wednesday, September 12, 2012 6:30 AM
  • Read this article

    http://www.codeproject.com/Articles/193611/DotNetMQ-A-Complete-Message-Queue-System-for-NET

    You can create a message to to the database in your application.  You can view all the source code of the project online.  Even if you don't use all of the code you can take a good portion of the code you need from this webpage.


    jdweng

    • Proposed as answer by Jason Dot Wang Thursday, September 13, 2012 7:38 AM
    • Marked as answer by Jason Dot Wang Wednesday, September 19, 2012 9:31 AM
    Wednesday, September 12, 2012 6:30 AM
  • Let me see if I have understood the problem:

    You have an in-memory DataTable which is loaded with data that you want to store in a database server. You want to do this efficiently without imposing a significant load on your client, because it is simultaneously receiving live data which you don't want to lose.

    My suggested approach:

    Use one thread for receiving the live data and storing it in the DataTable. As soon you have completed loading this DataTable, start a new thread for uploading its contents into the database server, while the main thread continues to receive data into another separate DataTable. The code that uploads the datatable into the server could use a SqlBulkCopy object, which can efficiently send the whole table to the server.


    Wednesday, September 12, 2012 6:30 AM
  • Thanks for the reply,

         Ya you have understood my question very well ,and there is no need of threading to receive the data because there is some events are there to receive the data from server and absolutely there is no problem in using that .

        So, what I need is after receiving the items values into their own datatable ,I want to store that into database efficiently without losing any data and the upcoming data also should continuously receive in backend.

    Thanks,


    Prabhakaran.K

    Friday, September 14, 2012 3:48 AM
  • Hi Prabhakaran88,

      I am hard to clarify if these replies  meet your requirements or not,so I mark these replies as answer together. If you have any concerns, please feel free to unmark them. Above all, I suggest that you could encourage their participations by marking these replies.

      Sincerely,

      Jason Wang


    Jason Wang [MSFT]
    MSDN Community Support | Feedback to us

    Wednesday, September 19, 2012 9:41 AM