none
Caching Large amount of data RRS feed

  • Question

  •  

    Hi,

    I have a requirement for caching large amount of data like 1-1.5 GB per user. The data is not static but its a transaction data which changes intermittently if not frequently. Front End is .NET based windows form and connects to backend oracle database through app. server where singleton remoting application is running. Fetching the data directly from oracle database takes app. 25-30 second for single record because one record may contain 50K detail record.

     

    We tried to implement it using Caching Application Block at the client side but due to the memory constraint of 2 GB in 32-bit system, application gives 'Out of memory' exception. We cannot go for SQL express at the client side and SQL Standard edition at the server to cache the data in database due to organization constraint.

     

    Now, we have implemented multiple copy of the windows service to dedicated to 1-2 users for each windows service to overcome the constraint of 2GB memory constraint of 32-bit system. We cannot afford to buy in-memory database like TimesTen. Considering the situation, can anybody answer my following question: We will eventually go to 64 bit system but right now it will be 32-bit system.

     

    1. Will this multiple windows service will work satisfactorily in production system?

    2. Is this going to be robust?

    3. What are the issues you see in the current design?

    4. Do you have any recommendation in handling the situation differently?

     

    I really need your feedback as I am in extremely dilemma. It would be really help me if somebody can give me direction.

     

    Thanks,

    Aloke

    Tuesday, September 23, 2008 1:20 PM

All replies

  •  

    The first thing I would do is try to challenge the requirement for caching the large amount of data.  What is the user doing with the data that requires it to be cached?  How would the user consume 50K records at one time?

     

     

    Tuesday, September 23, 2008 1:29 PM
  •  Aloke Dutta wrote:

    1. Will this multiple windows service will work satisfactorily in production system?

    Depending on number of users, this may or may not work.  How many users you have?  Assuming you have 50 users, that means you will have 25 instances of windows service (2 users for each service).  But now you are hit with the same physical memory constraints that you ran into.  The OS has to perform way too many page file swaps, that the CPU will be busy doing just that...  So it is going to be slow.

     

     Aloke Dutta wrote:

    2. Is this going to be robust?

    Based on my understanding of Robustness, this design is not Robust.

     

     Aloke Dutta wrote:

    3. What are the issues you see in the current design?

    Caching 1-1.5 GB of transactional data sounds wrong.

     

     Aloke Dutta wrote:

    4. Do you have any recommendation in handling the situation differently?

    Need to know more about why you need to cache such a huge data...  Probably a detailed explanation would help.

     

    { Gaja; }

    Tuesday, September 23, 2008 1:53 PM
  • Hi,
    Do you want to cache 1.5 GB Per user? I hope none of the application in the universe go with this design. If you explain your current design more, we can see where the pitfall.

    Thanks,
    Suresh M
    Tuesday, September 30, 2008 6:05 AM
  • 1.5GB is too much to keep it in memory, eh..

    What you can try is to use a local DB which can be used without installation or even integrated into your app: SqlCompact (?)
    FireBird, BerkleyDB, and implement a cache provider (or how is it called, CacheStorage? I don't remember) for the Caching Application Block so you can have this cached data locally, but still in DB, not in memory.

    At least it will work much faster then without caching, and you could (almost) don't care about the memory.
    Wednesday, October 1, 2008 10:45 AM