locked
huge pain for huge cashe requirements that dont cope well with asp pool recyling RRS feed

  • Question

  • User2110873642 posted

    i have a 80mb object (class object) that i store in a application variable as cashe. i want to store or open it as fast as possible when the website recycles.

    what is the best method? i think a database cant handle it. ive now tried serialization to disk, but its still too slow. takes like 2 minutes to read, then store it. i wish i could just keep it in memory somehow, even trough the application starting and stopping.

    Wednesday, March 11, 2020 1:14 AM

Answers

  • User2110873642 posted

    I chose to use Application Initilialising instead. works fine, and easy to implement.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Tuesday, March 17, 2020 10:57 PM

All replies

  • User475983607 posted

    i have a 80mb object (class object) that i store in a application variable as cashe. i want to store or open it as fast as possible when the website recycles.

    what is the best method? i think a database cant handle it. ive now tried serialization to disk, but its still too slow. takes like 2 minutes to read, then store it. i wish i could just keep it in memory somehow, even trough the application starting and stopping.

    There are a lot of benefits for using a database.  The most obvious for you is persistence.  Another benefits is asynchronous IO.  Your application can run processes while the database inserts or fetches data.  Caching becomes far more effective because you are not forced to cache the entire data structure.  You'll be able to use cache as intended.  Moving to a database will improve performance probably by orders of magnitude.  

    Wednesday, March 11, 2020 2:24 PM
  • User2110873642 posted

    only like 1-2% of the data doesnt have to be cashed. for that little difference its better to just serialize EVERYTHING without any database involved.

    my website is faster than any database driven site will ever get. its just the startup times that gives a bottleneck. but i tuly cant live without the cashe as it is. 

    i wish i could just acces the cashe out of process, but i dont know any solution that doesnt have to parse, and gives true adress acces.

    Wednesday, March 11, 2020 5:23 PM
  • User475983607 posted

    only like 1-2% of the data doesnt have to be cashed. for that little difference its better to just serialize EVERYTHING without any database involved.

    In my experience, cache is dynamic.  Bits are added and bits are dropped depending on the current demand.  Serialization would only involve the data that expires.

    my website is faster than any database driven site will ever get. its just the startup times that gives a bottleneck. but i tuly cant live without the cashe as it is. 

    I don't agree.  Many developers on this forum maintain applications that contain millions of records. You app is having a hard time with 80M which is pretty small.

    i wish i could just acces the cashe out of process, but i dont know any solution that doesnt have to parse, and gives true adress acces.

    Look into Sqlite.

    Wednesday, March 11, 2020 6:13 PM
  • User2110873642 posted

    That is really not what i need.

    all of that 80 MB needs to be accurate, up to date and changable every single page-hit. the data is used for specific stuff that may not even 1 second be outdated. otherwise calculations mess up.

    a database will take hours for every pageload. everytime a page loads, there isnt 1 field of data in the cashe that isnt read, and nearly half of it changes. its not so much a database, its litteraly ram storage. it was never meant as a database. i need it for: Extreme Transaction Processing. i cant afford to have SQL serialize every transaction and transfer 80MB down the wire. this approuch also needs me to cashe the result anyway, to avoid multiple calls per page.

    Wednesday, March 11, 2020 6:56 PM
  • User288213138 posted

    Hi fazioliamboina,

    i have a 80mb object (class object) that i store in a application variable as cashe. i want to store or open it as fast as possible when the website recycles.

    what is the best method? i think a database cant handle it. ive now tried serialization to disk, but its still too slow. takes like 2 minutes to read, then store it. i wish i could just keep it in memory somehow, even trough the application starting and stopping.

    You can try to use the redis. The redis is a flexible, open-source, in-memory data structure store, used as database, cache, and message broker. Redis allows the user to store vast amounts of data without the limits of a relational database.

    The performance of Redis is extremely high. The reading speed is 110,000 times/s and the writing speed is 81,000 times/s.

    Redis supports data persistence. You can save the data in memory to disk, and it can be loaded and used again when restarting.

    More information about Redis you can refer to this link: https://redis.io/

    Best regards,

    Sam

    Thursday, March 12, 2020 2:28 AM
  • User-821857111 posted

    SqLite is actually an option that you should consider. It's at least worth running some tests. One of its features is that it performs extremely efficiently if you issue multiple small commands to it instead of one big query: https://www.sqlite.org/np1queryprob.html

    Otherwise, you should look at Redis or one of the other cache providers, but from what you are saying, you need to look at your infrastructure first. If you want a robust caching service, take a look at Amazon or Azure's offers 

    https://azure.microsoft.com/en-gb/services/cache/

    https://aws.amazon.com/redis/

    Thursday, March 12, 2020 7:32 AM
  • User2110873642 posted

    well, i tried redis and ncashe, but they are both too slow. the problem is the interprocess communication by itself.

    it has to be in-proc, or maybe some kind of hack to loan memory from another process by adress, instead of trough communication.

    could memorymappedfile do something like that? from what ive seen, it appears that it requires streams and you cant just use it as a variable adress.

    Friday, March 13, 2020 1:03 PM
  • User475983607 posted

    Given your vague but intensive memory requirements, I find your decision to write this application in a manage code perplexing. 

    Anyway, the latest C# versions have APIs for allocating memory. 

    https://docs.microsoft.com/en-us/dotnet/standard/memory-and-spans/

    That might be another path.  It hard to determine as the community cannot run or review your design/code.  Plus your application requirements are still a bit vague.   It is impossible to if your design is the greatest thing since sliced bread, absolutely horrendous, or somewhere in between.

    Friday, March 13, 2020 2:03 PM
  • User2110873642 posted

    I chose to use Application Initilialising instead. works fine, and easy to implement.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Tuesday, March 17, 2020 10:57 PM