Exceeding 2gb .NET limit RRS feed

  • Question


    We have run into the 2gb limit on the use of .Net in our application.


    Our application takes data from analytical instruments which output enormous amounts of data. The instruments typically run overnight and unattended. The application can have 1000's of double arrays (inside of DataSet's) that can be of arbitrary length. The LOH is reaching values of over 1.3gb.


    What is happening is that the LOH is being fragmented. There is no means for compacting the LOH.


    All these elements taken together result in an OutOfMemory exception that appears during peak runs. The program halts (usually overnight) and we are in real trouble.


    It is not feasible to go to x64 computers at this time.


    The " 4GT RAM Tuning" tuning approach is not recommended by Microsoft for our application problem.


    Allocating a very large segment of the LOH and doing our own memory management is not feasible because of the large number of arrays and the speed of the data coming from the instruments.


    One approach we are considering is to launch another process, perform all the heavy calculations in the second process and then kill the second process. This in effect allows us to have a separate LOH that we can destroy and recreate as needed. The computer would have to have 4gb of Ram or more to avoid disk thrashing.


    One question is how best to launch the second process from the first. The Process class would do it but ideally if there was means to launch the second process which would then expose one or more interfaces to the first process. Is there a way to do this.


    Another question on the two process approach is how to effectively communicate large amounts of data between the two processes. We have looked at making the second process a  Self-Hosting Wcf application (Exe) but we are not sure how well this will handle the large amounts of data going between the two processes. We did not think running Wcf under IIS would work because of the large amounts of data and the size of the LOH needed. 


    Could anyone comment on the concept of using two separate processes?


    Could anyone comment on the idea of communicating between the two processes using Wcf?


    Finally does anyone have another approach we might use to resolve the problem. Again, in essence, if the LOH performed a compaction our problem would be solved but we are told by Microsoft that there is no way to perform this compaction.


    Any input would be appreciated.






    Friday, October 24, 2008 1:48 AM


All replies

  • There's no way that the productivity loss and the amount of tricky redesign you'll need to do will ever be less than the $180 bucks it will take to run Vista x64.
    Hans Passant.
    Friday, October 24, 2008 4:30 AM
  • Indeed.  The 64-bit migration can't be nearly as much work as writing this kind of multi-process and data-partitioning setup.  And since you're already using .NET you can just run the existing executable, and get your unlimited address space automatically.

    edit: Hold it, could it be that those instruments connected to your system don't have 64-bit drivers?  In that case you have my sincere sympathies.  Looking for another supplier with 64-bit drivers might still be cheaper than rewriting the application, though.

    Friday, October 24, 2008 6:48 AM
  • You could put your data into a database, then let the other application read it back out of the database. Going to disk and back may be quite slower than a straight memory transfer, but it would allow multiple applications to process the data simultaneously, and to scale those other processing applications onto other network PCs.
    Please mark the post if it answers your question.
    Friday, October 24, 2008 6:32 PM
  • You have raised an interesting point. What if we had the second process that handles the massive calculations on the x64 computer. 

    I assume that .Net Remoting or Wcf can be used to communicate between a x32 WinForm process on a x32 computer and a x64 WinForm process running on a separate x64 computer. Am I correct?

    Your answer suggests that Vs2008 (running as WOW) will run under a Vista x64 computer. And that it will produce .Net x64 code. Is this correct?

    Your answer also suggests that a x32 .Net process (created by Vs2008 on a x32 computer) will have a larger then 2 gb memory if run on a x64 computer. Is that correct.

    Obviously I could use some pointers on using either the Xp x64 or Vista x64 OS's and the use of Vs2008 on an x64 computer to produce x64 code.

    Thanks for the input and appreciate any inputs.


    Saturday, October 25, 2008 1:33 AM
  • No, 32-bit processes are still subject to the 2 GB virtual memory space limit on Vista x64.  But one of the wonderful goodies of .NET is that you have to do nothing at all to get it to run in 64-bit mode.  The JIT compiler automatically produces 64-bit machine code.  Just copy your executables and you're done, problem solved.
    Hans Passant.
    Saturday, October 25, 2008 1:55 AM
  • I am a bit confused.

    A x32 process on a x64 computer is still limited to the 2gb memory limitation but will Vs2008 (on a x64 computer) produce a x64 process that will have a larger memory limitation?

    In other words, I assumed that Vs2008 running on an x64 computer would produce x64 processes that would have the full benefits of x64 including the a much larger memory allocation in the LOH. Am I correct?




    Saturday, October 25, 2008 2:17 AM
  • It has nothing to do with Visual Studio or the kind of machine you develop your software on.  .NET compilers don't produce machine code, they produce IL (Intermediate Language).  That output doesn't get turned into machine code until you run your program.  The .NET Just In Time compiler then compiles the IL to machine code.  And that will be 64-bit machine code on a 64-bit operating system, 32-bit machine code on a 32-bit operating system.

    Hans Passant.
    Saturday, October 25, 2008 2:31 AM
  • I understand about the JIT. Thanks.

    The question remains. Will a x64 application running on a x64 computer created using Vs2008 have a LOH larger then the (roughly) 2gb size of an application running on a x32 computer.

    Any inputs would be appreciated.


    Sunday, October 26, 2008 1:29 AM
  • Yes.
    Hans Passant.
    Sunday, October 26, 2008 3:19 AM
  • Appreciate the inputs.



    Monday, October 27, 2008 3:42 PM