locked
Migrating Application to Windows 2012 Server RRS feed

  • Question

  • User-1970886443 posted

    Hi - I've been tasked with moving an application (main web app with many sub apps) from a windows 2008 server to a windows 2012 server. I've successfully managed to configure the application from an IIS perspective i.e. app pools running and connected to the appropriate sites. However from a functionality view point initial testing is failing with an error linked to crystal reports. Below is some of the logs from the Event Viewer on the 2012 server:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Application Error" />
      <EventID Qualifiers="0">1000</EventID>
      <Level>2</Level>
      <Task>100</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2017-09-08T09:28:57.000000000Z" />
      <EventRecordID>81300</EventRecordID>
      <Channel>Application</Channel>
      <Computer>2012 Server Name</Computer>
      <Security />
      </System>
    - <EventData>
      <Data>w3wp.exe</Data>
      <Data>8.5.9600.16384</Data>
      <Data>5215df96</Data>
      <Data>ntdll.dll</Data>
      <Data>6.3.9600.18725</Data>
      <Data>59380775</Data>
      <Data>c0000374</Data>
      <Data>00000000000f1c00</Data>
      <Data>1d8c</Data>
      <Data>01d3288269c6b056</Data>
      <Data>c:\windows\system32\inetsrv\w3wp.exe</Data>
      <Data>C:\Windows\SYSTEM32\ntdll.dll</Data>
      <Data>235f3484-9478-11e7-80f7-005056a67f15</Data>
      <Data />
      <Data />
      </EventData>
      </Event>

    I've been comparing the installed software on the current 2008 server and i see these packages:

    CrystalReports2005setup

    Crystal Reports Basic Runtime for VS2008 (x64)

    Both of which have been installed on the 2012 server along with:

    SAP Crystal Reports 2016 SP3

    SAP Crystal Reports Runtime engine for .Net Frameowrk (x64)

    SAP BusinessObjects BI platform .Net SDK  Redistributable 32bit 4.2 SP3

    I used robocopy to copy the code from 2008 to 2012 and all the CR DLL's copied with no issues. I've found myself chasing my tail a bit on this one and if anyone can offer any advice that I can follow up it'd be greatly appreciated. The application itself was developed using VS2008 and .net framework 3.5.

    Friday, September 8, 2017 12:46 PM

All replies

  • User1771544211 posted

    Hi pmcm,

    Have you compared the IIS settings between the Windows Server 2008 and 2012?

    Based on my experience, this issue is often related to the permissions of the IIS worker process. Please check the account(Application Pool Identity) under which worker process w3wp.exe is running have read/write permissions of your reports folder.

    More about the IIS Application Pool Identity : https://docs.microsoft.com/en-us/iis/manage/configuring-security/application-pool-identities

    Best Regards,

    Jean

    Monday, September 11, 2017 5:11 AM
  • User-1970886443 posted

    Thanks for the reply Jean Sun. I checked the App pool ID which was Network Service. I gave this ID modify permissions on the report folder and retested, app failed again. Added Network Service with modify permissions to the top level folder to cover all folders. retested, app failed again with this error (from event viewer):

    Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
    Faulting module name: ntdll.dll, version: 6.3.9600.18725, time stamp: 0x59380775
    Exception code: 0xc0000374
    Fault offset: 0x00000000000f1c00
    Faulting process id: 0x216c
    Faulting application start time: 0x01d32ae75238de73
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
    Report Id: 885919dd-96db-11e7-80f8-005056a67f15
    Faulting package full name: 
    Faulting package-relative application ID: 
    
    ------------------------------------------------------------------
    - System 
    
      - Provider 
    
       [ Name]  Application Error 
     
      - EventID 1000 
    
       [ Qualifiers]  0 
     
       Level 2 
     
       Task 100 
     
       Keywords 0x80000000000000 
     
      - TimeCreated 
    
       [ SystemTime]  2017-09-11T10:25:29.000000000Z 
     
       EventRecordID 89988 
     
       Channel Application 
     
       Computer A0185-WEB0853-S.ad.allstate.com 
     
       Security 
     
    
    - EventData 
    
       w3wp.exe 
       8.5.9600.16384 
       5215df96 
       ntdll.dll 
       6.3.9600.18725 
       59380775 
       c0000374 
       00000000000f1c00 
       216c 
       01d32ae75238de73 
       c:\windows\system32\inetsrv\w3wp.exe 
       C:\Windows\SYSTEM32\ntdll.dll 
       885919dd-96db-11e7-80f8-005056a67f15 
        
        
    

    I've compared the version of the C:\Windows\SYSTEM32\ntdll.dll on the 2008 server (6.0.6002.19623) and 2012 server (6.3.9600.18725) would this be causing a problem? Tried to copy the dll from the 2008 server to the 2012 server into the SYSTEM32 - paste failed due to Access denied.

    Monday, September 11, 2017 10:40 AM
  • User753101303 posted

    Hi,

    According to https://blogs.msdn.microsoft.com/calvin_hsia/2015/01/30/heap-corruption-exception-0xc0000374/ 0xc0000374 seems to be a "heap corruption exception" and it seems incorrect pinvoke could be part of the problem.

    Are you moving from a 32 bit to a 64 bit OS ?

    Monday, September 11, 2017 10:57 AM
  • User-1970886443 posted

    no both servers are 64bit

    Monday, September 11, 2017 12:41 PM
  • User-1970886443 posted

    I used the DebugDiag tool to try and get more information on this error. Here's my results from the Error section:

    WARNING - DebugDiag was not able to locate debug symbols for \ntdll.dll, so the information below may be incomplete.
    
    In w3wp__Isg-Edm-Repository__PID__5640__Date__09_11_2017__Time_11_23_22AM__542__Second_Chance_Exception_C0000374.dmp the assembly instruction at ntdll!RtlpNtMakeTemporaryKey+1c0 in C:\Windows\System32\ntdll.dll from Microsoft Corporation has caused a corrupted heap exception (0xC0000374) when trying to perform an unknown operation on memory location 0xc27fabc0 on thread 38

    I've contacted Microsoft to request some technical assistance. I will update this thread upon any findings.

    Tuesday, September 12, 2017 9:25 AM
  • User1771544211 posted

    Hi pmcm,

    I've contacted Microsoft to request some technical assistance. I will update this thread upon any findings.

    Thanks for your feedback.

    I think you could check if some resources that used by the crystal reports is changed when migrate from windows 2008 to windows 2012. Such as the datasource of the crystal reports.

    Best Regards,

    Jean

    Thursday, September 14, 2017 8:32 AM
  • User-1970886443 posted

    How would I begin to check if the crystal resources are being changed?

    Thursday, September 14, 2017 8:37 AM
  • User1771544211 posted

    Hi pmcm,

    Do you have the source project? You can open the source project in Visual Studio, then use Visual Studio Report Designer > Verify Database to check some settings.

    And you can debug your project in Visual Studio to see if it can show some exception message that may be helpful to identify the root cause.

    Best Regards,

    Jean

    Thursday, September 14, 2017 8:43 AM
  • User-1970886443 posted

    After I passed 2 dumps to the Microsoft support tech here's the analysis I got back:

    Dump 1:
    ========
    w3wp__Isg-Edm-Repository__PID__4408__Date__09_14_2017__Time_11_14_08AM__289__Second_Chance_Exception_C0000374.dmp
    
    Debug session time: Thu Sep 14 11:14:11.000 2017 (UTC - 4:00)
    System Uptime: 1 days 3:53:13.891
    Process Uptime: 0 days 0:05:11.000
    
    Crashing callstack is:
    
    0:038> k
    # Child-SP          RetAddr           Call Site
    00 00000073`c485c7b0 00007ffc`dcab4e42 ntdll!RtlReportCriticalFailure+0x8c
    01 00000073`c485c8c0 00007ffc`dcab5a40 ntdll!RtlpHeapHandleError+0x12
    02 00000073`c485c8f0 00007ffc`dca6a56f ntdll!RtlpLogHeapFailure+0xa4
    03 (Inline Function) --------`-------- ntdll!RtlpProbeUserBufferUnsafe+0x74ec3
    04 (Inline Function) --------`-------- ntdll!RtlpProbeUserBuffer+0x74ecd
    05 00000073`c485c920 00000000`6cb8cabc ntdll!RtlFreeHeap+0x74eff
    06 00000073`c485c9c0 00007ffc`d3ec8757 msvcr90!free+0x1c
    07 00000073`c485c9f0 00007ffc`7481a132 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
    08 00000073`c485ca90 00007ffc`748257e1 DomainBoundILStubClass.IL_STUB(Void*)+0x42
    09 00000073`c485cb50 00007ffc`74825781 crdb_adoplus!<Module>.DbFieldInfo.{dtor}(DbFieldInfo*)+0x11
    0a 00000073`c485cb80 00007ffc`74825727 crdb_adoplus!<Module>.DbFreeFieldList(UInt32, DbFieldInfo**)+0x31
    0b 00000073`c485cbd0 00007ffc`d3ec6aaa DomainBoundILStubClass.IL_STUB(Int32, Int64)+0x7
    0c 00000073`c485cc00 00000000`651f6b36 mscorwks!UMThunkStubAMD64+0x7a
    0d 00000073`c485cc90 00000000`65273538 crqe+0x16b36
    0e 00000073`c485ce40 00000000`653b1c64 crqe+0x93538
    0f 00000073`c485d130 00000000`653ac697 crqe+0x1d1c64 
    10 00000073`c485d1a0 00000000`65accab9 crqe+0x1cc697
    11 00000073`c485d240 00000000`65ef0bc1 crpe32!CRPEConnectionInit+0x53859
    12 00000073`c485d3c0 00000000`65b05500 crpe32!PESetBackgroundThreadProc+0xbf31
    13 00000073`c485d410 00000000`65b061f7 crpe32!CRPEConnectionInit+0x8c2a0
    14 00000073`c485d7c0 00000000`65b08b51 crpe32!CRPEConnectionInit+0x8cf97
    15 00000073`c485e080 00000000`65b07fe7 crpe32!CRPEConnectionInit+0x8f8f1
    16 00000073`c485e210 00000000`65b07a5b crpe32!CRPEConnectionInit+0x8ed87
    17 00000073`c485e390 00000000`65b09479 crpe32!CRPEConnectionInit+0x8e7fb
    18 00000073`c485ec50 00000000`6601edc5 crpe32!CRPEConnectionInit+0x90219
    19 00000073`c485f1e0 00000000`65ee20f2 crpe32!MWCleanupProcess+0x10cb95
    1a 00000073`c485f4b0 00000000`65ee63bd crpe32!CRPEConnectionInit+0x468e92
    1b 00000073`c485f5e0 00000000`65ee627b crpe32!PESetBackgroundThreadProc+0x172d
    1c 00000073`c485f710 00000000`65ee5ebf crpe32!PESetBackgroundThreadProc+0x15eb
    1d 00000073`c485f750 00000000`65ee5c55 crpe32!PESetBackgroundThreadProc+0x122f
    1e 00000073`c485fa30 00000000`6633e919 crpe32!PESetBackgroundThreadProc+0xfc5
    1f 00000073`c485fb60 00000000`663c7b4f crpe32!SetHelpApp+0x9b109
    20 00000073`c485fc60 00000000`663c7c03 crpe32!SetHelpApp+0x12433f
    21 00000073`c485fc90 00007ffc`dac613d2 crpe32!SetHelpApp+0x1243f3
    22 00000073`c485fcc0 00007ffc`dc9d54e4 kernel32!BaseThreadInitThunk+0x22
    23 00000073`c485fcf0 00000000`00000000 ntdll!RtlUserThreadStart+0x34
    
    
    This is a native heap corruption and hence the crash is result of that.
    
    Specifically, the heap that is corrupted is C runtime version 9.0 native heap.
    
    NOTE:  ONLY components linked to msvcr90.dll (C runtime version 9) will use this heap not just any component.  None of ours are linked like this (our components use "current" C runtime and are not tied to a specific version). 
    
    From this call stack we can see that Crystal Reports is using this specific heap and in this case are calling Free on an invalid address.
    
    Specifically, Crystal Reports component called Free on an address that has already been freed -- this is so called "double free" and produces a crash just like this one.  It is not allowed to call Free (clean up) same location twice.
    
    Error type: HEAP_FAILURE_BLOCK_NOT_BUSY
    Details:    The caller performed an operation (such as a free or a size check) that is illegal on a free block.
    
    
    Based on this callstack, the culprit appears to be:
    
    09 00000073`c485cb50 00007ffc`74825781 crdb_adoplus!<Module>.DbFieldInfo.{dtor}(DbFieldInfo*)+0x11
    0a 00000073`c485cb80 00007ffc`74825727 crdb_adoplus!<Module>.DbFreeFieldList(UInt32, DbFieldInfo**)+0x31
    
    
    This "makes sense" -- this is the Destructor for this DbFieildInfo in crdb_adoplus.dll  (they are usually used to do cleanup, so seeing a call to Free from there makes sense).
    
    
    Dump 2:
    ========
    
    w3wp__Isg-Edm-Repository__PID__5340__Date__09_14_2017__Time_11_33_08AM__936__Second_Chance_Exception_C0000374.dmp
    
    
    Debug session time: Thu Sep 14 11:33:11.000 2017 (UTC - 4:00)
    System Uptime: 1 days 4:12:13.985
    Process Uptime: 0 days 0:18:56.000
    
    Crashing callstack:
    
    # Child-SP          RetAddr           Call Site
    00 000000fa`151cca40 00007ffc`dcab4e42 ntdll!RtlReportCriticalFailure+0x8c
    01 000000fa`151ccb50 00007ffc`dcab5a40 ntdll!RtlpHeapHandleError+0x12
    02 000000fa`151ccb80 00007ffc`dca6a56f ntdll!RtlpLogHeapFailure+0xa4
    03 (Inline Function) --------`-------- ntdll!RtlpProbeUserBufferUnsafe+0x74ec3
    04 (Inline Function) --------`-------- ntdll!RtlpProbeUserBuffer+0x74ecd
    05 000000fa`151ccbb0 00000000`6cb8cabc ntdll!RtlFreeHeap+0x74eff
    06 000000fa`151ccc50 00007ffc`d3ec8757 msvcr90!free+0x1c
    07 000000fa`151ccc80 00007ffc`747fa132 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
    08 000000fa`151ccd20 00007ffc`748057e1 DomainBoundILStubClass.IL_STUB(Void*)+0x42
    09 000000fa`151ccde0 00007ffc`74805781 crdb_adoplus!<Module>.DbFieldInfo.{dtor}(DbFieldInfo*)+0x11
    0a 000000fa`151cce10 00007ffc`74805727 crdb_adoplus!<Module>.DbFreeFieldList(UInt32, DbFieldInfo**)+0x31
    0b 000000fa`151cce60 00007ffc`d3ec6aaa DomainBoundILStubClass.IL_STUB(Int32, Int64)+0x7
    0c 000000fa`151cce90 00000000`651f6b36 mscorwks!UMThunkStubAMD64+0x7a
    0d 000000fa`151ccf20 00000000`65273538 crqe+0x16b36
    0e 000000fa`151cd0d0 00000000`653b1c64 crqe+0x93538
    0f 000000fa`151cd3c0 00000000`653ac697 crqe+0x1d1c64
    10 000000fa`151cd430 00000000`65accab9 crqe+0x1cc697
    11 000000fa`151cd4d0 00000000`65ef0bc1 crpe32!CRPEConnectionInit+0x53859
    12 000000fa`151cd650 00000000`65b05500 crpe32!PESetBackgroundThreadProc+0xbf31
    13 000000fa`151cd6a0 00000000`65b061f7 crpe32!CRPEConnectionInit+0x8c2a0
    14 000000fa`151cda50 00000000`65b08b51 crpe32!CRPEConnectionInit+0x8cf97
    15 000000fa`151ce310 00000000`65b07fe7 crpe32!CRPEConnectionInit+0x8f8f1
    16 000000fa`151ce4a0 00000000`65b07a5b crpe32!CRPEConnectionInit+0x8ed87
    17 000000fa`151ce620 00000000`65b09479 crpe32!CRPEConnectionInit+0x8e7fb
    18 000000fa`151ceee0 00000000`6601edc5 crpe32!CRPEConnectionInit+0x90219
    19 000000fa`151cf470 00000000`65ee20f2 crpe32!MWCleanupProcess+0x10cb95
    1a 000000fa`151cf740 00000000`65ee63bd crpe32!CRPEConnectionInit+0x468e92
    1b 000000fa`151cf870 00000000`65ee627b crpe32!PESetBackgroundThreadProc+0x172d
    1c 000000fa`151cf9a0 00000000`65ee5ebf crpe32!PESetBackgroundThreadProc+0x15eb
    1d 000000fa`151cf9e0 00000000`65ee5c55 crpe32!PESetBackgroundThreadProc+0x122f
    1e 000000fa`151cfcc0 00000000`6633e919 crpe32!PESetBackgroundThreadProc+0xfc5
    1f 000000fa`151cfdf0 00000000`663c7b4f crpe32!SetHelpApp+0x9b109
    20 000000fa`151cfef0 00000000`663c7c03 crpe32!SetHelpApp+0x12433f
    21 000000fa`151cff20 00007ffc`dac613d2 crpe32!SetHelpApp+0x1243f3
    22 000000fa`151cff50 00007ffc`dc9d54e4 kernel32!BaseThreadInitThunk+0x22
    23 000000fa`151cff80 00000000`00000000 ntdll!RtlUserThreadStart+0x34
    
    Exact same situation as in previous dump.
    
    SUGGESTIONS:
    =============
    The issue is caused by a 3rd party component called crdb_adoplus.dll. We might need to contact support for Crystal Reports (Business Objects) and work with them further on fixing this issue.  Considering this is an older component, there may be newer builds that may have this issue addressed in them.
    
    

    Tuesday, September 19, 2017 8:07 AM
  • User1771544211 posted

    Hi pmcm,

    SUGGESTIONS: ============= The issue is caused by a 3rd party component called crdb_adoplus.dll. We might need to contact support for Crystal Reports (Business Objects) and work with them further on fixing this issue. Considering this is an older component, there may be newer builds that may have this issue addressed in them.

    Thanks for your feedback and sharing.

    Best Regards,

    Jean

    Wednesday, September 20, 2017 7:59 AM