IIS App Domain restarting unexpectedly

Question

User993979300 posted

I have web application hosted on IIS.   Asp.Net application with handlers made of "HttpTaskAsyncHandler" implemented. 

We recently facing the where the IIS App domain getting restarted automatically with out anything configured on iis level.  

1. No error logs in application level, but it was clear that the application restarted and initialized
2. No error logs in EventViewer
3. No sign of app pool recycle. 
4. Process Id not changed after getting restarted.  
5. No logs event after enabling all possible logs on App pool advanced settings
6. DebugDiag not capturing this as crash.

We are clueless on this and the issue started recently only. 

Can you advice how to debug/fix this issue.

Thanks in advance 

Answer 1

User753101303 posted

Hi,

So it doesn't seems an application pool recycling. To start with which bad behavior do you see when it happens? This is following a code change?.

Edit: "It was clear that the application restarted and initialized" How do you know that?. Actually it is even excepted if the application is idle for more than 20 minutes by default if I remember.

Answer 2

User993979300 posted

Hi, 

Thanks for the reply. 

How do you know that?

So, the  restarted and initialized was confirmed based on the logs where we do initilaization, it clearly shows that the app got started and initialized.  

Actually it is even excepted if the application is idle for more than 20 minutes by default if I remember.

We definitely know there are bunch of active requests at that time and IIS logs clearly shows that

Answer 3

User475983607 posted

If I understand your custom initialization code logs a restart but the IIS logs and event viewer do not show the application restarted?

Answer 4

User753101303 posted

Ok so you are 100% sure that Application_Start is called (not something called from Applcation_Start and that could be called maybe from somewhere else, really Application_Start) and without any recycling events (seems you told all recycling events are configured at the IIS level to write a Windows log entry and it seems consistent with the unchanged process ID).

I wonder if using AppDomain.Unload could cause this kind if behavior (but never messed with that myself). My first move would be perhaps to check if AppDomaon is used somewhere ???

Also you talked avout HttpTaskAsyncHandler? it happens since using this or maybe since some other code change? Seems weird.

Answer 5

User303363814 posted

What is happening in your async handlers?

Does anything in

How to run Background Tasks in ASP.NET - Scott Hanselman's Blog

or

The Dangers of Implementing Recurring Background Tasks In ASP.NET | You’ve Been Haacked

apply to your situation

Answer 6

User993979300 posted

Exactly!

Answer 7

User993979300 posted

If I understand your custom initialization code logs a restart but the IIS logs and event viewer do not show the application restarted?

Exactly!

Answer 8

User993979300 posted

What is happening in your async handlers?

Async handlers continuous to work and the restart happend in a fraction of seconds,  promblem is Im holding some user information which get lost because of the restart. 

I have collected some dump on the Restart. here it that

[11/26/2020 10:57:32 AM]
  System.Globalization.CultureNotFoundException on thread 2040. DetailID = 20
  Thread created. New thread - System ID: 9672
  System.Exception on thread 2040. DetailID = 21
  System.TypeInitializationException on thread 2040. DetailID = 22

This is the first point of exception but "CultureNoTFoundException"?  what is that?  I haven't used the any culture related in my code. 

Answer 9

User993979300 posted

Here is the full trace for DetailID:20 mentioned above,

DetailID = 20
    Count:    1
    Type:     System.Globalization.CultureNotFoundException
    Message:  Culture is not supported.
    Stack:    
        [HelperMethodFrame]
        System.Globalization.CultureInfo..ctor(System.String, Boolean)
        System.Web.HttpServerUtility.CreateReadOnlyCultureInfo(System.String)
        System.Web.UI.Util.IsCultureName(System.String)
        System.Web.Compilation.StandardDiskBuildResultCache.FindSatelliteDirectories()
        System.Web.Compilation.BuildManager.RegularAppRuntimeModeInitialize()
        System.Web.Compilation.BuildManager.Initialize()
        System.Web.Compilation.BuildManager.InitializeBuildManager()
        System.Web.HttpRuntime.HostingInit(System.Web.Hosting.HostingEnvironmentFlags, System.Security.Policy.PolicyLevel, System.Exception)
        System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
        System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
        [GCFrame]
        [GCFrame]
        [ContextTransitionFrame]
        [GCFrame]
        [GCFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [TPMethodFrame]
        System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
        DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
        [ComMethodFrame]

Answer 10

User993979300 posted

Here is the stacktrace for Error DetailID:20  

DetailID = 20
    Count:    1
    Type:     System.Globalization.CultureNotFoundException
    Message:  Culture is not supported.
    Stack:    
        [HelperMethodFrame]
        System.Globalization.CultureInfo..ctor(System.String, Boolean)
        System.Web.HttpServerUtility.CreateReadOnlyCultureInfo(System.String)
        System.Web.UI.Util.IsCultureName(System.String)
        System.Web.Compilation.StandardDiskBuildResultCache.FindSatelliteDirectories()
        System.Web.Compilation.BuildManager.RegularAppRuntimeModeInitialize()
        System.Web.Compilation.BuildManager.Initialize()
        System.Web.Compilation.BuildManager.InitializeBuildManager()
        System.Web.HttpRuntime.HostingInit(System.Web.Hosting.HostingEnvironmentFlags, System.Security.Policy.PolicyLevel, System.Exception)
        System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
        System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
        [GCFrame]
        [GCFrame]
        [ContextTransitionFrame]
        [GCFrame]
        [GCFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [GCSafeCollectionFrame]
        [TPMethodFrame]
        System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
        System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
        DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
        [ComMethodFrame]

Answer 11

User1535942433 posted

Hi Gopichandarbe,

As far as I think, you could examine the thrown CultureNotFoundException's InvalidCultureName property. This will tell you which Culture is trying be used but is not found on the system.

More details,you could refer to below articles:

https://dejanstojanovic.net/aspnet/2018/february/reasons-for-aspnet-application-restarts-on-iis-server/

https://stackoverflow.com/questions/24332304/why-do-i-get-culture-is-not-supported-and-what-if-anything-should-i-do-abou

Best regards,

Yijing Sun

Answer 12

User753101303 posted

Not sure how this is related. Does it prevent your app to run?

For now I would still investigate about which exact thing happens. For example I see https://docs.microsoft.com/en-us/dotnet/api/system.web.applicationshutdownreason?view=netframework-4.8 that you might be able to track from the Application_End event.

You are sure you are not just wrting files under your web site or if the primary thing you see is a "user information being lost" this is not just a browser session expiration (this information is in a Session variable ???)

Edit: I gave this a quick try and it seems to work. So I would start by seeing if Application_End happens and which value I have for  System.Web.Hosting.HostingEnvironment.ShutdownReason

For example a web.config file change shows "ConfigurartionChange" and it seems the behavior is similar (the process id is unchanged but AppDomain.Id is incremented).

Answer 13

User475983607 posted

A few things we're seen in the past that cause restarts are uploading or saving files in the bin folder, shared hosting, memory leaks.  

Other than that, the usual suspect is a code change.  Look through source control for the latest code changes.  

Answer 14

User-943250815 posted

In addition to all suggestions and if apply, why not use global.asax and place some routine to write your own log in a text file, here you can find a list of events and when it is fired https://www.techrepublic.com/article/working-with-the-aspnet-globalasax-file/

Also in Application_End event you can get "ApplicationShutdownReason" Enumerator, you can find more about here https://docs.microsoft.com/en-us/dotnet/api/system.web.applicationshutdownreason?redirectedfrom=MSDN&view=netframework-4.7.2

A sample of what I use in Application_End, you can reproduce for each event as you consider necessary

Sub Application_End(ByVal sender As Object, ByVal e As EventArgs)
  Dim myShutdownReason As ApplicationShutdownReason = Hosting.HostingEnvironment.ShutdownReason
Dim myReason As String = myShutdownReason.ToString
WriteMyLog(myReason)
End Sub

Private Sub WriteMyLog(ByVal myLogStr As String)
    Dim DirBase As String = HttpRuntime.AppDomainAppPath                                           ' Application Dir
    Dim DirLog As String = IO.Path.GetFullPath(DirBase & "\log\")                          ' Log Directory
    Dim LogFile As String = DirLog & "MyApplication.log"                               ' Full path log file
    Dim DH As String = Now.ToString("yyyy-MM-ddTHH:mm:ss.fffzzz")                                  ' Date Time
    Using sw As IO.StreamWriter = IO.File.AppendText(LogFile)                                   
      sw.WriteLine(DH & " > " & myLogStr )                                                           
    End Using
  End Sub