none
Intermittent authentication error on IIS-hosted WCF service after software deployment or server reboot RRS feed

  • Question

  • We have noticed this weird symptom in our production and QA environments for quite a while. We have been trying to understand the root cause but no luck so far.

    We have WCF services hosted under IIS. From time to time some services would become in accessible after server reboot or code deployment - the client gets System.ServiceModel.Security.MessageSecurityException with message "The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Negotiate,NTLM'.". The error would persist until the app pool is recycled or 'iisreset' is performed.

    The system information:

    OS Name:                   Microsoft Windows Server 2008 R2 Standard
    OS Version:                6.1.7601 Service Pack 1 Build 7601
    OS Manufacturer:           Microsoft Corporation
    OS Configuration:          Member Server
    OS Build Type:             Multiprocessor Free

    Below are the details:

      • The error normally happens following a server reboot for software install/upgrade/patch, or a software deployment (via BMC BladeLogic). But unfortunately we have not been able to manually reproduced the symptom by performing those actions.
      • When the error happens, it happens to just one application on the server; other applications on the same server work just fine. 
      • The error does not impact mex endpoints - WSDL can still be viewed in browser either locally or remotely while the client gets the authentication error on application endpoints. On the broken application, all application endpoints throw the authentication error.  
      • The client gets the authentication error regardless of where it is run, either locally or remotely. 
      • The client get the authentication error regardless if a valid URL is used or not - e.g. "http://localhost/PaymentGateway/PaymentService.svc" is the valid URL, but the client get the same authentication error with an invalid URL such as "http://localhost/PaymentGateway/xxxxx.svc".
      • Each IIS application has a dedicated app pool. Let's say application A uses app pool A, application B uses app pool B; assume application A is broken and is giving authentication error, application B is working. If I switched application A to app pool B, the error went away. When I switched it back to app pool A the error happened again. However if I switched application B to use app pool A, application B continues to work without the error. (I cannot explain this. This is weird). 
      • Each application has both Anonymous authentication and Windows authentication enabled in IIS, but the client that gets the authentication error only uses basic HTTP binding. When I disabled Windows Authentication in IIS on the broken application, the client stops getting the authentication error; after I re-enabled Windows Authentication in IIS, the client continues to work. 

    I did some research and found other reportings of similar symptoms but unfortunately no sure answer about the root cause:

    1. http://www.get-vm.com/blog/2015/08/20/unable-to-expireorpower-on-vra-managed-machine  reported authentication error after a VM is powered on and the use of iisreset command. It mentioned the order of NTLM and Negotiate providers but my tests show that either order works for me, so I guess the key is iisreset command.
       
    2. http://stackoverflow.com/questions/9686186/iis-application-using-application-pool-identity-loses-primary-token has report of the same symptom in the comment section, the difference from us is our WCF services use NetworkService identity.  
    Nice find! We have an issue when we deploy our WCF service that it starts responding with 401 on calls with negotiate authentication. We stop iis when we deploy and start it after the files are copied. After we start the service it responds with 401 to some calls (which worked before iis stop) The calls are coming with negotiate authentication. A reboot solves this. The WCF services uses app pool identity. I think it may be the same root cause. – mortb Oct 7 '14 at 12:18

    We'll keep monitoring our production and QA environments and update with new findings. 




    Tuesday, March 29, 2016 1:49 PM

All replies

  • Hello,

    >>"The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Negotiate,NTLM'.".

    Based on the error information, it seems that the client does not send the correct credential. As you said the each application has both Anonymous authentication and Windows authentication enabled, could you please try to disable the Anonymous authentication to only enable the Windows authentication?

    Then could you please try to share your service's and client's config file in here?
    It will be better if you can also check your IIS log to see if you can get some helpful information about the issue.

    Best Regards,
    Amy Peng


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, March 31, 2016 6:23 AM
    Moderator
  • Thanks Amy.

    First of all, the IIS log that is related to the error has something like:

    2016-03-28 15:24:51 ::1 POST /PaymentGateway/PaymentGatewayAdmin.svc client=CSG&environment=AC8&database=PM 80 - ::1 - 401 2 5 708
    2016-03-28 15:24:51 ::1 POST /PaymentGateway/PaymentGatewayAdmin.svc client=CSG&environment=AC8&database=PM 80 - ::1 - 401 2 5 283


    When I turn off Anonymous authentication and leave only Windows authentication on, my test client would get:  (MessageSecurityException) The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'Negotiate,NTLM'.

    While the real client has configuration file, the test client does not use a configuration file - it uses BasicHttpBinding.

    Please use the link below to get the <system.serviceModel> section of the server configuration file:

    https://onedrive.live.com/redir?resid=AFEA388C5FB59113!111&authkey=!AGpXlSijc7HLrG0&ithint=file%2cconfig


    Thursday, March 31, 2016 3:28 PM
  • I did some further investigation. One instance of the authentication error appeared to have started following an IISRESET - application A worked normally after its own deployment for over 20 minutes; there were some other applications deployments after A's deployment and some deployments invoked IISRESET. Application A survived several IISRESET but got into the authentication error after another IISRESET.

    So it appears that certain condition (race condition?) may cause an IIS application (app pool) to lose the configured Anonymous authentication when IIS is started (either IISRESET or system restart).

     
    Monday, April 4, 2016 6:26 PM
  • Hello,

    >>Application A survived several IISRESET but got into the authentication error after another IISRESET

    After meeting the authentication error, have you tried to check the IIS settings to see if anything is changed? As far as I known, the IISRESET may lose settings/configuration in IIS.

    For more information, please try to refer to the following article:
    https://support.microsoft.com/en-sg/kb/286196 .

    Since this issue is more related with the IIS, it will be better if you can try to post your question in the IIS forum for better support:

    #IIS forums:
    http://forums.iis.net/ .

    Thanks for your understanding.

    Best Regards,
    Amy Peng


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Thursday, April 7, 2016 2:12 AM
    Moderator