none
Sharepoint 2013 portals down (TFS Integration) RRS feed

  • Question

  • Hi Team,

     

    We have an issue with Sharepoint integrated with TFS.  We run a large on-premise instance of TFS, 23 collection dbs totalling > 4TB in data, with > 5000 users. 

     

    We have 2 TFS App Tiers (primary/secondary) with Sharepoint installed on both.  Sharepoint URL is behind a load balancer and uses HTTPS (as is TFS url).  Sharepoint SQL databases are installed on a separate SQL instance to TFS collection databases.

     

    Issue:

    We had a port down alert on the Sharepoint SQL instance on 3rd September, which lasted 10 minutes.  After which, sharepoint stopped working.  After trying/checking various things we are now stuck and getting errors when trying to repair the TFS -> Sharepoint access grants via the TFS Admin console.

     

    Our support team can access the Sharepoint Central Admin site on the primary app tier server only.  However the portal sites are not available, and users receive an HTTP 500 error.  Further, we get HTTP 200 errors when trying to repair the connection with TFS and the sharepoint sites.

     

    Subset of log entries below:

     

    TFS Admin Console, trying to reconfigure Sharepoint Extensions:

     

    [Info   @17:44:15.794] Validating auto-configure for SharePoint Web application

    [Info   @17:44:15.872] Validation Pending:  The user is not a farm administrator.  Therefore, there must be an existing access grant between Team Foundation Server and SharePoint Products.

    [Warning@17:44:15.872]

    Exception Message: TF400324: Team Foundation services are not available from the server.

    Technical information (for administrator):

      The underlying connection was closed: An unexpected error occurred on a send. (type TeamFoundationServiceUnavailableException)

    Exception Stack Trace:    at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.ReadResponse(HttpWebResponse webResponse, WebException webException)

       at Microsoft.TeamFoundation.Client.Channels.TfsHttpWebRequest.SendRequest()

       at Microsoft.TeamFoundation.Client.Channels.TfsHttpRequestChannel.Request(TfsMessage message, TimeSpan timeout)

       at Microsoft.TeamFoundation.Client.Channels.TfsHttpClientBase.Invoke(TfsClientOperation operation, Object[] parameters, TimeSpan timeout, Object[]& outputs)

       at Microsoft.TeamFoundation.Client.SharePoint.SharePointTeamFoundationIntegrationService.CheckIsFarmAdministrator()

       at Microsoft.TeamFoundation.Admin.SharePointConfigurationHelper.ValidateAutoconfigureWebApplication(Uri adminUrl, Uri siteUrl, ITFLogger logger)

     

    TF255329: The following site could not be accessed: <site>. The server that you specified did not return the expected response. Either you have not installed the Team Foundation Server Extensions for SharePoint Products on this server, or a firewall is blocking access to the specified site or the SharePoint Central Administration site.

     

    TF255439: You do not have the permissions required to integrate with the following SharePoint Web application: <servername>. The validation check failed for the following user account: domain\service account.  This account must be a direct member of the Farm Administrators group on the web application.

     

     

    Sharepoint ULS Log:

    09/10/2019 15:37:39.67         w3wp.exe (0x2844)                               0x1854        SharePoint Foundation                 Claims Authentication                 fsq7        High            SPSecurityContext: Request for security token failed with exception: System.IO.FileLoadException: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)    Server stack trace:      at System.Net.HttpWebRequest.GetRequestStream()     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)     at System.S...        89dc029f-17f6-507b-b038-f8471ce19ad1

     

    Event Log:

    Event ID 8306

    An exception occurred when trying to issue a security token: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401).

     

     

    What we have tried:

    • Confirmed support user is a member of the Farm Administrators group.
    • Confirmed Sharepoint service account is a member of the Farm Administrators group.
    • Checked connectivity between Sharepoint SQL Server and TFS App Tiers (where Sharepoint is installed) for service account and support user account.
    • Checked SQL permissions on Sharepoint for service account and support user account.
    • Rebooted all servers.
    • Various IIS resets.
    • Checked SecurityTokenServiceApplicationPool is running.
    • Confirmed 'Enable Kernel-mode authentication' is checked against 'Sharepoint Web Services' site.
    • Database Status on Sharepoint Central Admin looks fine.
    • Our UAT test environment has the same stack, and does not have this issue.

     

    Our TFS Stack:

    • TFS 2017 Update 3
    • SharePoint Foundation 2013 with SP1
    • SQL Server 2012 Enterprise SP4 (sharepoint)
    • SQL Server 2016 Enterprise SP2 (TFS) in Always On configuration

     

     

    Please assist as many users are affected.

     

    Many thanks,

    Scott.

    Thursday, September 12, 2019 12:07 PM

All replies

  • Hi ScottRobertson,

    To troubleshoot the issue, check things below:

    1. Clear Team Foundation cache follow the steps below.

          1.1.Close all instances of Visual Studio.

          1.2.Open the Task Manager and check if any TFS Services are running. Select each of them and click on End Process Tree.

          1.3. Delete the contents from the following folders:

                 C:\Users\<<Your Alias>>\AppData\Local\Microsoft\Team Foundation

                 C:\Users\<<Your Alias>>\AppData\Local\Microsoft\VisualStudio

                 C:\Users\<<Your Alias>>\AppData\Local\Microsoft\VSCommon

    2. Check if any services related TFS are not started.

    3. Restart SQL.

    There is a similar post:

    https://social.msdn.microsoft.com/Forums/en-US/756079b0-d102-4c11-9985-43df1e61e3f2/tf400324-team-foundation-services-are-not-available-from-server?forum=tfsadmin

    More reference:

    If you have problems with TFS or Visual Studio flush the user cache … or not.

    Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

    Best regards,

    Sara Fan


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Friday, September 13, 2019 1:44 AM
    Moderator
  • Thank you Sara,

    TFS services are up and running fine. 

    It's the Sharepoint integration part we are having trouble with.

    We have managed to open a support case with MSFT, so will proceed through that. 

    Thanks for your help.

    Friday, September 13, 2019 6:57 AM
  • Hi ScottRobertson,

    If you have any update of the issue, you could share in the forum.

    It will help others have the same issue.

    Best regards,

    Sara Fan


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Friday, September 13, 2019 7:18 AM
    Moderator
  • Hi Sara,

    Just looping back on this incase anyone has a similar issue..

    The issue was down to a monitoring agent (AppDynamics in our case) causing issues with the IIS worker processes for Sharepoint 2013.

    Stopped the AppDynamics agent - and all is good in the world.

    Now working internally with AppDynamics team to configure the agent, so it does not affect Sharepoint app pools.

    Cheers.

    Thursday, September 19, 2019 3:30 PM
  • Hi ScottRobertson,

    Thank you for your sharing.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Friday, September 20, 2019 8:43 AM
    Moderator