none
Windows Azure Pack Preview - Subscription Active - Out of Sync Error

    Question

  • Hi,

    I have followed every step-by-step guide, every comment and blog post, and even broken many best practice rules to try get this software to work but still.... NOTHING!. I cannot create tenants in VMM and subscriptions never change from Out of sync state. I have reinstalled this software so many times I am just about over it. So... I reinstalled one more time, this time taking time to see what goes wrong where and the following event log errors are logged:

    1. When creating a test hosting Plan in the Admin Site

    Source: Microsoft-WindowsAzure-Server-Management-AdminApi
    EventID: 204
    
    GET
    Fanout
    https://hnwasspf01.xxx.dom:8090/provider/defaultquota
    NotFound
    Not Found
    StatusCode: NotFound , Code: , Message: GetDefaultQuotaAPI not supported.

    2. When trying to create a user / tenant with a subscription:

    Source: Microsoft-WindowsAzure-Server-Management-AdminApi
    EventID: 12
    
    HttpStatusCode: NotFound, ErrorCode: NotFound, ClientErrorMessage:Resource not found.
     ---> 
    Microsoft.WindowsAzure.Server.Management.Core.ManagementServiceException: Resource not found.
       at Microsoft.WindowsAzure.Server.Management.Core.SqlManagementStorageClient.HandleError(SqlException exception)
       at Microsoft.WindowsAzure.Server.Common.StorageClient.<ExecuteProcedureReaderAsync>d__6`1.MoveNext()
     <---
    
    3991233bbc0d4a72979d23d520a5ed1e.2013-08-29T00:48:54.9026442Z
    a07dfa69-a40f-4f90-9c2f-563a09888b50-2013-08-29 00:48:54Z
    XXXX\Dean
    <ESTIMATED> users/
    
    Source: Microsoft-WindowsAzure-Server-Management-AdminApi
    EventID: 204
    
    POST
    Fanout
    https://hnwasspf01.xxx.dom:8090/provider/subscriptions
    InternalServerError
    Internal Server Error
    StatusCode: InternalServerError , Code: , Message: Failed to create subscription. Reason: Message : An error occurred while processing this request., InnerMessage: <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml">
      <head>
        <title>Request Error</title>
        <style>BODY { color: #000000; background-color: white; font-family: Verdana; margin-left: 0px; margin-top: 0px; } #content { margin-left: 30px; font-size: .70em; padding-bottom: 2em; } A:link { color: #336699; font-weight: bold; text-decoration: underline; } A:visited { color: #6699cc; font-weight: bold; text-decoration: underline; } A:active { color: #336699; font-weight: bold; text-decoration: underline; } .heading1 { background-color: #003366; border-bottom: #336699 6px solid; color: #ffffff; font-family: Tahoma; font-size: 26px; font-weight: normal;margin: 0em 0em 10px -20px; padding-bottom: 8px; padding-left: 30px;padding-top: 16px;} pre { font-size:small; background-color: #e5e5cc; padding: 5px; font-family: Courier New; margin-top: 0px; border: 1px #f0f0e0 solid; white-space: pre-wrap; white-space: -pre-wrap; word-wrap: break-word; } table { border-collapse: collapse; border-spacing: 0px; font-family: Verdana;} table th { border-right: 2px white solid; border-bottom: 2px white solid; font-weight: bold; background-color: #cecf9c;} table td { border-right: 2px white solid; border-bottom: 2px white solid; background-color: #e5e5cc;}</style>
      </head>
      <body>
        <div id="content">
          <p class="heading1">Request Error</p>
          <p>The server encountered an error processing the request. See server logs for more details.</p>
        </div>
      </body>
    </html>
    
    Source: Microsoft-WindowsAzure-Server-Management-AdminApi
    EventID: 12
    
    HttpStatusCode: InternalServerError, ErrorCode: ErrorFromUnderlyingResourceProviders, ClientErrorMessage:One or more errors occurred while contacting the underlying resource providers. The operation may be partially completed.
     ---> 
    Microsoft.WindowsAzure.Server.Management.Core.ManagementServiceException: One or more errors occurred while contacting the underlying resource providers. The operation may be partially completed. Details: Failed to create subscription. Reason: Message : An error occurred while processing this request., InnerMessage: <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml">
      <head>
        <title>Request Error</title>
        <style>BODY { color: #000000; background-color: white; font-family: Verdana; margin-left: 0px; margin-top: 0px; } #content { margin-left: 30px; font-size: .70em; padding-bottom: 2em; } A:link { color: #336699; font-weight: bold; text-decoration: underline; } A:visited { color: #6699cc; font-weight: bold; text-decoration: underline; } A:active { color: #336699; font-weight: bold; text-decoration: underline; } .heading1 { background-color: #003366; border-bottom: #336699 6px solid; color: #ffffff; font-family: Tahoma; font-size: 26px; font-weight: normal;margin: 0em 0em 10px -20px; padding-bottom: 8px; padding-left: 30px;padding-top: 16px;} pre { font-size:small; background-color: #e5e5cc; padding: 5px; font-family: Courier New; margin-top: 0px; border: 1px #f0f0e0 solid; white-space: pre-wrap; white-space: -pre-wrap; word-wrap: break-word; } table { border-collapse: collapse; border-spacing: 0px; font-family: Verdana;} table th { border-right: 2px white solid; border-bottom: 2px white solid; font-weight: bold; background-color: #cecf9c;} table td { border-right: 2px white solid; border-bottom: 2px white solid; background-color: #e5e5cc;}</style>
      </head>
      <body>
        <div id="content">
          <p class="heading1">Request Error</p>
          <p>The server encountered an error processing the request. See server logs for more details.</p>
        </div>
      </body>
    </html>
       at Microsoft.WindowsAzure.Server.Management.Core.QuotaManager.ThrowFanoutException(ResourceProviderException resourceProviderException)
       at Microsoft.WindowsAzure.Server.Management.Core.QuotaManager.<CreateSubscriptionAsync>d__0.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.WindowsAzure.Server.Management.Core.Controllers.SubscriptionsControllerBase.<ProvisionSubscription>d__0.MoveNext()
     <---
    
    

    Any assistance with this would be good. For now, this build is seriously broken. Nothing I have done almost everything under the sun with permissions, Domain Admin rights, SysAdmin rights, Database DBO rights and NOTHING. The same error each time.

    Thanks

    Dean


    Thursday, August 29, 2013 1:49 AM

Answers

  • Hi,

    I have figured this out. The App Pool account needs to have access to the following local groups either directly or through domain group membership:

    • SPF_Admin
    • SPF_Provider
    • SPF_Usage
    • SPF_VMM

    This is NOT (clearly) documented anywhere and the most the documentation states is :

    Make sure that the application pool account exists in the domain and that it has sufficient permissions to manage the server. http://technet.microsoft.com/en-us/library/dn266007.aspx

    Anyway... happy days. And here is a great guide for the previous version... this DOES state how the app pool account must be managed. - http://www.hyper-v.nu/archives/marcve/2013/01/installing-and-configuring-system-center-service-provider-foundation/

    Thanks for the contributions.

    • Marked as answer by Dean Brighton Friday, October 04, 2013 12:15 PM
    Friday, October 04, 2013 12:15 PM

All replies

  • Hi Dean,

    Thanks for posting!

    According to your description, it seems the issue is related to Windows Azure Pack , I will move this thread to Windows Azure Pack Discussions forums to get a better support.

    Thanks for your support.


    Will
    <THE CONTENT IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED>
    Thanks
    MSDN Community Support

    Please remember to "Mark as Answer" the responses that resolved your issue. It is a common way to recognize those who have helped you, and makes it easier for other visitors to find the resolution later.

    Thursday, August 29, 2013 8:29 AM
    Moderator
  • are you able to visit the URL in events from browser?
    Thursday, August 29, 2013 11:16 AM
    Moderator
  • Yes. The Admin Site and Tenant Site are all good. When I create a tenant it does not get created in VMM and any subscription to a plan does not work. Always says Out of Sync.

    Thursday, August 29, 2013 11:48 AM
  • Hi Dean!

    I've alerted the SC/SPF folks about your post.  Someone should be able to assist soon.

    Thanks!

    Ryan

    Friday, August 30, 2013 12:35 AM
  • This is often a configuration issue.

    Please follow the steps below and see if any of these fix the issue.  If not, let's follow up offline.

    •        Enable SPF VMM, Admin, Provider IIS Application Pool identities running as domain user (not network service account)
    •        SPF Application Pool identity needs admin access to VMM and admin access on the SPF SQL DB
    •        Configure SPF IIS with Basic Authentication
    •        Create Local User on SPF Server, add to SPF Local Groups (VMM, Admin, Provider, Usage)
    •        Use the Local User to register with the Service Management Portal and API (not a domain user)
    •        No need to create any tenants from the SPF PowerShell cmdlets, this is handled automatically when users sign up for a subscription
    •        If using a service account, login to the SPF server with this account once

     

    Friday, August 30, 2013 5:25 PM
  •        Enable SPF VMM, Admin, Provider IIS Application Pool identities running as domain user (not network service account) [DB] - Confirmed. This has always been the case

    SPF Application Pool identity needs admin access to VMM and admin access on the SPF SQL DB  [DB] - Confirmed. Each time I have given AppPool account DBO rights. Is a member of SCVMM admins.

    Configure SPF IIS with Basic Authentication [DB] - Confirmed this is the default

    Create Local User on SPF Server, add to SPF Local Groups (VMM, Admin, Provider, Usage) [DB] - Confirmed. This used to  be a domain account but no difference.

    Use the Local User to register with the Service Management Portal and API (not a domain user) [DB] - As above, used to be domain account but local makes not difference.  

    No need to create any tenants from the SPF PowerShell cmdlets, this is handled automatically when users sign up for a subscription [DB] - This is not happening and no tenants are created using powershell either.

    If using a service account, login to the SPF server with this account once [DB] - Never did this one before and it makes no difference. Problem still there.

     


    Sunday, September 01, 2013 11:39 AM
  • First, please make sure you are using R2 version of SPF.

    And, for the URL in error message, can you access directly in browser?

    https://hnwasspf01.xxx.dom:8090/provider/defaultquota

    Friday, September 06, 2013 2:18 AM
    Moderator
  • I can confirm that I am installing SPF from the Orchestrator 2012 R2 Preview media. I can confirm that that URL returns a 404 and that the IIS logs provide the same infomation. Like the provider has no idea about that defaultquote call.
    Friday, September 06, 2013 8:57 AM
  • Hi Dean, you may consider post the question in system center forum.

    Friday, September 06, 2013 9:31 AM
    Moderator
  • Are you kidding? What in this whole thread suggests that this problem relates to System Center in general? What is happening over at Microsoft these days??!! You guys are slipping and you are letting your community down. At this rate, I may as well wait for the release version of this non-working product. I am working on significant private cloud deployments and a demonstration of this would secure significant work and licensing for Microsoft but cannot even get the smallest bit of functionality to work!!! Come on guys... This is VERY slack. If you have nothing directly to contribute to solving this case, please stay out of it. I am happy to submit a bug report for this but it appears as though Microsoft no longer do betas or any testing for that matter. Is there a Connect project for this solution? There have been so many issue with System Center products that I wonder how anyone could ever implement a private cloud solution without hitting some roadblocks along the way. It is very painful and dealing with Microsoft has become very painful too. Buck up Microsoft.
    Friday, September 06, 2013 10:57 AM
  • Hi Dean, its far from talking about bugs. WAP doesn't work as expected due to SPF returns a 404. To fix this issue, we need to look into why SPF returns 404. Follow this way, we can find out the cause and get this issue fixed.

    Monday, September 09, 2013 7:21 AM
    Moderator
  • Hi,

    I have figured this out. The App Pool account needs to have access to the following local groups either directly or through domain group membership:

    • SPF_Admin
    • SPF_Provider
    • SPF_Usage
    • SPF_VMM

    This is NOT (clearly) documented anywhere and the most the documentation states is :

    Make sure that the application pool account exists in the domain and that it has sufficient permissions to manage the server. http://technet.microsoft.com/en-us/library/dn266007.aspx

    Anyway... happy days. And here is a great guide for the previous version... this DOES state how the app pool account must be managed. - http://www.hyper-v.nu/archives/marcve/2013/01/installing-and-configuring-system-center-service-provider-foundation/

    Thanks for the contributions.

    • Marked as answer by Dean Brighton Friday, October 04, 2013 12:15 PM
    Friday, October 04, 2013 12:15 PM
  • From my experience I have configured it like this and with success!

    1. Install all WAP roles spread over different VMs with the default PreReqs

        Using the default certificates!

    2. The post-config of SPF needs special attention, see this link

        http://technet.microsoft.com/en-US/library/dn458596.aspx

        I have used 4 different domain user accounts during install of SPF

        and added 1 Local account for WAP2SPF (because people mentioned this)

    (In my first implementation of WAP 1.0, I have never used a Local Account. Only one domain account, as described by Marc van Eijk on Hyper-v.nu, see the link that Dean Brighton posted above)

    3. First of all it seems that it works, as I could register VMM via SPF and also see my configured cloud-resource within VMM. But when I created a plan, somehow I couldn’t get it to work. The Subscription and Plan where out-of-sync.

    As noted in the link above “Manage Web Services and Connections in Service Provider Foundation”

    - The domain account used for the VMM web service App-Pool on SPF needs to be a VMM Admin in SCVMM

    And this one I have overlooked:

    - The account used in the local SPF_Admin group for on SPF needs also to be a VMM Admin in SCVMM!! 

    This was written on this link: http://technet.microsoft.com/en-us/library/dn457804.aspx#BKMK_SPFAdmin

    “Requirements for using VM Clouds”

    Some extra info, I have not done these steps:

    - Never made changes to SQL or it DB (also never did this in WAP 1.0)

    - Never logged in once on the SPF server with the Local Account

    (Sometimes the syncing takes some time and the webpage does not tell me that it is finished. I have to force a page refresh)


    (So I see that this post is dated, but I wanted to share my experience)

    Friday, December 13, 2013 1:40 PM
  • I had this same issue on Azure Pack with SCVMM 2012 R2

    Called Microsoft tech support and the UR1 SQL script fixed the issue

    KB2904712

    /* script starts here */
    ALTER Procedure [dbo].[prc_RBS_UserRoleSharedObjectRelation_Insert]
    (
            @ID uniqueidentifier,
            @ObjectID uniqueidentifier,
            @ObjectType int,
            @RoleID uniqueidentifier,
            @UserOrGroup varbinary (85),
            @ForeignAccount nvarchar (256),
            @IsADGroup bit,
            @ExistingID uniqueidentifier = NULL OUTPUT
    )
    AS
    SET NOCOUNT ON
         SELECT @ExistingID = [ID] FROM [dbo].[tbl_RBS_UserRoleSharedObjectRelation]
         WHERE [ObjectID] = @ObjectID AND [RoleID] = @RoleID
      AND
      -- Select owner OR Select all which matches ForeignAccount or UserOrGroup OR
      -- both ForeignAccount and UserOrGroup is NULL 
      (([UserOrGroup] = @UserOrGroup OR [ForeignAccount] = @ForeignAccount) OR
      ([UserOrGroup] IS NULL AND @UserOrGroup IS NULL AND [ForeignAccount] IS NULL AND @ForeignAccount IS NULL))
          /* Ignore duplicate entries */
          IF (@ExistingID IS NULL)
          BEGIN
         INSERT [dbo].[tbl_RBS_UserRoleSharedObjectRelation]
                   ([ID]
                   ,[ObjectID]
                   ,[ObjectType]
                   ,[RoleID]
                   ,[UserOrGroup]
                   ,[ForeignAccount]
                   ,[IsADGroup]
                   ,[IsOwner]
                   )
        VALUES
        (
                @ID,
                @ObjectID,
                @ObjectType,
                @RoleID,                       
                @UserOrGroup,
                @ForeignAccount,
                @IsADGroup,
                0
        )
          END
    SET NOCOUNT OFF
    RETURN @@ERROR
    /* script ends here */

    Wednesday, March 12, 2014 6:18 PM
  • Hi All,

    Issue is related with bug in SCVMM 2012 R2. Azure portal v2 is fine here

    So,Apply 2 patches on VMM Server and one script need to be run on VMM Database.

    Sharing the details as well.

    http://support.microsoft.com/kb/2904712


    Kirpal Singh

    Saturday, March 15, 2014 7:34 AM