none
Repository has more than 10 non-decryptable secrets backups (host). RRS feed

  • Question

  • Hi,

    The function worked perfectly, as soon as put a bit load on it with a small load test I got this very strange error on my function-app deployments. Has anyone encountered this error before? Any ideas?

    Thanks and kind regards AndiP 

    Error:

    The function runtime is unable to start. Microsoft.Azure.WebJobs.Script.WebHost: Repository has more than 10 non-decryptable secrets backups (host).

    Session Id: 04cf7019749e4a77a050141bc1496c36

    Timestamp: 2018-04-01T21:23:36.149Z

    Sunday, April 1, 2018 9:34 PM

Answers

  • Ok, I figured out the root of the issue: both your -ne and your -we apps have their WEBSITE_CONTENTSHARE set to the -ne name (maybe it was a typo?). So they end up conflicting for secrets.

    If you change the -we app to use a different content share, this will resolve the issue.

    David

    • Proposed as answer by David Ebbo Monday, April 2, 2018 5:46 PM
    • Marked as answer by Andreas Pollak Tuesday, April 3, 2018 8:12 AM
    Monday, April 2, 2018 5:40 PM

All replies

  • Stacktrace of the log-streaming service...

    2018-04-01T22:01:41  Welcome, you are now connected to log-streaming service.
    2018-04-01T22:02:33.938 [Info] Reading host configuration file 'D:\home\site\wwwroot\host.json'
    2018-04-01T22:02:34.366 [Info] Host configuration file read:
    {}
    2018-04-01T22:02:34.507 [Verbose] Host configuration applied.
    2018-04-01T22:02:34.507 [Info] Starting Host
    (HostId=****myazurefunction****, Version=1.0.11612.0, InstanceId=6e6e98dc-b814-4b35-8ee4-efe25951b26d, ProcessId=7928, AppDomainId=2, Debug=True, ConsecutiveErrors=0, StartupCount=1, FunctionsExtensionVersion=~1)
    2018-04-01T22:02:34.726 [Verbose] Debug file watch initialized.
    2018-04-01T22:02:34.741 [Verbose] File event source initialized.
    2018-04-01T22:02:34.773 [Verbose] Created directory snapshot.
    2018-04-01T22:02:35.616 [Verbose] Function metadata read.
    2018-04-01T22:02:35.695 [Verbose] Binding providers loaded.
    2018-04-01T22:02:36.382 [Verbose] Binding providers initialized.
    2018-04-01T22:02:36.398 [Info] Loaded custom extension 'BotFrameworkConfiguration'
    2018-04-01T22:02:36.420 [Info] Loaded custom extension 'SendGridConfiguration'
    2018-04-01T22:02:36.420 [Info] Loaded custom extension 'EventGridExtensionConfig'
    2018-04-01T22:02:36.101 [Info] Executing HTTP request: {
      "requestId": "279f5c21-988c-4e4c-8d0e-335bd8509cb0",
      "method": "GET",
      "uri": "/"
    }
    2018-04-01T22:02:36.101 [Info] Executing HTTP request: {
      "requestId": "1b6269c7-2e58-4889-9d0b-387d7c731bc5",
      "method": "GET",
      "uri": "/"
    }
    2018-04-01T22:02:36.782 [Verbose] Extension loading complete.
    2018-04-01T22:02:39.632 [Error] A ScriptHost error has occurred
    2018-04-01T22:02:39.632 [Error] System.InvalidOperationException : Repository has more than 10 non-decryptable secrets backups (host).
       at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.PersistSecretsAsync[T](T secrets,String keyScope,Boolean isNonDecryptable) at
          C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.cs : 422
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at async Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.GetHostSecretsAsync() at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.cs : 92
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at async Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetOrCreateExtensionKey(String extensionName) at
      C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.cs : 71
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetExtensionWebHookRoute(String extensionName) at
      C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.cs : 64
       at Microsoft.Azure.WebJobs.Extensions.EventGrid.EventGridExtensionConfig.Initialize(ExtensionConfigContext context)
       at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders(ExtensionConfigContext context)
       at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices(JobHostConfiguration config)
       at Microsoft.Azure.WebJobs.JobHost.InitializeServices()
       at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider(JobHost host) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.cs : 362
       at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize() at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs : 496
       at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.cs : 184
    2018-04-01T22:02:39.741 [Info] Stopping Host

    Sunday, April 1, 2018 10:10 PM
  • Hi Andi,

    A puzzling error indeed.

    I don't quite understand it and will ask others to shine in. It seems related to this issue. For now, try Paul Batum's steps in that same issue.

    I think it may result from having multiple apps try to share the same content share. It should not be related to the load you put on it though, at least in theory.

    thanks,
    David

    Sunday, April 1, 2018 10:25 PM
  • Indeed, I checked your app and you have all those secret snapshot files in d:\home\data\Functions\secrets. Cleaning them out (or moving them somewhere else) will likely get you around that error, but we need to understand how it gets into that state.

    Also, the fact that the snapshot files large and growing secret size (with every copy) is very interesting. Are you doing anything related to the host/function keys?

    Sunday, April 1, 2018 10:41 PM
  • Ok, I figured out the root of the issue: both your -ne and your -we apps have their WEBSITE_CONTENTSHARE set to the -ne name (maybe it was a typo?). So they end up conflicting for secrets.

    If you change the -we app to use a different content share, this will resolve the issue.

    David

    • Proposed as answer by David Ebbo Monday, April 2, 2018 5:46 PM
    • Marked as answer by Andreas Pollak Tuesday, April 3, 2018 8:12 AM
    Monday, April 2, 2018 5:40 PM
  • Thanks so much David! You saved my day.

    For some reasons my deployment script modified the WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE applications settings and set them to the same value in both regions.

    I also have a theory on the load-test scenario. I had the app deployed for at least a week before I started the load-test. A larger number of requests worked perfectly before it went to Unavailable and Function host not starting. The reason might be that the Test-Node had been getting the same region for a while (Trafficmanager) and after flipping the regions the conflict with the keys started to occur.

    Monday, April 2, 2018 11:29 PM