Service Fabric nodes not registering with the cluster RRS feed

  • Question

  • Hello,

    We modified the base ARM template (from Azure) to create a cluster with Application Gateway, RDP Jumpbox, NSG, Cluster Security, AD Security, Virtual Network Gateway and 2016-Datacenter-with-Containers. It was working correctly when we were deploying these resources in increment. 

    We then deleted the Resource Group to test the template when starting from scratch. Nodes were note registering with the cluster. In the Application Log of one of the node we can see the following error:

    "ERROR: System.NullReferenceException: Object reference not set to an instance of an object.
       at Microsoft.Azure.ServiceFabric.Extension.Core.NodeBootstrapAgent.GetNodeDescription(AzureContainerInformation containerInfo)
       at Microsoft.Azure.ServiceFabric.Extension.Core.NodeBootstrapAgent.<RunOnce>d__f.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at Microsoft.Azure.ServiceFabric.Extension.Core.NodeBootstrapAgent.<RunAsync>d__0.MoveNext()"

    Our investigation led us to the azure cluster endpoint "https://eastus2.servicefabric.azure.com/runtime/clusters/<clusterGuid>"
    Calling this Url manually is returning us a 4 bytes json response with the string "null" as it content. (not to be confused with an empty response when providing the wrong certificate)

    So we proceeded to dumb down our template a lot, removing NSG, AD Security and certificates. Still getting the same Null exception.

    So without providing too much detail about our ARM template, our question is: In which scenario can the cluster endpoint mentioned above return the following string : "null". Even with security turned off.

    And how can we troubleshoot this error. The ARM template does not return any error. No errors are visible in the Azure Portal for the Service Fabric Cluster resource (it is only stating "Waiting for nodes")

    Thanks in advance,

    • Edited by Math Friday, February 17, 2017 10:49 PM
    Friday, February 17, 2017 10:42 PM