The following forum(s) have migrated to Microsoft Q&A (Preview): Azure Service Fabric!
Visit Microsoft Q&A (Preview) to post new questions.

Learn More

 locked
Restart Service Fabric Cluster RRS feed

  • Question

  • Hi,

    We have a service fabric cluster deployment among 3 VMs without internet connection.

    Within service fabric explorer, the warning message comes out

    Unhealthy event: SourceId='System.UpgradeOrchestrationService', Property='ClusterGoalStateReachable', HealthState='Warning', ConsiderWarningAsError=false.
    Goal state is unreachable. Please check cluster internet connectivity or disable autoprovisioning of cluster upgrades by updating the cluster configuration.

    I turned off the "Service Fabric Host Service" on each VM, changed ClusterConfig.json to add "fabricClusterAutoupgradeEnabled": false under the "properties", and restarted all of the three "Service Fabric Host Service".

    However, the same warning message is still there.

    Did I restart the service fabric cluster correctly for turning on/off the "Service Fabric Host Service"? Why does the warning not go away?

    Thanks

    Friday, November 25, 2016 3:01 PM

Answers

  • Use 'Reset Local Cluster' not restart "Service Fabric Host Service" on each VM.

    Frank

    Saturday, November 26, 2016 1:10 AM
  • 'Local Cluster Manager' in System Tray.

    Frank

    Monday, November 28, 2016 3:43 PM
  • Currently, the warning will persist for 48 hours after disabling "fabricClusterAutoupgradeEnabled" through config upgrade, after which it will expire.

    The refresh interval is tracked to be fixed in a coming release, such that the warning goes away once the config upgrade completes.

    Tuesday, November 29, 2016 1:10 AM
  • Try running

    Get-ServiceFabricClusterConfiguration

    and see if it reflects the fabricClusterAutoupgradeEnabled setting value. If it doesn't, run a config upgrade using the updated config.

    Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath config.json

    Edit: After you update this value you will need to wait 48 hours for the warning to go away.
    Tuesday, November 29, 2016 6:57 PM

All replies

  • Try change ClusterConfig.json to add "fabricClusterAutoupgradeEnabled": false under the "properties and reset cluster.

    Frank

    Friday, November 25, 2016 5:17 PM
  • That's what I did as described in the original post. And the warning message is still there after restart "Service Fabric Host Service" on each VM.
    Friday, November 25, 2016 7:27 PM
  • Use 'Reset Local Cluster' not restart "Service Fabric Host Service" on each VM.

    Frank

    Saturday, November 26, 2016 1:10 AM
  • What is 'Reset Local Cluster' not restart'?  it's not a service fabric cmdlet

    Where could I find it?

    Saturday, November 26, 2016 3:45 PM
  • 'Local Cluster Manager' in System Tray.

    Frank

    Monday, November 28, 2016 3:43 PM
  • We don't have this "Local Cluster Manager" in our UAT service cluster.

    Do we have other ways to do it? The VMs are restarted over weekend but the the warning messages are still there.

    Does ClusterConfig.json need to be renamed after add "fabricClusterAutoupgradeEnabled": false ?

    Monday, November 28, 2016 9:12 PM
  • Currently, the warning will persist for 48 hours after disabling "fabricClusterAutoupgradeEnabled" through config upgrade, after which it will expire.

    The refresh interval is tracked to be fixed in a coming release, such that the warning goes away once the config upgrade completes.

    Tuesday, November 29, 2016 1:10 AM
  • We made the changes on last Friday and the warnings are still there.

    Any thing else I could try to see where the problem is?

    Tuesday, November 29, 2016 9:50 AM
  • Try running

    Get-ServiceFabricClusterConfiguration

    and see if it reflects the fabricClusterAutoupgradeEnabled setting value. If it doesn't, run a config upgrade using the updated config.

    Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath config.json

    Edit: After you update this value you will need to wait 48 hours for the warning to go away.
    Tuesday, November 29, 2016 6:57 PM
  • The FabricClusterAutoupgradeEnabled  shows True from Get-ServiceFabricClusterConfiguration.

    When I run Start-ServiceFabricClusterConfigurationUpgrade, the exception is generated below.

    PS C:\servicefabric> Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath .\ClusterConfigV2.json
    Start-ServiceFabricClusterConfigurationUpgrade : Exception of type 'System.Exception' was thrown.
    At line:1 char:1
    + Start-ServiceFabricClusterConfigurationUpgrade -ClusterConfigPath .\C ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (Microsoft.Servi...usterConnection:ClusterConnection) [Start-Ser
       viceFa...gurationUpgrade], Exception
        + FullyQualifiedErrorId : StartClusterConfigurationUpgradeErrorId,Microsoft.ServiceFabric.Powershell.Sta
       rtClusterConfigurationUpgrade

    Also checked the MSDN and can't see there is a switch for cluster connection.

    https://docs.microsoft.com/en-us/powershell/servicefabric/vlatest/Start-ServiceFabricClusterConfigurationUpgrade?redirectedfrom=msdn

    Tuesday, November 29, 2016 9:06 PM
  • Hi Max,

    Could you please help the issue for running Start-ServiceFabricClusterConfigurationUpgrade below?

    I already run  "Connect-ServiceFabricCluster jdev-p-v-brs01:19000" but Start-ServiceFabricClusterConfigurationUpgrade  keeps complaining the connection exception.

    Tuesday, December 6, 2016 5:04 PM