locked
Stateless Web API service RRS feed

  • Question

  • Hi,

    I followed this tutorial and created the application and service.

    https://azure.microsoft.com/en-gb/documentation/articles/service-fabric-reliable-services-communication-webapi/

    After deployed the application into a cluster with 6 nodes, can see from the Service Fabric Explorer that 1 node is in Ready status while the other 5 are in In Build status, and all their Health State are OK.

    However, after a couple of minutes, the Health State of the Nodes with In Build status changes to Warning or Unknown, then the following messages are printed out.

    Any idea what is happening here? Not sure it's the code issue or cluster set up issue.

    Unhealthy replicas: 83% (5/6), MaxPercentUnhealthyReplicasPerPartition=0%.
    Unhealthy replica: PartitionId='9b68ed16-d90f-493f-98a6-731d17ede19f', ReplicaOrInstanceId='131229855213693831', AggregatedHealthState='Warning'.
    Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
    Replica had multiple failures in API call: IStatelessServiceInstance.Open(); Error = System.Reflection.TargetInvocationException (-2146232828)
    Exception has been thrown by the target of an invocation.
       at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
       at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
       at Microsoft.Owin.Hosting.ServerFactory.ServerFactoryAdapter.Create(IAppBuilder builder)
       at Microsoft.Owin.Hosting.Engine.HostingEngine.Start(StartContext context)
       at WebService.OwinCommunicationListener.OpenAsync(CancellationToken cancellationToken) in C:\WebApplication\WebService\OwinCommunicationListener.cs:line 116
       at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__10.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
       at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__0.MoveNext()

    Monday, November 7, 2016 10:00 AM

Answers

  • That error usually means that there is some problem opening the communication listener. Usually in this case it is because the port is already occupied, or because something is preventing the port from being acld correctly or the service. You can look at your version of OwinCommunicationListener on line 116 to see exactly what the issue is.

    Incidentally, do you have a static port defined for this endpoint, and is this a local cluster on a single machine? If that's the case then yeah only one instance would win the race, at which point it would have sole control over that port on that box and all the other instances wouldn't be able to come up.

    Tuesday, November 8, 2016 11:48 PM
  • You just leave the static port out of the config (so declare the endpoint but leave out the port). Services in Service Fabric are made discoverable through the naming service - the stable name (fabric:/app/service) gets mapped to the addresses the service is listening to at runtime, and clients can use an API (via HTTP or C# or whatever) to discover the listening addresses and direct their calls there. Think of it like some super basic DNS.  

    There's almost always some stateless gateway listening on a known port (80;443) which then performs this lookup and then forwards the calls on. If you go through this lab it will walk you through some examples.

    Friday, November 11, 2016 7:58 PM
  • Stratford. That is what I would do.  When your in dev, then deploy using the 1node cluster.  If you need to deploy to 5 nodes in dev, then just understand that 4 nodes will fail (no big deal if your expecting it).  When you in production on 5 nodes, then you need a load balancer in front like normal.  In azure, that would be the azure LB. Maybe same thing available in Azure stack.  Or use your own locally.  Adding another gateway in front seems to only add another layer of overhead. 

    Sunday, November 13, 2016 6:00 AM

All replies

  • That error usually means that there is some problem opening the communication listener. Usually in this case it is because the port is already occupied, or because something is preventing the port from being acld correctly or the service. You can look at your version of OwinCommunicationListener on line 116 to see exactly what the issue is.

    Incidentally, do you have a static port defined for this endpoint, and is this a local cluster on a single machine? If that's the case then yeah only one instance would win the race, at which point it would have sole control over that port on that box and all the other instances wouldn't be able to come up.

    Tuesday, November 8, 2016 11:48 PM
  • Thanks Matt. We are using a static port on a single VM.

    How could it be configured to use dynamic port for the stateless service? And How will the client know which port to connect to?

    Friday, November 11, 2016 4:51 PM
  • You just leave the static port out of the config (so declare the endpoint but leave out the port). Services in Service Fabric are made discoverable through the naming service - the stable name (fabric:/app/service) gets mapped to the addresses the service is listening to at runtime, and clients can use an API (via HTTP or C# or whatever) to discover the listening addresses and direct their calls there. Think of it like some super basic DNS.  

    There's almost always some stateless gateway listening on a known port (80;443) which then performs this lookup and then forwards the calls on. If you go through this lab it will walk you through some examples.

    Friday, November 11, 2016 7:58 PM
  • Hi Matt,

    Thanks for reply.

    The WebAPI application in question is actually our front end that offers to clients to call.

    By leaving out the static port config to make it dynamic, do we have to add the stateless gateway?

    You Lab 1 documentation mentioned "Lab 2 will cover stateless gateway pattern and placement constrains".

    Is Lab 2 available for download somewhere to show stateless gateway pattern?

    We are deploying the application into our on premises cluster. How could the load balancer be configured?

    Ideally, we can deploy the WebAPI application into multiple physical machine in cluster with same port number and load balance among physical node. 


    Friday, November 11, 2016 9:44 PM
  • Stratford. That is what I would do.  When your in dev, then deploy using the 1node cluster.  If you need to deploy to 5 nodes in dev, then just understand that 4 nodes will fail (no big deal if your expecting it).  When you in production on 5 nodes, then you need a load balancer in front like normal.  In azure, that would be the azure LB. Maybe same thing available in Azure stack.  Or use your own locally.  Adding another gateway in front seems to only add another layer of overhead. 

    Sunday, November 13, 2016 6:00 AM
  • Hi Matt,

    Would it be possible to share some information for the "stateless gateway pattern and placement constrains"?

    We would like to make a decision about whether to use F5 load balancer to balance the WebAPI entry point or build the gateway you mentioned?

    Thanks

    Wednesday, November 16, 2016 5:19 PM
  • Hi William,

    Thanks for reply.

    I am quite new to load balance.

    If we publish the WebAPI into the cluster with dynamic port number assigned, how could the load balancer be configured to find the correct path on each node? believe the port numbers will be changed if the cluster is restarted.

    Wednesday, November 16, 2016 5:21 PM
    • Almost all of the service fabric samples show a stateless gateway service serving http and proxying calls from clients to the actual stateful services using Resolve. Please take a look at those.
    • Placement constraints
    • Even if you build the gateway you will need something then to balance traffic over the stateless gateways. But that still would be good because you really don't want to have to use static ports since it will limit your flexibility. You would configure the LB to point to the gateway which would run on a known/static port, but then all your other services should allocate ports dynamically.
    Wednesday, November 16, 2016 9:11 PM
  • Hi,

    I am facing the Target invocation exception, when SF tries to restart the service.

    We are using the OnOpenAsync() method to do some dependent service initializations and in the catch block of OnOpenAsync() i am cancelling the cancellation token like  

    catch (Exception ex)
                {
                    LogUtilities.LogError("Failed Run Async method", ex);

                    CancellationTokenSource source = new CancellationTokenSource();
                    cancellationToken = source.Token;

                    source.Cancel();
                    await Task.Delay(10000, cancellationToken);

    }

    This triggers Service fabric to restart the stateless service lifecycle and it calls 

    protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
            {
                try
                {
                    var endpoints = Context.CodePackageActivationContext.GetEndpoints()
                        .Where(endpoint => endpoint.Protocol == EndpointProtocol.Http || endpoint.Protocol == EndpointProtocol.Https)
                        .Select(endpoint => endpoint.Name);

                    return endpoints.Select(endpoint => new ServiceInstanceListener(
                        serviceContext => new OwinCommunicationListener(serviceContext, ServiceEventSource.Current, endpoint), endpoint));

                }
                catch (Exception ex)
                {
                    LogUtilities.LogError("Failed to create service replica listener.", ex);
                }
                return null;

            }

    Since the service endpoints are already configured, this method throws a Target invocation exception saying "Failed to listen on prefix 'http://+:8181/' because it conflicts with an existing registration on the machine."

    And is not able to reach till OnOpenAsync() in the restart iterations.

    Whats the most ideal solution to handle this scenario ?

    Thanks,

    Nijas

    Tuesday, November 12, 2019 10:36 PM
  • Hi,

    I am facing the Target invocation exception, when SF tries to restart the service.

    We are using the OnOpenAsync() method to do some dependent service initializations and in the catch block of OnOpenAsync() i am cancelling the cancellation token like  

    catch (Exception ex)
                {
                    LogUtilities.LogError("Failed Run Async method", ex);

                    CancellationTokenSource source = new CancellationTokenSource();
                    cancellationToken = source.Token;

                    source.Cancel();
                    await Task.Delay(10000, cancellationToken);

    }

    This triggers Service fabric to restart the stateless service lifecycle and it calls 

    protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
            {
                try
                {
                    var endpoints = Context.CodePackageActivationContext.GetEndpoints()
                        .Where(endpoint => endpoint.Protocol == EndpointProtocol.Http || endpoint.Protocol == EndpointProtocol.Https)
                        .Select(endpoint => endpoint.Name);

                    return endpoints.Select(endpoint => new ServiceInstanceListener(
                        serviceContext => new OwinCommunicationListener(serviceContext, ServiceEventSource.Current, endpoint), endpoint));

                }
                catch (Exception ex)
                {
                    LogUtilities.LogError("Failed to create service replica listener.", ex);
                }
                return null;

            }

    Since the service endpoints are already configured, this method throws a Target invocation exception saying "Failed to listen on prefix 'http://+:8181/' because it conflicts with an existing registration on the machine."

    And is not able to reach till OnOpenAsync() in the restart iterations.

    Whats the most ideal solution to handle this scenario ?

    Thanks,

    Nijas

    Tuesday, November 12, 2019 10:38 PM