How can I spread requests across multiple REST servers equally RRS feed

  • Question

  • User-1608281605 posted

    Hi all;

    We have an ASP.NET WebAPI2 REST server. Requests to it are async where a request is submitted and returns a guid. And then the caller polls for the status on that guid. We have customers that will feed over 1,000 requests and then continue to feed up and poll. And it can be 15 minutes until the last of the 1,000 is complete.

    With a single server this works fine. Each request we create a Task in our app and IIS then handles processing them within its thread pool.

    But load balancing is a problem. Handing 500 requests each to two servers is not necessarily equal. Once could complete in 2 minutes and the other take 13 minutes. Asking each how many pending jobs won't work because a job can be under 1 second or over 10 minutes.

    Logically the only thing I can think of is there's an intermediate store of requests. Then when a server has an available thread, it grabs the next request, processes it, and then saves it off.

    Is this the best approach here?

    And if so, does IIS have a way of signaling my app when it has an available thread from its pool?

    thanks - dave

    Tuesday, April 21, 2020 12:55 PM

All replies

  • User475983607 posted

    I do not see the problem unless the application is designed to store state on the server.  If that's the case then you need to redesign the application or implement sticky session.  

    If the clients are other machines, then your application should send a complete message to the other machine when the process competes. 

    Tuesday, April 21, 2020 3:54 PM
  • User-474980206 posted

    you options will depend on your load balancer. Round robin is the typical configuration, but most support server load feature. There are typically hard to get correct. In load feature, the balancer needs to poll all the servers for their load. 

    IIS does not have a background thread feature, you are probably using an asp.net feature like worker thread pools, which run in the asp.net process, not IIS. You can easily create a counter of in process. if 5 sequential 1 sec requests are the same load as one 5 sec request, then counts may work. its number running at a time that matters, not how long they run. but this will depend on your knowledge. some request may need more cpu or i/o so you will need a weighting algorythm.   

    Tuesday, April 21, 2020 7:33 PM