none
Moving the Worker role into the Web role

    Question

  • Admittedly this could just be a hack, or I may end up learning something new about Azure, but tell me what you think of this idea:

    Suppose I have a website that may need to kick off a long running process every now an then.  The SDK says I should put a message into a queue, my worker role polls and sees the message, it DoesWork(),and reverts back to polling. 

    The problem is that the recommended solution would run me about $180+/month for a non-redundant site. (under the preliminary CTP plan)

    Now, what if I were to take that long running process, move it into an ASPX page that users won't access such as ~/private/worker/CountTo1Million.aspx where that aspx file counts to a million in Page_Load(), and outputs its completion result in a table, and also in the response body. 

    Now this is where the hack comes in:  I have a non-azure console application issuing a GET at that URL to get my process to "wake up and check the queue".  Essentially, I'm creating a new Session and tying up one thread (of an unknown quantity of Azure thread) of my webrole to do my work.  I just cut my costs in half, or made my site fault tolerant at a better price.

    Would that solution work in reality?  What scalability issues would I run into? Can that ASPX page spawn other threads?  What is the lifetime of those server-side threads?  Would those threads be subject to recycling of the app pool?  Am I missing anything else?
    Sunday, September 20, 2009 5:42 PM

Answers

  • I had similar a while back. The symtom was something like RDRuntimeLoader.exe stopped responding. My problem (which ended up being hard to debug) was that RoleManager wasn't available in Application_Start, so I couldn't get config settings to get to TableStore and ended up throwing an exception, starting a chain of events that wasn't very gracefully handled and causing the app crash.

    The resolution for me was to move it to Application_BeginRequest and then making a singleton class that had execution control so it only ran once.

    It's now in Session_Start to stop it running quite so often... but otherwise is similar.
    • Marked as answer by ChrisLaMont Tuesday, September 22, 2009 4:28 AM
    Monday, September 21, 2009 8:02 PM
  • @Chris: With help from bwc's comment above (acknowledged with thanks), I finally have a working solution for your requirements. Of course, when eventually your user base rises, you can easily move the worker** to a full worker role.

    ** ERROR HANDLING DETAILS EXCLUDED FOR BREVITY; MOST IMPORTANTLY, YOU HAVE TO ENSURE THAT ERRORS DO NOT BREAK THE INFINITE LOOP.

    // SNIPPET OF GLOBAL.ASAX
        private void Session_Start(object source, EventArgs e)
            {
                 WorkerRole.Instance.StartWorkerRole();
            }
        }

        public class WorkerRole
        {
            private QueueStorage queueStorage = null;
            private MessageQueue queue = null;
            private Thread backgroundThread;
            private static readonly string QUEUE_NAME = "testqueue";

            public  static readonly WorkerRole Instance = new WorkerRole();

            public void StartWorkerRole()
            {
                if (backgroundThread != null) return;
                backgroundThread = new Thread(WorkOnQueue);
                backgroundThread.IsBackground = true;
                backgroundThread.Name = "LightWeigthWorker";
                backgroundThread.Start();
            }

            private void InitQueue()
            {
                var account = StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration();
                queueStorage = QueueStorage.Create(account);
                queue = queueStorage.GetQueue(QUEUE_NAME);
                if (!queue.DoesQueueExist())
                    queue.CreateQueue();
            }


            private void WorkOnQueue()
            {
                InitQueue();
                while (true)
                {
                    if (queue.ApproximateCount() == 0)
                    {
                        Thread.Sleep(10000);
                        continue;
                    }
                    
                    Message message = queue.GetMessage();
                     // TODO: Do something with message.
                    queue.DeleteMessage(message);
                }
            }
        }


    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    • Marked as answer by ChrisLaMont Tuesday, September 22, 2009 4:28 AM
    Tuesday, September 22, 2009 1:29 AM

All replies

  • So, when no one is using your site, in whose web session will the CountTo1Million.aspx be running?

    Your use case may just be sufficiently fulfilled with an asynchroneous asp.net process ... if the wait-time of the long-running process is not in the magnitude of ... forever.

    1. A user clicks a submit button
    2. The thread spins off an asynchroneous process
    3. Calling thread returns immediately
    4. Spun-off process continues and completes without ASP.NET timing out.

    ** Note, though, that the Azure pricing you now use is based on a minimum of 2 compute instances. So, your saving may not be half.


    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    Monday, September 21, 2009 1:03 AM
  • Hello, in addition to what Pita suggested, you must be aware that by default, the VM has a single CPU with a single core. So if you have to delegate a lot of resources to process the long running job, you may experience problems to serve the clients' requests. You can deal with this issue by creating a new instance of the web role, but by doing so, you need to pay for the additional instance, which is identical to the price of running a worker role instance.

    If, however, you don't expect too many sessions, and you won't have too many long running tasks, you can just spawn a new thread or a new process in your web role to do whatever you want.

    In the cloud machine, there's no IIS recycling. However, the VM itself may be taken offline for OS updates and other maintanence tasks sometimes. That's why we recommend you to create at least 2 instances of your role to ensure a 99.5% online.
    Lante, shanaolanxing This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, September 21, 2009 4:03 AM
  • Thanks for all the great information... so if I do need a very light "worker" role for queue polling, I suppose I could kick it off on application_start() rather than the hack I mentioned earlier.  Does that sound right?

    What is the best way to spawn such a process?  Should it be a thread or even a process?  Ideally I'd like to have a process with some isolation where it could monitor the status of my job and restart it if possible. 

    Monday, September 21, 2009 2:29 PM
  • If you want an asynchroneous thread in web role, then, it's a thread which you just spin off of the main user thread.

    if you want a worker role that will just start and do some pending work and then stop, then you can set up another instance which perhaps, wakes up at a certain time of day and works for a certain period of time. Your users will have to wait for the result of their request to be fulfilled, though. You may start the worker role for 2 hours of the night, for instance ... and users can be told to allow 24 hours for their requests to be processed.

    Then, a user's input simply queues the job. When the worker role starts, it clears out the queue and dies when the queue goes to zero (you can use an on-premise monitor that uses the Management API to handle this). Essentially, you have a webrole and worker role but the worker role just works for a few hours everyday and dies when it has nothing to do.
    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    Monday, September 21, 2009 2:54 PM
  • I think I'd like to have a process that stays active throughout the lifetime of the application.   What are your thoughts on this design?: 
    -  ASP will launch a long running process (seen below as ASPWorker) on Application_OnLoad(). 
    -  It will then spawn a few subthreads (sub processes?) to do work. 
    -  The class "MyASPWorker" will then check the thread health, and will manage them as appropriate.
    - When the application shuts down, then all threads are killed, and any cleanup is done.

    I'm not sure which method Azure will permit me to launch a thread/process/appdomain etc. so I'm mentioning all the ways I know of...

    Application_OnLoad()
    {
    //todo: (Option 1)create a new AppDomain
    //http://msdn.microsoft.com/en-us/library/system.appdomain.aspx

    //todo: (Option 2) create new process: System.Diagonistics.Process.Start

    //todo: (option 3) create a new thread

    }


    internal class MyASPWorker
    {
      Thread backgroundThread =
      new Thread(new ThreadStart(j.Performjob));
      backgroundThread.Start();

      while(true)
       {
    //todo: application logic to respawn threads if they die
    //todo: allow a configurable number of j.Performjob to be spawned


       } //end while : Will terminate threads when Application Shutdown recieved
      return 0;
    }


    internal class EmployeeClass
    {
    public void Performjob()
    {

      // Get some information about the thread.

      Console.Writeline("ID of Employee thread is {0}",
      Thread.CurrentThread.GetHashCode() );

      //Do the job.
      Console.write("Employee says:");
      for (int i=0;i<10;i++)
      {
       Console.write(i +",");
      }
      console.writeline();
    }
    }
     

    Monday, September 21, 2009 3:18 PM
  • The application has one main thread which sets itself up and then starts listening to requests. I don't know if it can create another user thread at the application context level (other than worker threads that fulfill requests).

    One way you can evaluate that is to try spinning a thread in application context (as your example indicates) and make the thread infinitely check the queue for work to do. If this works, then we can evaluate the impact of this on regular web requests.



    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    Monday, September 21, 2009 3:36 PM
  • A couple of heads up following on from those other people have said. Asp.net has a limited number of threads that are available for processes. Anything that involves a thread being taken from the Thread Pool means that there is one less available for an incoming web request. You can use the ThreadPool class to directly create a thread without making a web request.

    So you'll run into scalability issues if you rely on this as an approach. This includes using ASP.NET to spawn a long running thread.

    An alternative is to use the System.Threading namespace classes to create a thread (with System.Threading.Thread in combination with ThreadStart/ParameterisedThreadStart). This doesn't impact the ThreadPool as far as I'm aware.

    I've previously run large scale websites that rely on a System.Threading.Timer to spawn threads periodically - a class that sits somewhere like App_Code, is set up on Application_Start and periodically fires a threaded event to do something like check a resource's status. This is an approach that works well and may well fit your model of checking a Queue. The Timer would fire an event, the event handler would check the Queue and call DoesWork() (which could be on a new thread if needed).

    However with Cloud Computing and Azure, you should be careful not to build in architectural bottlenecks. This approach mentioned above gives us a system where there is always a 1:1 relationship between pseudo-Worker and Web Roles. The platform is designed to be scalable, and this is an important reason for the separation of worker and web roles. I'd be careful to check the ramificaitons of this over the long term for your application.
    Monday, September 21, 2009 4:12 PM
  • I just tried to start a background thread (with a threadStart that polls the queue) in Application_Start of an ASP.NET MVC application and role instant start fails. Anyone finds a way to get it working?



    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    Monday, September 21, 2009 6:37 PM
  • I had similar a while back. The symtom was something like RDRuntimeLoader.exe stopped responding. My problem (which ended up being hard to debug) was that RoleManager wasn't available in Application_Start, so I couldn't get config settings to get to TableStore and ended up throwing an exception, starting a chain of events that wasn't very gracefully handled and causing the app crash.

    The resolution for me was to move it to Application_BeginRequest and then making a singleton class that had execution control so it only ran once.

    It's now in Session_Start to stop it running quite so often... but otherwise is similar.
    • Marked as answer by ChrisLaMont Tuesday, September 22, 2009 4:28 AM
    Monday, September 21, 2009 8:02 PM
  • @Chris: With help from bwc's comment above (acknowledged with thanks), I finally have a working solution for your requirements. Of course, when eventually your user base rises, you can easily move the worker** to a full worker role.

    ** ERROR HANDLING DETAILS EXCLUDED FOR BREVITY; MOST IMPORTANTLY, YOU HAVE TO ENSURE THAT ERRORS DO NOT BREAK THE INFINITE LOOP.

    // SNIPPET OF GLOBAL.ASAX
        private void Session_Start(object source, EventArgs e)
            {
                 WorkerRole.Instance.StartWorkerRole();
            }
        }

        public class WorkerRole
        {
            private QueueStorage queueStorage = null;
            private MessageQueue queue = null;
            private Thread backgroundThread;
            private static readonly string QUEUE_NAME = "testqueue";

            public  static readonly WorkerRole Instance = new WorkerRole();

            public void StartWorkerRole()
            {
                if (backgroundThread != null) return;
                backgroundThread = new Thread(WorkOnQueue);
                backgroundThread.IsBackground = true;
                backgroundThread.Name = "LightWeigthWorker";
                backgroundThread.Start();
            }

            private void InitQueue()
            {
                var account = StorageAccountInfo.GetDefaultQueueStorageAccountFromConfiguration();
                queueStorage = QueueStorage.Create(account);
                queue = queueStorage.GetQueue(QUEUE_NAME);
                if (!queue.DoesQueueExist())
                    queue.CreateQueue();
            }


            private void WorkOnQueue()
            {
                InitQueue();
                while (true)
                {
                    if (queue.ApproximateCount() == 0)
                    {
                        Thread.Sleep(10000);
                        continue;
                    }
                    
                    Message message = queue.GetMessage();
                     // TODO: Do something with message.
                    queue.DeleteMessage(message);
                }
            }
        }


    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    • Marked as answer by ChrisLaMont Tuesday, September 22, 2009 4:28 AM
    Tuesday, September 22, 2009 1:29 AM
  • I never expected to see really great code come out of this thread, but THANK YOU!  With any luck, this pattern will work its way into the official documentation.

    Does anyone know how many threads are in the pool, and if this can be configured by my web.config or some other method?  I think the attribute I'm asking about is "maxWorkerThreads", and also the quantity of w3wp processes.

    edit: I just came across this setting and think it may tell me the answer I'm looking for. 
    http://msdn.microsoft.com/en-us/library/system.web.configuration.processmodelsection.maxworkerthreads.aspx

    I guess the question is, will I be able to read/update this property in production? I hope I can if the effect is isolated to my single instance.  I hope I'm denied access if affects other users on my VM.

    Tuesday, September 22, 2009 4:55 AM
  • IIS has 250 maxthreads (and I think that's per processor).
    Pita.O:
    http://www.arizentax.com/
    http://www.cgi.com
    Tuesday, September 22, 2009 7:15 AM
  • It's been about a year since I last approached this.  With the advent of Hosted Web Core on the Worker Role, I'm not sure which is better, putting the web role on the worker role, or the worker on the web.
    Wednesday, September 22, 2010 4:09 PM
  • By the way, you can simply put code in WebRole.cs (override the Run() method), and then the code looks exactly the same as it does in a worker role.

    This week's Cloud Cover episode (should air on Friday at channel9.msdn.com/shows/cloud+cover) will show an example of this.

    Wednesday, September 22, 2010 6:43 PM