locked
Migrating specific C# desktop application to Azure PaaS: chosing services. RRS feed

  • Question

  • Hello! The goal is to choose required Azure services to migrate desktop application to the cloud in the right way.

    I've faced a problem because there are too many of them.

    The .NET C# WPF application is a kind of ads aggregator engine. It's goal is to retrieve new ads data from N public ads sources and then perform some work with them in the DB. The DB is already moved to Azure SQL DB. Technically it's a scheduler that runs tasks(jobs) one by one. After last task completed - work begins again from the first one. Tasks are: task that loads ads URLs, parses data from it and then writes it to DB. One task per ads source. And one more task to analyze data after all sources parsed - just some operations with the DB. Some parsing tasks may failed, so it's skipped in next iterations until operator fix it.

    For parallel loading and parsing data TPL Dataflow library is used. N Loading blocks load ads URLs using different proxy, and one block parsing data from loaded HTMLs(CPU-bound work). It's degree of parallerism is limited by cores count, so now it's a bottleneck of the application.(*unwanted limitation 1)

    Machine performance limit also not allows to run data parsing tasks simultaneously.(*unwanted limitation 2)

    So I need to choose architecture to avoid these two limitations: CPU-bound parsing work must be provisioned with required computing power, need somehow run parsing jobs in parallel, then after all complete perform analyze task.

    So should or shouldn't I use these Azure services?:

    • WebApp's WebJobs (or it's not fit to heavy CPU-bound tasks?)
    • Worker Role
    • Azure Batch
    • Scheduler
    • maybe anything else?

    What can optimal architecture be? Total rewriting is not an obstacle. I would be glad for any help.

    Can't post application scheme because my account is not verified.

    Monday, September 21, 2015 12:24 AM

Answers

  • Hi there,

    This is a tough one to crack since a good solution might depend details in the code and how it is able to execute in parallel and optimize CPU and IO usage.

    Web Roles running in a Web App might can have an impact on the Web App's performance, if there is no web app deployed that doesn't matter but I have seen cases where people had to move to using the Web Job SDK on an Azure VM (IaaS), this also offers the most flexibility in picking the right VM size for the workload

    Azure batch is mainly useful for handling a very big load of work as single event. It help distributing the work on x number of nodes so you don't need to setup all the plumbing of getting nodes to pickup on tasks etc.

    Worker Roles still are a very popular method because of the flexibility in VM sizes and the easy of scaling, f.e. using queue's to setup a competing consumer scenario (where multiple VM's pick parsing tasks from the queue whenever they are done with the previous one).

    When you look at rewriting the solution you might be interested in knowing Azure Service Fabric is heading our way (currently in an early preview). This is a micro-services technology that might be helpful in these kinds of solutions.

    http://azure.microsoft.com/en-us/campaigns/service-fabric/


    Please mark answered question as anwered to let others know about it.

    • Marked as answer by AsValeO Monday, September 21, 2015 7:40 PM
    Monday, September 21, 2015 9:37 AM
  • Hi AValeO,

    Batch does sound like it's possibly a decent fit. 

    Scheduler is more for running periodic work (every hour/day/week/whatever), but doesn't handle spinning up VMs on your behalf or anything like that.

    WebJobs I'm not very familiar with so can't speak to that.

    WorkerRoles could be used, but you'd have to manually set up a group of machines, manage their lifetime, and create some sort of queuing mechanism in order to distribute work to them.  It's a lot of work to get a basic solution going.

    Azure Batch at a high level does basically all of the queuing and VM management required by the WorkerRoles solution for you - so instead of having to manage the queue yourself you can just submit a "Job" and add "Tasks" - each Task could then do some processing/write some stuff to your DB and shut down (freeing up the worker for more tasks to be run).  You can even create an auto-scaling formula to dynamically change the number of worker VMs you have dynamically based on load (if you want).

    I suggest if you're interested taking a look at some of the getting started samples for using the Azure Batch API: https://github.com/Azure/azure-batch-samples/tree/master/CSharp/GettingStarted

    • Marked as answer by AsValeO Wednesday, September 23, 2015 12:25 AM
    Wednesday, September 23, 2015 12:14 AM

All replies

  • Hi there,

    This is a tough one to crack since a good solution might depend details in the code and how it is able to execute in parallel and optimize CPU and IO usage.

    Web Roles running in a Web App might can have an impact on the Web App's performance, if there is no web app deployed that doesn't matter but I have seen cases where people had to move to using the Web Job SDK on an Azure VM (IaaS), this also offers the most flexibility in picking the right VM size for the workload

    Azure batch is mainly useful for handling a very big load of work as single event. It help distributing the work on x number of nodes so you don't need to setup all the plumbing of getting nodes to pickup on tasks etc.

    Worker Roles still are a very popular method because of the flexibility in VM sizes and the easy of scaling, f.e. using queue's to setup a competing consumer scenario (where multiple VM's pick parsing tasks from the queue whenever they are done with the previous one).

    When you look at rewriting the solution you might be interested in knowing Azure Service Fabric is heading our way (currently in an early preview). This is a micro-services technology that might be helpful in these kinds of solutions.

    http://azure.microsoft.com/en-us/campaigns/service-fabric/


    Please mark answered question as anwered to let others know about it.

    • Marked as answer by AsValeO Monday, September 21, 2015 7:40 PM
    Monday, September 21, 2015 9:37 AM
  • Valery, thanks a lot for your detailed answer.

    Can you claim that you would have chosen Service Fabric to implement new solution like this?

    Monday, September 21, 2015 7:47 PM
  • Hi AValeO,

    Batch does sound like it's possibly a decent fit. 

    Scheduler is more for running periodic work (every hour/day/week/whatever), but doesn't handle spinning up VMs on your behalf or anything like that.

    WebJobs I'm not very familiar with so can't speak to that.

    WorkerRoles could be used, but you'd have to manually set up a group of machines, manage their lifetime, and create some sort of queuing mechanism in order to distribute work to them.  It's a lot of work to get a basic solution going.

    Azure Batch at a high level does basically all of the queuing and VM management required by the WorkerRoles solution for you - so instead of having to manage the queue yourself you can just submit a "Job" and add "Tasks" - each Task could then do some processing/write some stuff to your DB and shut down (freeing up the worker for more tasks to be run).  You can even create an auto-scaling formula to dynamically change the number of worker VMs you have dynamically based on load (if you want).

    I suggest if you're interested taking a look at some of the getting started samples for using the Azure Batch API: https://github.com/Azure/azure-batch-samples/tree/master/CSharp/GettingStarted

    • Marked as answer by AsValeO Wednesday, September 23, 2015 12:25 AM
    Wednesday, September 23, 2015 12:14 AM
  • Definitely, things became clearer, i'll give a try with Azure Batch.

    Thank you for useful info, Matthew!

    Wednesday, September 23, 2015 12:34 AM