How to design an Azure application with several roles?

Question

Hello guys!

We are working on a cloud-based application in Windows Azure.
The application is separated into 2 main parts and 3 different WebFrontEnds.

As a requirement we need to have 3 WebRoles, 2 WorkerRoles and 1 Database (SQL Azure).
Data needs to be viewed and manipulated in the WebRole.
WorkerRole needs to do automatic procedures like 'generate bills'.
We decided to use C# and ASP.NET as well as Entity Framework for database access.

What would be the best way to design such features?

Examples:
Should we use all classes in the WebRoles and access the database from the WebRoles?
Should we pass the objects between WorkerRoles and WebRoles by using MessageQueues?
Should we use TableStorage to store the objects and send references by using MessageQueues?

Thanks for any help!

Markus

Answer 1

There's no easy one-size-fits-all answer. The best path depends on things like your requirements, budget, timeline, performance goals, etc.

For example, EF is generally easy and quick to code, but it's not the fastest or most flexible data access option (which is ADO.NET).

MessageQueues are good if you need notifications, since SQL Azure doesn't support Service Broker. If you can use (infrequent) polling, you might want to use table-based queues in SQL Azure instead.

TableStorage is good for cases where you have lots of data that you access relatively infrequently (due to the per-transaction costs). SQL Azure is better for smaller datasets or frequent access, since it doesn't have per-transaction costs.

Does your architecture require persistent queuing? If not, you may be able to use async HTTP requests to pass data between your workers and web roles.

You might consider simplifying your architecture by moving your workers into background threads that run in the web roles.

Will you need lots of scalability? Then data partitioning or SQL federations or AppFabric may come into play.

Lots of tradeoffs....

---

Check out my book: Ultra-fast ASP.NET: Building Ultra-Fast and Ultra-Scalable Websites using ASP.NET and SQL Server

Answer 2

Thanks for your feedback!

The project is limited in time as it is a project for master's degree at my university.
We implement a prototype until June - so performance and scalability is not a matter of concern for now.

If we got funds from a company we would need reasonable scalability.
Budget is very low since we have to pay for it.

We have around 30 tables with strings & integers that are stored in our database (actually implemented in
SQL server). Data needs to be pulled only if a user logins and wants to display or manipulate data.
It works fine with ASP.NET and EF in a simple WebRole but I agree with you that the flexibility may be better in ADO.NET.

Our main design question refers to data handling between the WebRoles, WorkerRoles and SQL Azure.
SQL Azure is used for storage and WorkerRole for background processes -> it's a must have.

Possibilities (we think):
1) We access our data from each WebRole and WorkerRole by using ADO.NET.
2) We access our data from each WorkeRole and communicate them to WebRoles.
a) by using MessageQueues (send whole objects)
b) by using HTTP requests
c) by using TableStorage

We have a good background in Java but we are beginners with .NET and Azure.
We would like to have a simple as possible approach for our implementation.

Thanks in advance for any advice!

Answer 3

Hi markustre.

If you have limited funds for this app I will move all three FE to one web role. How to made it you'll find at http://msdn.microsoft.com/en-us/library/windowsazure/gg433110.aspx.

I think best approach for this case is to use Web role(s) as FE, SQL Azure as data storage, Worker role as compute comsuming server and Queue(s) for talking between roles bi-directionally.

Web role will contain one or all three UI for user logging in and uploading or inserting some user custom data. This data will web role save into some tables(maybe as unprocessed data). After data are saved in database, web role will insert message into Queue(named ToProcess) with description witch data and what task have to make worker role. Worker role will run in loop as it is normal for worker role and constantly will grab messages from ToProcess queue(there is an option to get up to 32 messages at a time). If there is some message it will process it as the message describes the action and data reference cause you'll use the same logic as it is in web role to work with the data. After worker role process the date it will move them to different table(maybe as processed date). You can notify back user with another message inserted in different Queue named FinishedTasks. In this message you'll decribe user and task which worker role finished(this logic can be in uprocessed task too to simplify communication between web roles <-> worker role and users <-> web role) or your web role simply take a look into database table with processed data and show them to user. If You generate some kind of files, you can insert them into blob storage and only reference them in the database table.

Hope it helps. If you got some question just ask I will try to help you mode deeply in your problem if I could ;-) 

---

Microsoft Platform Developer Cloudikka blog

Answer 4

Hi Petr.

Thank you very much for your feedback. It's a real help!

Your approach:

- Worker Role puts data from database in some tables in TableStorage.
- Web Role access tables in TableStorage.
- If user changes values Web Role sends a message to Worker Role to update data.
- Worker Role updates data in database and sends a notification message back.

Is that correct?

Does the Web Role need to listen to all incoming notification messages from WorkerRole in a separate thread?

As a simple as usual approach for data access:
Would you prefer ADO.NET, Entity Framework or LINQ-to-SQL (or something else) for data access?

Alternative:
If we allowed both roles to access data from database by using ADO.NET would there be any drawbacks (e.g. concurrent access)?

Thanks in advance!

Markus

Answer 5

In a general sense, without knowing the requirements of your system in terms of scalability, monthly service cost, total use, responsiveness, performance, event-driven vs. polling, total data size, and so on, I would suggest storing all data in SQL Azure. You can use table-based queues to pass messages from WorkerRoles to WebRoles and back. That may require polling, but there's no per-transaction cost. For some example T-SQL, see:

http://rusanu.com/2010/03/26/using-tables-as-queues/

Use ADO.NET for data access, so you can leverage async requests, command batching and multiple result sets when/where needed (not available in EF). For scaling, you can partition or federate the database.

MessageQueues may be more appropriate if you have a requirement for being event-driven -- but there is a monetary cost per transaction.

HTTP requests may be more appropriate if you don't need the data that's being passed around to be persistent; by skipping persistence and polling they can also have a performance benefit.

---

Check out my book: Ultra-fast ASP.NET: Building Ultra-Fast and Ultra-Scalable Websites using ASP.NET and SQL Server

Answer 6

Hi Markus,

Data Access - Create one dll for data access layer do database. Both Web and Worker role will use it to CRUD actions same way. Just reference it in both roles and use it.

Your approach:

- Worker Role puts data from database in some tables in TableStorage.
- Web Role access tables in TableStorage.
- If user changes values Web Role sends a message to Worker Role to update data.
- Worker Role updates data in database and sends a notification message back.

My approach(maybe simplier then I describe before):

- Web role will use UI to insert data into SQL Azure table(s) with information that it it is not processed(bit column named Processed set to false)

- (Optional 1) Web role insert message into Queue(named UnprocessedQueue) to notify worker role, there is some work to do

- (Optional 1) Worker role peek the UnprocessedQueue for new messages

- (Optional 1) If there are some messages, take them(up to 32 messages at once) and process them

- (Optional 1) Each message got some information which entity(for one table it is id, for more it is entity type and id) worker role have to process

- (Optional 2) Worker role will query for entities in SQL Azure table(s) for entities with bit column named Processed set to false in some time interval

- Both of optional approaches will result in some data set which you have to process in some foreach loop

- In foreach you have to process the data and save them in the same or different table(you have to determine what will be better) and after it mark this entity as processed with bit column Processed set to true and by setting new bit column named NewOrUpdated set to true.(each time entity is processed you have to set NewOrUpdated to true and you'll use this bit value later in web role)

- When somebody login to see if the data update is done....you'll select entity with bit value Processed set to true and NewOrUpdated set to true

- Usually you'll show the list of processed items...when user open one item to view the result you can mark it as NewOrUpdated to false or you can wait after user mark it as viewed. This mechanism will help you to simplify the notification between job done worker role and web role with UI.

This is my point of view at this kind of app, but it depends on real data and usage. Its only some kind of practise I'll use, but it nos universal solution.

Hope this one helps you to understand to my ideas :-)

---

Microsoft Platform Developer Cloudikka blog