Should Azure applications be single-tenant or multi-tenant?

Question

I want to build an application that can be used by multiple customers, in this case local municipalities. However, I'm not sure if I should try to cram all of the municipalities into a single project, or spin up a new project for each muni. I'm thinking it could be very painful to try to create a single application that could support per-muni customization (work flows, business rules, data). Also, I'd like to provide differing levels of service for the munis (usage, basic site, mobile site, email notification). So single-tenant seems like a good way to go, especially if I just have a handful of customers. However, I could see this quickly get out of hand if I have something on the order of 50 or more customers. Upgrades would be grueling process, monitoring could be quite nasty. Is it possible to take a hybrid approach, perhaps by having a single hosted service and multiple storage accounts? Billing is also an obvious concern, but I can see pluses/minuses for both single & multi tenant cloud apps.

I know it's early, but does Microsoft have any recommendations on this? What do others think? What are others doing?

-Andy

Answer 1

This is a quintessential question whether premises or Azure, and difficult to answer. Business model, base application characteristics, SLAs, and level of client customizations all bear heavily on the answer.  It's unavoidably a trade-off of pain avoidance up front or pain avoidance later.

Microsoft's S+S guidance is at: http://msdn.microsoft.com/en-us/architecture/aa699384.aspx

Answer 2

Eugenio Pace adapted his LitwareHR multitenanted human resources app to SQL [Server] Data Services and he's just started a series on best practices for Azure at http://blogs.msdn.com/eugeniop/.

--rj

---

OakLeaf Blog

Answer 3

Roger,

While you're on the line, do you have any thoughts, ideas, links, etc. on the topic of [multitenant] data extensibility mechanisms? There was some discussion on Oren's blog a while back, but this seems like one of the real pain points in SaaS - how to extend data on a customer-by-customer basis without forking the base code.

We've seen some really bad single and multi-table extension schemes, have looked at a two-column XML schema/data extension scheme, and a couple of even more hackneyed approaches - don't like any of them really and haven't come up with anything that makes me particularly happy.

- Joseph

Answer 4

This doesn't specifically answer your question, but what you're doing sounds similar to the thinking that's going on right now for MEBAs (Mult-Enterprise Business Applications). Here's  David Chou's blog post on MEBA http://blogs.msdn.com/dachou/archive/2008/12/01/multi-enterprise-business-applications-meba-as-cloud-based-next-generation-b2b-business-processes.aspx might be a subject worth following.

---

David Pallmann, Director, .NET Application Development, Neudesic

Answer 5

Just sent David and Pat Helland a link and invitation to jump in on this thread...

I think everyone understands and agrees there is are real per-client / per-community extensibility needs in MEBA and [shared db] SaaS applications, but I see little in the way of specific guidance, patterns, or mechanisms for accomplishing this.

Answer 6

Thanks for looping me in! Excellent thread. :)

First of all - I agree with you guys; Microsoft hasn't yet provided any specific guidance, patterns, etc. for cloud applications and especially Azure applications. I think the fair thing to say is that they're still being defined and developed. So here are just my thoughts; not necessarily the "official" word from Microsoft.

If you haven't yet, give this article a quick read - http://msdn.microsoft.com/en-us/library/aa479069.aspx. The maturity model section kind of talks about the core question in this thread, but note that the recommendation is that level 4 isn't necessarily "more appropriate" than level 1.

Both multi-tenant and single-tenant applications will work well on Azure. This is because many of the "traditional" trade-offs between single and multiple tenant architectures, such as software licensing, scalability, etc., are now significantly different on Azure. In general, from an operational infrastructure perspective, Azure blurs the lines between single and multi-tenant architectures more so than typical outsourced managed hosting providers. However, complexities in design, development, and management still exist, though at a high level the trade-offs seem a bit more balanced now with Azure. For example, it would be more complex to design and build a multi-tenant application, but on the other hand, management costs (as stated earlier in this thread) would be higher when managing multiple instances of single-tenant applications.

Thus I think, the answer is, as you'd expect, "it depends". :) We will likely not see a general answer offered for this question (decision matrix more likely), because not one side is clearly more appropriate than the other for all scenarios. A number of trade-off factors, based on different aspects, can tip the scale in either direction, depending on the scenario, and sometimes depending on the degree to which certain factors are leveraged. For example, in this case, how much of the muni-specific workflows and customizations can be abstracted into data (level 4 in the SaaS maturity model with multi-tenant architecture and isolated data/storage) and how complex that work requires, how many customers will this application support, how complex is it in managing these applications, what type of application is this intended to be (more read than written?), etc. All of these aspects can influence the approach, but it's difficult to say that either single or multi-tenant architectures is inherently better than the other, and which one type of architectures is recommended for Azure.

I think there is a general preference towards multi-tenant architectures today (myself included) because that seems more "modern" and more technically challenging to do; it seems nice to be able to "write and deploy once, support all customers" especially in today's exponential times when most applications can aspire to grow to massive user bases. But it doesn't mean all applications should be written as multi-tenant; some applications don't have to be.

Just my thoughts. Hope it makes sense for you guys, and please let me know if you think any of this seems flawed.
Best! -David

Answer 7

Ahh yes! The ubiquitous "it depends!" : ) I'm happy to see that Azure can support multiple deployment scenarios, at least. I've only just started combing through the links that everyone has posted -- TONS of great information! Thank you!

1) After some cursory reading, I wonder if cloud-based architectures deserve a paradigm shift that might make single/multi tenancy a non-issue, or at least a different kind of problem. Maybe I'm  dragging bad architecture habits onto the cloud when I should be thinking about completely new ways of writing applications. David's post would seem to indicate this. Like they say, "You can write Fortran in any language."

2) Someone else recently posted a question about managing Azure projects via an API. I think a management API would ease some of the pain. ISVs could create their own provisioning applications, ostensibly from services they've already published.

3) J. Healy's comments resonate with me. Glenn Block has done some great work helping us understand how to build composite UIs, but data extensibility seems to remain a bit of a black art. Has anyone done any work in this arena? It would probably apply to both on-premises and cloud-based applications.

At this point, I think it might behoove me to write a small Azure application incorrectly and then refactor it. There's so very much to learn...

-Andy

Answer 8

One thing to consider, is if you write your application to be multi-tenanted, you should ensure that you can still split it out on multiple services and not lock yourself into a requirement to have all tenants in a single Azure solution.  The problem you will find is that there is limited scalability of a single instance as it is currently configured with 2GB of memory, which really means about a gig of memory for your asp.net application.  If you have a large application with many tenants, and you use caching, you will quickly run out of memory and may even trigger app pool recycles, which will then make your application perform less.  I know you can always throw more instances at your Azure service, but you will still find yourself using too much memory for a large application.  It is always best to design it to be multi-tenanted but allow for a federation of multiple farms in your model where each farm can handle a subset of tenants.

Your question is one that I am very familiar with as we are a CMS vendor and single-tenant versus multi-tenant is a question we have to decide for every portal that we build.  There aren't any fixed answers, but I have found that always designing with multi-tenant in mind gives you more options in the future...

Good luck!
Shan McArthur
www.shanmcarthur.net

Answer 9

In general, it would be great if Microsoft could come up with a generic, easily deployable SaaS 'container' and 'MMC' for Azure and MEBA apps, and at the PDC you could see the foundation being laid for that. Andy and I both work with the same public sector utility and many of the [niche] workflow-oriented apps we deal with such as Permitting, Laboratory Information Management systems (LIMS), and Pretreatment all need to be heavily customized to client domain attributes, rules, and workflows.

The rules and workflow customizations seem reasonably doable given .NET 3.5 / 4.0 tooling and things like IoC/MEF/MAF, but the custom attributes (data extensibility) is still problematic. As Andy mentioned, it would be nice if there were [consistent] notion of 'composition' which architecturally, or pattern-wise, just 'flowed' from UI to Workflows to Services to Rules to Data. The lack of mechanism / patterns / guidance around the data extensibility is the part that has everyone 'rolling their own' ad hoc (and often Rube Goldbergish) solutions to this thorny aspect of SaaS, whether single or multi tenant.

- Joseph

[ P.S. Not that a standalone Microsoft Rules Server 2010 and Enterprise Rules Manager wouldn't be a really nice thing to have available... ]

Answer 10

A few thoughts - building on Shan McArthur's comments. Just something I'm exploring myself in terms of coming up with some architectural guidance around this topic too.

Like Andy's feedback (#1 in his post), I do think the cloud-based architectures present a paradigm shift, so we can look at different ways of addressing some issues, as we try to bring traditional server-based development into the cloud. Specifically in this case, as we talk about a large/complex application designed for multi-tenancy, in Azure we can look at not building that in one project/deployment. Instead, it might work better to build it in multiple projects, as separate web and worker role deployments, and weaved together in a service-oriented manner. Or from a different perspective, build it into multiple logical/functional partitions/layers (again, in a service-oriented manner), where each layer/partition can be auto-scaled, and managed independently of the others.

This means we can worry less about the current 2GB memory cap in each container (which should be something different as Azure nears beta), as the work is distributed out to more layers/tiers of containers, vertically speaking. We can also look to the Storage service in Windows Azure to serve as data cache instead of trying to use the in-memory cache, and leverage the Queues in Windows Azure to split one task into multiple/parallel worker roles (just like how a Google search query is run, in terms of parallel processing techniques). Performance characteristics will be undoubtedly a little different, but we're gaining a coupe of trade-off benefits too. Furthermore, cloud applications are better suited as stateless apps, as stateful apps that require in-memory caching, server stickiness with load-balancer intelligence, memory/cache replication to help with fault-tolerant design, etc., would make the infrastructure much more complex and error prone than necessary. Even though Azure will support stateful apps, I think stateless apps will enable many more differentiating scenarios, and will be easier to build and manage.

And from a design perspective, when integrating all of those different thoughts, a new type of cloud architecture may be n-tiered in the cloud (instead of the traditional 3-tiered presentation/business logic/data) where we split an application into as many layers/tiers as possible (by finding the right balance against performance and management trade-offs), integrate everything in a service-oriented composition manner instead of functional composition in one project, and where we try to spend as little time in each tier as possible. This type of design may help us mitigate many of the issues such as memory constraints in each container, scaling out for multi-tenancy, dealing with eventual fabric faults, and managing a complex application.

Thus, I think that, not replicating the same architecture we've had in on-premise server environments, will work better for some scenarios. That to me represents a paradigm shift, in giving us different approaches to solve some issues.

Anyway, again, just my thoughts and something I'm working on now. Will definitely flesh out more of this line of thinking, and find some flaws too.
Best! -David

Answer 11

Hi Andy,

Multi-Tenant:

Pros: 

a) Single code base

b) Single database

c) Easy to automate customer acquisition (free trials)

d) Simpler upgrades and updates

e) Reduced monitoring requirements

f) All clients updated at same time

Cons:

a) Risk of customers accessing other customers data

b) Requires 100% declarative configuration and customization

c) Exceptions within one instance may bring down all instances

d) Complicates DNS domain management. Customers typically cannot easily assign A records or CNames to your URL

e) Eventually will need to create multiple "pods" to scale

Single-Tenant:

Pros:

a) Single code base

b) Single database schema

c) All instances isolated. Reduced risk of unauthorized cross-domain data access.

d) Provision instances as needed (lower start-up costs)

e) Easy for customers to manage DNS A Records and CNames (provide static IP per instance)

f) Supports upload of additional physical files to application folder

Cons:

a) Free trials and automated customer acquisition more difficult. Requires pre-provisioning instances.

b) Multiple database sharing same schema mean more upgrade time

c) More sites/applications to monitor

Notes:

Both Multi and Single Tenant architectures should restrict customization to declarative configuration only. Forking files should never be an option (in fact, physical access to any core build files should never be allowed).

If physical file customization required (such as ASPX Master pages), exclude them from periodic updates.

Note that multi-tenant solutions share a common database. Single-tenant solutions share a common database schema, but each application instance uses its own database. Looks like you're already researching 3rd and 4th normal schemas to accomplish this. Tuple tables very powerful for declarative configuration. Hierarchical registry structures also excellent for cataloguing and persisting changes per instance.

Both architectures tend to utilize shared common services, such as SMTP email server or CDNs.

Plug-In architectures that allow customers to write custom business code will serve both architecture in the long run (DLR is great for interpretive execution of custom defined business rules). Extremely *high* risk of cross domain data access when customer code allowed to execute their own code in multi-tenant.

A service like www.pingdom.com makes it extremely easy and cheap to monitor multiple single-tenant applications. We scaled well beyond 50 applications with 1 support and operations employee.

The 2GB ceiling on current Azure OS images is *the* gating resource factor in this design decision IMO, given that most scaleable websites effectively use some sort of cache or in memory database. However, the Google trend is to use file system based tables, and maybe that is the intended usage of Azure's FS based Tables to reduce need for more RAM.

Because multi-tenant architectures must eventually be replicated as multiple-separate pods, and because virtualization makes it easy to create a single pre-configured application instance for repeated on-demand deployment, I think single-tenant is increasingly becoming a viable option.

-Mike Leach

www.CubicCompass.com

Answer 12

David, terrific comments once again! I read through all of the material you passed along and did a bunch of reading on Azure Table Services and SQL Data Services. I really think the multi-project approach you're talking about is a fantastic direction. It feels like the right kind of architecture for a cloud-based app. I've been considering the data side of things for a few days now and it seems like there might be a way to split up tenats using the PartitionKey in both ATS and SDS, but still allow for data extensions via entities. Because entities don't have fixed fields, I should be able to allow multiple customers to have divergent schemas and still use the same code for getting at their data.

Two things concern me:

1) I'm not sure I'll be able to use an existing ORM technology (Entity Framework, et. al.). The code I've seen so far seems to either use StorageClient or roll their own data layer. Having worked with LINQ to SQL and NHibernate in the past, this is a bit "two steps back" for me. Maybe that's just the cost of doing business on the cloud, but I would love to see something like the Entity Framework on the cloud.

2) Mike brings up an excellent point about plug-in architectures in a multi-tenant application. How do I prevent a customer logic from getting at everyone else's data? I suspect that a custom data layer, combined with a per-tenant PartitionKey might do the trick. Or, use a completely separate SDS/ATS instance for each customer, ala the "Shared Database, Separate Schemas" approach mentioned in Chong's "Multi-Tenant Data Architecture" article.

Answer 13

dachou said:

And from a design perspective, when integrating all of those different thoughts, a new type of cloud architecture may be n-tiered in the cloud (instead of the traditional 3-tiered presentation/business logic/data) where we split an application into as many layers/tiers as possible (by finding the right balance against performance and management trade-offs), integrate everything in a service-oriented composition manner instead of functional composition in one project, and where we try to spend as little time in each tier as possible. This type of design may help us mitigate many of the issues such as memory constraints in each container, scaling out for multi-tenancy, dealing with eventual fabric faults, and managing a complex application.

This is exactly the architectural 'direction' we've been working towards. Coming up with specific implementation patterns for essentially 'sharding' an application, both with respect to the customization requirements of MEBA and a PaaS like Azure, is challenging to say the least. It also strikes me there are domain-centric dependencies driving the process as well; that broad domain classes might have very different compositional requirements.  A heavy emphasis on the manageability associated with provisioning, deploying, and administrating the constituent services / components would also seem important as compositional complexity increases.

Answer 14

Mike Leach said:

Notes: Both Multi and Single Tenant architectures should restrict customization to declarative configuration only. Forking files should never be an option (in fact, physical access to any core build files should never be allowed).

If physical file customization required (such as ASPX Master pages), exclude them from periodic updates.

Tuple tables very powerful for declarative configuration. Hierarchical registry structures also excellent for cataloguing and persisting changes per instance.

Mike, I see you've moved upscale to LO from the old hood - glad to see you're still doing well as ever.

With regard to customizations, some classes of domains require far more customizations than others. In many niche [software] markets code is forked for each and every client (Stockamp, one of your LO neighbors is a good example). Andy and I have been attempting to address one of those niches and come up with customization patterns that will allow the requisite level of customization without forking the codebase. 

That said, declarative customization alone is not likely to fit the bill in the case of some broad classes of domains. So the question is around how to accomplish this. The tuple table pattern (ttp) and registry approaches become unwieldy beyond a limited level of of customization. I've seen ttp taken to some outlandish (architectural) levels where essentially the entire application was ttp and bottlenecked through a single table. As Andy said, we're investigating potential patterns based on Azure's partition keys and general db sharding concepts, among others.

In general, we're looking for patterns / mechanisms for providing a high level of customization without forking the code and where the methods themselves don't create excessive [architectural / design] obfustication.

Mike Leach said: Plug-In architectures that allow customers to write custom business code will serve both architecture in the long run (DLR is great for interpretive execution of custom defined business rules). Extremely *high* risk of cross domain data access when customer code allowed to execute their own code in multi-tenant.

Within a domain class, DSL's which can be leveraged across tenants, yet still allow customization, seem like they offer a lot of potential, but still require matching patterns on the data side of things to both implememt and alleviate the cross-tenant risks you point out. 

Mike Leach said: Because multi-tenant architectures must eventually be replicated as multiple-separate pods, and because virtualization makes it easy to create a single pre-configured application instance for repeated on-demand deployment, I think single-tenant is increasingly becoming a viable option.

I certainly agree that virtualization makes single-tenancy all the more attractive, but you still require a multi-tenancy discipline relative to doing nothing which creates forking nightmares down the road (no matter how much money a client throws at you...).

Answer 15

Andy S. said:

1) I'm not sure I'll be able to use an existing ORM technology (Entity Framework, et. al.). The code I've seen so far seems to either use StorageClient or roll their own data layer. Having worked with LINQ to SQL and NHibernate in the past, this is a bit "two steps back" for me. Maybe that's just the cost of doing business on the cloud, but I would love to see something like the Entity Framework on the cloud.

2) Mike brings up an excellent point about plug-in architectures in a multi-tenant application. How do I prevent a customer logic from getting at everyone else's data? I suspect that a custom data layer, combined with a per-tenant PartitionKey might do the trick. Or, use a completely separate SDS/ATS instance for each customer, ala the "Shared Database, Separate Schemas" approach mentioned in Chong's "Multi-Tenant Data Architecture" article.

Again, just my understanding and thoughts at this point :) -
1) Yes - what is available to us today in Azure CTP does not support traditional ORM technology, and as you observed, the programming model in dealing with data is very different. I think the intention is to follow some of the lessons learned and best practices in large-scale web apps today, that highly denormalized databases work better in massively scaled and available architectures (sweet spot for cloud applications), than comparatively normalized databases developed using highly abstracted ORM techniques (sweet spot still, for on-premises server apps). I think it is pretty interesting how Microsoft took this direction in building out Azure, instead of just throwing up provisioning facilities for Windows Server instances and call that a cloud platform. Though the long-term plan is to eventually blur the differences between the on-premise software and Azure cloud platforms, especially from a programming model perspective. Thus we could expect to see some support for ORM-type of frameworks available for SDS at some point.

2) I think the intention is to use .NET Services as the access control layer, even for accessing data hosted in SQL Data Services (and in Windows Azure Storage Services too). We can specify access control policies and identity mappings so that, in multi-tenant applications, end user credentials only have access to what they're allowed. Once we leverage .NET Services as the access control layer, and incur the work of setting up identity federation and claims-based authorization, that facility can be used throughout the entire application. It certainly makes the application a bit more complicated (having more platform dependencies and potentially more performance overhead), but I think that is an essential part of the "cloud architecture" direction we're discussing here, especially when we talk about a service-oriented, layered/tiered/partitioned, multi-project design in building out one complex cloud application.

Best! -David

Answer 16

dachou said:

2) I think the intention is to use .NET Services as the access control layer, even for accessing data hosted in SQL Data Services (and in Windows Azure Storage Services too). We can specify access control policies and identity mappings so that, in multi-tenant applications, end user credentials only have access to what they're allowed. Once we leverage .NET Services as the access control layer, and incur the work of setting up identity federation and claims-based authorization, that facility can be used throughout the entire application. It certainly makes the application a bit more complicated (having more platform dependencies and potentially more performance overhead), but I think that is an essential part of the "cloud architecture" direction we're discussing here, especially when we talk about a service-oriented, layered/tiered/partitioned, multi-project design in building out one complex cloud application.

We've also talked about the use of a 'bag-of-claims' as a licence for tenants and the need for a 'license manager' relative to constructing, distributing, and administering such licenses.

Answer 17

I was surprised that azure didn't do multi-tenancy out of the box. it's a large enough challenge that you would imagine MS would have attempted to solve it and give developers writing s+s apps a core architecture. looking through eugenio pace's blog a little while back, I bumped into this post:

http://blogs.msdn.com/eugeniop/archive/2008/09/11/northwind-hosting-exists-it-s-better-than-what-you-saw-and-it-s-called-saasgrid.aspx

I decided to check SaaSGrid out (http://www.apprenda.com) and use the SDK. it seems to be a cloud platform that deals with multi-tenancy issues plus a whole bunch of other stuff out of the box. They offer an SDK download and an e-mail i got gives me access to a sandbox, so it seems easy enough to take it for a spin. i haven't signed up for the sandbox yet but the documentation and API show that you write code in .NET and that you write single tenant SQL Server database definitions and they deal with multi-tenancy either through "commingled" to use eugenio's terms or database per customer (not sure how they achieve that automatically, but seems very cool). After reading their site a little bit, it also appears that you can host your app on any one of their partners. Maybe if I take it for a spin, I can post a follow on.

-Aruntha

Answer 18

First of all, I found this thread very informative and thanks to all for taking time out and posting the information.

As per my understanding, multi-tenancy is not something that can be provided out of the box by a platform (we need to consider all the aspects), it is something that needs to be considered while developing the product / application.

Also multi-tenancy can be implemented at various levels e.g I can very well have a multi-tenant code base along with being single-tenant at database level. Similarly I can achieve database level multi-tenancy by introducing additional column (CustomerID) or seperating it out with schema definition.

There is not a fixed approach, it all depends upon the type of application you are buildng. E.g. if your application requires a very high level of configurability and is very resource intensive (Business / Corporate Performance Management Applications) in that case it is not advisable to take the approach of using additional column in tables to implement multi-tenancy.

Also when we talk about multitenant application, we will also require some seperate management screens (outside the application) to perform additional tasks which I am not sure of how can be enabled incase of Out of the box functionality. E.g. you might need to mange the list of all the information (e.g Billing related information) at Customer level.

These are just my personal thoughts and people are welcome to make corrections.

Thanks,
Ajay

http://ppsinfo.blogspot.com/

Answer 19

Hi Ajay,

I just wanted to point out that in the case of using SaaSGrid (which was mentioned above), multitenancy IS provided out of the box by the platform.  SaaSGrid literally takes a regular single instance, single tenant .NET application and transforms it into a fully commercialized multitenant SaaS offering.  The multitenancy is also enforced at all tiers of the application.  SaaSGrid accomplishes this by "interweaving" multitenancy into the applications architecture in a minimally invasive fashion.  You can even choose at deploy time whether you want to go the full single instance multitenant model (with all data being commingled) OR whether you want a single database per customer.

Also, SaaSGrid provides all of the mission critical SaaS management systems and tools as part of the platform.  Things like a provisioning system, billing system, etc.  I would definately encourage you to check it out.

You can feel free to contact me directly as well.

- Jesse

http://www.apprenda.com