Azure sdk 1.3 and machine key

Question

We recently moved to azure sdk 1.3 and we have the following issue

 The machine key in the web.config changes to a different one. This behavior can be observed when we are debugging any azure web role in visual studio. Because of the change in machine key, already existing users do not log on to the system. Only the newly created users log in.

We would like to know how to ensure that the machine key does not change.

Thanks and Regards,

Diwakar

Answer 1

Hi Diwakar,

You have to set the Machine Key explicitly for all instances of your web roles. Checkout this post

Original discovered this via this post.

Answer 2

Thanks SparkCode. I will try the above mentioned and update the post accordingly. But, what is the reason behind automatic changing of machine key? It surprised me why the machine keys for all the roles are changed when the role starts. I couldnt find any documentation for the same.

Regards,

Diwakar

Answer 3

I've started noticing that with SDK v1.3, several of my projects now have an explicit <machineKey> entry in the Web.config file, which I assume Visual Studio added at some point. It wasn't there originally. I don't know if this happens when you convert your project from v1.2.

 

 

 

Answer 4

In our projects, we do have explicit machine key but still visual studio updated the machine keys.

Answer 5

Hi SparkCode,

            Unfortuantely, this didnt work. I read the web.config in Application_BeginRequest (to test) and tried to save the configuration and got Access denied exception. The problem started to occur when we moved to azure sdk 1.3. We did not have this problem with azure sdk 1.2. We have already explicitly mentioned the machine key section in the web.config but the runtime writes in the web.config.

Regards,

Diwakar

Answer 6

Hi Diwakar,

That's strange - haven't updated to SDK 1.3 as yet but if you are setting a machine key explicitly in web.config and it is changing on deployment to another key ..... that is not expected behavior ?

i.e. this could be a problem with 1.3 IMHO because it means that cookies cannot be decrypted and encrypted due to the change in the Machine Key. That is - they are potentially encrypted/decrypted with one key and then using another different one. Perhaps SDK 1.3 is ignoring values set in the web.config file and automatically setting a machine key for you ?

Have you tried not setting a machine key at all and seeing if this is now handled automatically? However, if the machine key is changing every time .... yes, that's a significant problem for cookies. Try deploying to Azure and see whether this behavior occurs on the platform or just locally ? 

If you have tried all these and nothing is working - then its a problem .... so respond to this thread ?

Answer 7

Any update on this or ?

Answer 8

I have tried different ways including not setting the machine key. In all cases, the web.config is updated. It happens on azure as well. If one has remote desktop access to azure, one can see that machine key is replaced.

Answer 9

The problem can be easily reproduced by following the steps below:

 - Create a new azure project and add a web role to it

 - Open the web.config and add machine key to it. Leave the web.config open

 - Debug the project and observe

You will be prompted with a dialog box that web.config is modified and do you want to reload it. If you reload it, the machine key line in the web.config is changed.

If the web.config is marked read-only, the debugger throws an exception and the role will not launch. The problem is reported in the post http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/e7cfbefa-afeb-4b7d-997d-55167936eb7d

I wonder why no body from Microsoft is accepting or denying the issue.

Regards,

Diwakar

Answer 10

We updated to 1.3 last night and after our deploy this morning we are seeing tons of forms auth errors in the logs.  Right now users can't login and even requests for static content (jpgs, js) sometimes fail.  We have a habit of deleting the staging environment quickly because of the dumb getting charged even if it is stopped rule so nothing to fall back on right now.  Tried reboot instance, redeploy package, repackage and deploy, all the same end result :(

Answer 11

We have a habit of deleting the staging environment quickly because of the dumb getting charged even if it is stopped rule so nothing to fall back on right now.

The obvious workaround is to keep a copy of the deployment in Azure Storage where you pay only 15c/GB/month. But, realistically, keeping a stopped single instance in Staging for one day would have cost less than $3.

Answer 12

We're working on some guidance around how to keep a stable machineKey with SDK 1.3.

Answer 13

Thanks Steve. Its such a relief. We are eagerly waiting for a workaround as this will be a pressing issue for us to go-live on azure.

Answer 14

I expect us to publish something in the next day or two, so hopefully not too long of a wait.

Answer 15

Until a better way to fix it is provided you can get around the issue by RDP'ing to your instance, take web.config from E:\approot and copy to E:\approot\_WASR_\0

Good thing this didn't happen with 1.2 before we had rdp!

Doesn't this totally hose any site that uses membership providers to encrypt and hash passwords?  None of our existing users could log in today and now any newly created accounts will not be able to login, ever.  Seems like a pretty big problem!!  Frustrating, how could something like this be missed?

Answer 16

Until a better way to fix it is provided you can get around the issue by RDP'ing to your instance, take web.config from E:\approot and copy to E:\approot\_WASR_\0

Good thing this didn't happen with 1.2 before we had rdp!

Doesn't this totally hose any site that uses membership providers to encrypt and hash passwords?  None of our existing users could log in today and now any newly created accounts will not be able to login, ever.  Seems like a pretty big problem!!  Frustrating, how could something like this be missed?

I totally agree with your fustation :|
I made a test and it seems that each instance of the web role change the machine key so that if I have 2 instances, they have 2 different validationkeys... So we loose all benefits of multiple instance ... each postback can potentiall fall into a "viestate Validation" exception, isn't it ?
The solution of changing the key with RDP doesn't resolve much... if a VM fails, Azure creates a new one with a new machinekey... In this way a simple hosting solution is better and cheeper...

I'm just disintalling SDK 1.3 and returning to 1.2 .... hoping that next realeses will be MUCH better...

Answer 17

I'm just disintalling SDK 1.3 and returning to 1.2 .... hoping that next realeses will be MUCH better...

That was our thought yesterday too but running the 1.2 installer would fail.  The logs showed some error about needing to use the latest sdk version.  Not sure if it was connecting out to MS to check the version or if something was just messed up on my system.  Let us know if you have more luck. 

Answer 18

I unistalled SDK 1.3 and Tools for VS 1.3 ..
then installed the Tools 1.2 and all was fine..

good luck

 

Answer 19

A possible workaround to this problem is in WebRole we add a startup task. This task will run in the background and wait to x amount of sec for role to initialize and then copy the Web.config to E:\approot\_WASR_\0 folder.

 <Startup>
      <Task commandLine="CopyWebConfig.cmd" executionContext="elevated" taskType="background">
      </Task>
 </Startup>

CopyWebConfig.cmd

@echo off
REM Sleep for 180 sec
ping -n 180 127.0.0.1 > NUL 2>&1
copy E:\approot\web.config E:\approot\_WASR_\0\web.config

 

Answer 20

Leaf.ly, I didn't think that different instances would get different machineKeys... are you sure about that?  (How did you do the test?)

Answer 21

Leaf.ly, I didn't think that different instances would get different machineKeys... are you sure about that?  (How did you do the test?)

Not sure if multiple instances get a different key as we only are running 1 right now.  The problem is we have keys specified in the web.config file because they are used in our membership provider to hash the passwords.  When we publish, the web.config gets updated to a different value so users logged in see error pages everywhere because the cookies are wrong, existing users can't login because the password can't be verified, and any users that sign up during that time will experience the same issues forever now that we have updated to the correct keys in production.

Answer 22

Leaf.ly, in fact, other folks on the Windows Azure team have just told me that this is not the case.  (The whole point of setting the machineKey element is to ensure that all instances share the same value.)

Answer 23

Leaf.ly, in fact, other folks on the Windows Azure team have just told me that this is not the case.  (The whole point of setting the machineKey element is to ensure that all instances share the same value.)

I know, that is why it is so surprising, but it definitely is the case.  I can provide you with a staging instance that you can log into and see that the config files are different.

Answer 24

If you could share the info about that deployment, that would be great, actually.  Can you email me at [e-mail] so we can check it out?

Answer 25

Just to close on this, there was some miscommunication on one side or the other.  I thought Leaf.ly was saying that different instances of the same role were receiving different machineKey values.  That's not the case.  All instances of a given role will get the same machineKey value.

Answer 26

The problem can be easily reproduced by following the steps below:

 - Create a new azure project and add a web role to it

 - Open the web.config and add machine key to it. Leave the web.config open

 - Debug the project and observe

You will be prompted with a dialog box that web.config is modified and do you want to reload it. If you reload it, the machine key line in the web.config is changed.

If the web.config is marked read-only, the debugger throws an exception and the role will not launch. The problem is reported in the post http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/e7cfbefa-afeb-4b7d-997d-55167936eb7d

Sorry - little confused as to the resolution of this bug. So does the above replicate the bug or not ? Is this still an Issue in 1.3 and if so - how to fix ?

Answer 27

Yes, that describes the issue.  Yes, it's still an issue with SDK 1.3.

As I said, a workaround is coming shortly.

Answer 28

hey steve - any outcome to this yet ?

Answer 29

See http://msdn.microsoft.com/en-us/library/gg508669.aspx for the workaround.  The basic idea is to overwrite the machineKey element during your role startup.

Answer 30

Thank you!  Can finally upgrade to v1.3!

---

Auto-scaling & monitoring service for Windows Azure applications at http://www.paraleap.com

Answer 31

This fixed the issue I was having with my persistent cookies not working after upgrading to 1.3.  Every time my app recycled, I would see the following event logged:

Event ID 1315
Forms authentication failed for the request. Reason: The ticket supplied was invalid.

 

Answer 32

Glad to hear it, plunkg.

Answer 33

Although that fix may stop the issues with a running application, it is enormous pain given that every time I debug my application my web.config file changes.  Thus making my source control think that something has changed.

Answer 34

Yes.  The workaround is just that, a workaround.  Fixing the issue (probably by simply not inserting a machineKey element when web.config is read-only) will happen in a later release.

Answer 35

Steve,

A quick question - will this workaround also work for HWC ?

i.e. how it is possible to fix this within HWC as the workaround seems to assume that you are using FullIIS ? The code references 

// get the site's web configuration
  var siteNameFromServiceModel = "Web"; // update this site name for your site. 

which I am assuming it grabs from the <sites> entry ?

If you are using HWC - you don't have this or ?

Answer 36

With HWC, if there's a machineKey in your web.config, it doesn't get touched, so there's no need for this.

Answer 37

thx steve :)

maybe needs to add that to the work around doc ? http://msdn.microsoft.com/en-us/library/gg494983.aspx

Workaround
A workaround is to programmatically update the site-level configuration during role instance start-up. This does not apply to configurations using Hosted Web Core and only applies to sites using FullIIS with the <sites> element.