Azure Instance Auto Scaling -Codeplex

Question

HI,

I am working on windows azure service instances auto scaling from a couple of days, however I have not been successful so far.

These assumptions are taken care of

1.       All hosted service is managed in same subscription

2.       All Storage Account is available in the same hosted service subscription.

3.       Both deployment files must be available on a blob in a storage account (Deployment files of the service that needs to be scaled)

4.       XML configuration file is available on a blob in a storage account.

5.       The Azure Table is created in advance.

6.       The X509 certificate is added to the Auto Scaling Worker Role and is available in the Subscription Certificate Store and in the worker role hosted service certificates list.

 

 

I am facing an exception in getDeployment Method in Servicecontroller.cs, please find the exception details below

 

System.ServiceModel.Security.MessageSecurityException: The HTTP request was forbidden with client authentication scheme 'Anonymous'. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)

   --- End of inner exception stack trace ---

 

And on Workerrole.cs in workOnDeployment() method

 {Microsoft.WindowsAzure.StorageClient.StorageClientException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden.

 

   at System.Net.HttpWebRequest.GetResponse()

   at Microsoft.WindowsAzure.StorageClient.EventHelper.ProcessWebResponseSync(WebRequest req, EventHandler`1 handler, Object sender)

   --- End of inner exception stack trace ---

   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()

   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.Execute()

   at Microsoft.WindowsAzure.StorageClient.RequestWithRetry.RequestWithRetrySyncImpl[TResult](ShouldRetry retryOracle, SynchronousTask`1 syncTask)

   at Microsoft.WindowsAzure.StorageClient.TaskImplHelper.ExecuteSyncTaskWithRetry[TResult](SynchronousTask`1 syncTask, RetryPolicy policy)

   at Microsoft.WindowsAzure.StorageClient.CloudBlob.DownloadToStream(Stream target, BlobRequestOptions options)

   at Microsoft.WindowsAzure.StorageClient.CloudBlob.DownloadByteArray(BlobRequestOptions options)

 

 

Please find below my XML Config file

<?xml version="1.0" encoding="utf-8" ?>

<root>

  <generalconfig

         scalingsecondsperloop="1"

         scalinglogtablename="ScalingLogTable"

         logtableentitiespartionkey="ScalingLogEntities"

         diagnosticsupperthreshold="70"

         diagnosticslowerthreshold="20"

         diagnosticsminutestosampleaverages="2"

         averagevmboottime="10"

         minutestosleepafterincreasevmnumber="10"

         minutestosleepafterdecreasevmnumber="10" />

  <subscriptions>

    <subscription name=[name] id=[ID]/> 

      <storages>

        <storage name=[name] key=[Key]/> 

      </storages>

 

      <services>

        <hostedservice name=[ServiceName] urlprefix=[DnsName.cloudapp.net] >

          <schedules>

            <schedule name="default" default="1">

              <description>

                Default schedule

              </description>

              <timelist>

                <timeschedule instances="2" default="1" />

                <timeschedule instances="5" hour="11" minute="30"/>   [Assuming 10 = 10 AM] 

                <timeschedule instances="5" hour="1" minute="40"/>

                  <timeschedule instances="5" hour="11" minute="50"/>

                           <timeschedule instances="5" hour="11" minute="45"/>    

                                

                                </timelist>

            </schedule>

         

              

 

          

          </schedules>

          <deployments>

            <deployment name=[Deployment name] storage=[storage account name] storagerelativepath=[Container name where package is stored]> [Assuming this is the service that needs to be scaled]

              <topschedules>

 

                <topschedule slot="production" initialhour="10" />

                <topschedule stop="1" initialhour="10" finalhour="14" />

               

                <topschedule delete="1" initialhour="10" finalhour="14"  />

              </topschedules>

              <roles>

                <role name="WebRole1" maxinstances="3" >

                  <schedule name="default" />

               

                </role>

                <role name="WorkerRole1" maxinstances="3" >

                  <schedule name="default" />

                </role>

              </roles>

            </deployment>

          </deployments>

        </hostedservice>

      </services>

    </subscription>

  </subscriptions>

</root>

 

Please let me know if I am missing something

 

Regards

Shruthi

Answer 1

I'm not familiar with that particular auto-scaling component, but such errors usually indicate an issue with the certificate.

Does your component have access to the appropriate certificate store?

---

Auto-scaling & monitoring service for Windows Azure applications at http://www.paraleap.com

Answer 2

Thanks Igor, i was indeed an issue with certificates.

However i am facing an exception

  System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at https://management.core.windows.net/{subscriptionId}/services/hostedservices/{serviceName}/deployments/{deploymentName}"), that could accept the message at the function workOnDeployment()on Workerrole.cs .

Any help in this regard is highly appreciated.

Regards

Shruthi

 

Answer 3

Once, not familiar with the component, but that error indicates that you're trying to request deployment information on a non-existent deployment.

 

You want to make sure that your subscriptionId, serviceName, and deploymentName are all properly spelled.

---

Auto-scaling & monitoring service for Windows Azure applications at http://www.paraleap.com

Answer 4

Hi,

I haven't used the auto scaling project. But since it is not a Windows Azure component, you can contact the author for issues encountered when using it. I would like to suggest you to get to the author via https://www.codeplex.com/site/users/contact/axferr?OriginalUrl=http://www.codeplex.com/site/users/view/axferr.
 
Moreover, please also make sure the parameters (such as subscription ID) are correct.
 
By the way, Windows Azure Management Services is a REST service, and it is not recommended to use WCF channel stack to consume REST services because the channel stack is designed for SOAP style services (operation oriented rather than resource oriented). It is recommended to use HttpWebRequest or WCF Web API's HttpClient. But since the issue is encountered when you use the auto scaling project, you can try to contact the author for resolving the issue.

 

Best Regards,

Ming Xu.

---

Please mark the replies as answers if they help or unmark if not.
If you have any feedback about my replies, please contact [e-mail].
Microsoft One Code Framework

Answer 5

Hi,

I will mark the reply as an answer. If you find it no help, please feel free to unmark it and follow up.

Thanks.

 

Best Regards,

Ming Xu.

---

Please mark the replies as answers if they help or unmark if not.
If you have any feedback about my replies, please contact [e-mail].
Microsoft One Code Framework

Answer 6

Hi Ming Xu,

I have wrote to the author weeks back , still waiting for the reply. Also i hav ensured subscriptionId, serviceName, and deploymentName are all properly spelled.

 

Regards

Shruthi

 

Answer 7

Hi,

Could you provide:

1. Call stack of this exception

2. The type of innerException (If there is). If it's a WebException you may try below code to catch exception and get more detailed information:

  catch (WebException ex) {
                HttpWebResponse response = (HttpWebResponse)ex.Response;

               using(StreamReader sr=new StreamReader(response.GetResponseStream()))
               {
               var s=sr.ReadToEnd();

//Provide the value of s please
               }

Also please see whether this article helps:

http://blogs.msdn.com/b/avkashchauhan/archive/2011/07/08/windows-azure-package-deployment-failed-with-error-quot-there-was-no-endpoint-listening-at-https-management-core-windows-net-quot.aspx

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 8

Hi Allen,

PFB the exception details

Stack trace

 

Server stack trace:

   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at Microsoft.Samples.WindowsAzure.ServiceManagement.IServiceManagement.EndGetDeployment(IAsyncResult asyncResult)

   at Microsoft.Samples.WindowsAzure.ServiceManagement.ServiceManagementExtensionMethods.GetDeployment(IServiceManagement proxy, String subscriptionId, String serviceName, String deploymentName)

   at AutoScalingWorker.ServiceController.getDeployment(IServiceManagement serviceManagement, String subscriptionId, DeploymentInfo deploymentInfo, HostedServiceInfo service) in C:\Users\prasadgs\Desktop\AutoScaling_alpha_\AutoScaling\AutoScallingWorker\ServiceManagement\ServiceController.cs:line 189

 

Inner Exception

 

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)

 

Regards

Shruthi

 

Answer 9

Hi Allen,

PFB the exception details

Stack trace

 

Server stack trace:

   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

 

Exception rethrown at [0]:

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)

   at Microsoft.Samples.WindowsAzure.ServiceManagement.IServiceManagement.EndGetDeployment(IAsyncResult asyncResult)

   at Microsoft.Samples.WindowsAzure.ServiceManagement.ServiceManagementExtensionMethods.GetDeployment(IServiceManagement proxy, String subscriptionId, String serviceName, String deploymentName)

   at AutoScalingWorker.ServiceController.getDeployment(IServiceManagement serviceManagement, String subscriptionId, DeploymentInfo deploymentInfo, HostedServiceInfo service) in C:\Users\prasadgs\Desktop\AutoScaling_alpha_\AutoScaling\AutoScallingWorker\ServiceManagement\ServiceController.cs:line 189

 

Inner Exception

 

System.Net.WebException: The remote server returned an error: (404) Not Found.

   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelAsyncRequest.CompleteGetResponse(IAsyncResult result)

 

Regards

Shruthi

 

Thanks for your reply. So the problem is, when you call the management API you get 404 error. What happens if you test the following URL in the browser and select the management cert? Do you get 404 error or a message says that you missed or provided wrong version header?

https://management.core.windows.net/[subscriptionid]/services/hostedservices

And what happens if you create a console application, use HttpWebRequest to send a GET request to above URL?

http://msdn.microsoft.com/en-us/library/windowsazure/ee460782.aspx

From the clues given so far it seems a proxy issue mentioned in this blog. Do you use HTTP proxy?

http://blogs.msdn.com/b/avkashchauhan/archive/2011/07/08/windows-azure-package-deployment-failed-with-error-quot-there-was-no-endpoint-listening-at-https-management-core-windows-net-quot.aspx

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 10

Hi Allen,

There is no Proxy setting configured.

I get 400 bad request error even when i test the  URL in the browser and select the certificate.

Regards

Shruthi

Answer 11

Hi,

Let's create a repro project to work on. What's the output when you create a new Console application and use code below to test? If it shows 404 error please provide your subscription id and management certificate to me. I will test to see whether it works on my side. My email is allenc at microsoft.com. If it works on my side but failed in your environment it's an environment specific issue. Maybe you're unaware of some proxy in your enviornment. If it failed on my side either I'll send you a working subscription id and certificate to let you test. Probably you used wrong subscription id.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

namespace ConsoleApplication
{
    class Program
    {
        static void Main(string[] args)
        {  // Values for the subscription ID and List Hosted Services operation.
            // 
            string subscriptionId = "replace with your subscription id";
            // The thumbprint value of the management certificate.
            // You must replace the string with the thumbprint of a
            // management certificate associated with your subscription.
            string certThumbprint = "replace with thumbprint of your management certificate";

            // The opperation to be performed. This value can be modified to reflect the operation being performed.
            string operationName = "hostedservices";

            // Build a URI for https://management.core.windows.net/<subscription-id>/services/<operation-type>
            Uri requestUri = new Uri("https://management.core.windows.net/"
                                    + subscriptionId
                                    + "/services/"
                                    + operationName);

            // Create the request and specify attributes of the request.
            HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(requestUri);

            // Define the requred headers to specify the API version and operation type.
            request.Headers.Add("x-ms-version", "2010-10-28");
            request.Method = "GET";
            request.ContentType = "application/xml";

         

            // Create a reference to the My certificate store.
            X509Store certStore = new X509Store(StoreName.My, StoreLocation.CurrentUser);

            // Try to open the store.
            try
            {
                certStore.Open(OpenFlags.ReadOnly);
            }
            catch (Exception e)
            {
                if (e is CryptographicException)
                {
                    Console.WriteLine("Error: The store is unreadable.");
                }
                else if (e is SecurityException)
                {
                    Console.WriteLine("Error: You don't have the required permission.");
                }
                else if (e is ArgumentException)
                {
                    Console.WriteLine("Error: Invalid values in the store.");
                }
                else
                {
                    throw;
                }
            }

            // Find the certificate that matches the thumbprint.
            X509Certificate2Collection certCollection = certStore.Certificates.Find(X509FindType.FindByThumbprint, certThumbprint, false);
            certStore.Close();

            // Check to see if our certificate was added to the collection. If no, throw an error, if yes, create a certificate using it.
            if (0 == certCollection.Count)
            {
                throw new Exception("Error: No certificate found containing thumbprint " + certThumbprint);
            }

            // Create an X509Certificate2 object using our matching certificate.
            X509Certificate2 certificate = certCollection[0];

            // Attach the certificate to the request.
            request.ClientCertificates.Add(certificate);

            try
            {
                // Make the call using the web request.
                HttpWebResponse response = (HttpWebResponse)request.GetResponse();

                // Display the web response status code.
                Console.WriteLine("Response status code: " + response.StatusCode);

                // Display the request ID returned by Windows Azure.
                if (null != response.Headers)
                {
                    Console.WriteLine("x-ms-request-id: "
                    + response.Headers["x-ms-request-id"]);
                    Console.WriteLine("Input any key to view response body");
                    Console.ReadKey();
                }

                // Parse the web response.
                Stream responseStream = response.GetResponseStream();
                StreamReader reader = new StreamReader(responseStream);

                // Display the raw response.
                Console.WriteLine("Response output:");
                Console.WriteLine(reader.ReadToEnd());
              
                Console.ReadKey();

                // Close the resources no longer needed.
                response.Close();
                responseStream.Close();
                reader.Close();
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
            }
            Console.ReadLine();

        }
    }
}

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 12

Hi Allen,

PFB the output of the console application,

Response status code: OK

x-ms-request-id: 8894c472ea7948e8a7f630432bf91611

<HostedServices xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><HostedService><Url>https://management.core.windows.net/  < My Subscription ID>/services/hostedservices/<my Service Name></Url><ServiceName> My Service Name </ServiceName></HostedService></HostedServices>

Thanks for the quick response.

Regards

Shruthi

Answer 13

Hi,

Thanks. It seems the management API call works in your environment. So the cause of the error might be:

1. You're calling get deployment API. This is the difference and may involve other problems such as wrong service name, etc.

2. Unknown code in your original code.

Now let's see whether it's caused by #1.

First please test whether service name is correct. Please replace service name in the below code and test (If you want to get staging deployment info, replace production with staging):

string operationName = "hostedservices/replacewithYourserviceName/deploymentslots/production";

Now test whether deployment name is correct. Please test below code. Please note the deployment name is got from the above call, as mentioned in http://msdn.microsoft.com/en-us/library/windowsazure/ee460804.aspx.

  string operationName = "hostedservices/replacewithYourserviceName/deployments/replacewithYourdeploymentName";

What's the result?

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 14

Hi  Allen,

Thanks for the reply. PFB the output  When I check for my service name ,

string operationName = "hostedservices/scalehp/deploymentslots/production";

Response status code: OK

x-ms-request-id: e4ec2d5593724542bed4da04e35919fd

 

Deployment xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Name>c1d27beb-0370-44bc-8514-099e699f7d3b</Name><DeploymentSlot>Production</DeploymentSlot>

<PrivateID>256a9961774c479b89d259a363290241</PrivateID><Status>Running</Status>

<Label>c2NhbGVocA==</Label>

<Url>DNS Name</Url><Configuration> Some huge alphanumerical</Configuration>

<RoleInstanceList><RoleInstance><RoleName>WorkerRole1</RoleName>

<InstanceName>WorkerRole1_IN_0</InstanceName><InstanceStatus>Ready</InstanceStatus></RoleInstance>

<RoleInstance><RoleName>WebRole1</RoleName><InstanceName>WebRole1_IN_0</InstanceName>

<InstanceStatus>Ready</InstanceStatus></RoleInstance></RoleInstanceList>

<UpgradeDomainCount>1</UpgradeDomainCount><RoleList><Role><

RoleName>WorkerRole1</RoleName><OsVersion>WA-GUEST-OS-1.16_201109-01</OsVersion></Role><Role>

<RoleName>WebRole1</RoleName><OsVersion>WA-GUEST-OS-1.16_201109-01</OsVersion></Role></RoleList></Deployment>

 

Deployment xmlns="http://schemas.microsoft.com/windowsazure" 

 string operationName = "hostedservices/scalehp/deployments/scalehp";

I get The remote server returned an error: (404) Not Found. PFB the screenshot for the deployment name – am I missing something?

 

Regards

 Shruthi

 

 

 

Answer 15

Hi,

It seems the deployment name is wrong. From http://msdn.microsoft.com/en-us/library/windowsazure/ee460804.aspx we can see the deployment name is got from the response of ..hostedservices/MyService /deploymentslots/production (or staging), which should be the bold one below in your scenario, not the name you got from portal :

Deployment xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Name>c1d27beb-0370-44bc-8514-099e699f7d3b</Name><DeploymentSlot>Production</DeploymentSlot>

<PrivateID>256a9961774c479b89d259a363290241</PrivateID><Status>Running</Status>



Please use this name to see whether it works. You may change your original code to call .../deploymentslots/production (or staging) instead.

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

 

Answer 16

Hi Allen,

I replaced the deployment name from scalehp ( the deployment name i had specified) to "c1d27beb-0370-44bc-8514-099e699f7d3b" (the deployment name i got from string operationName = "hostedservices/scalehp/deploymentslots/production"; )

PFB the output

<Deployment xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Name>c1d27beb-0370-44bc-8514-099e699f7d3b</Name><DeploymentSlot>Production</DeploymentSlot>

<PrivateID>256a9961774c479b89d259a363290241</PrivateID><Status>Running</Status>

<Label>c2NhbGVocA==</Label><Url>http://scalehp.cloudapp.net/</Url><Configuration> Some huge alphanumerical</Configuration><RoleInstanceList><RoleInstance><RoleName>WorkerRole1</RoleName>

<InstanceName>WorkerRole1_IN_0</InstanceName><InstanceStatus>Ready</InstanceStatus>

</RoleInstance><RoleInstance>

<RoleName>WebRole1</RoleName><InstanceName>WebRole1_IN_0</InstanceName>

<InstanceStatus>Ready</InstanceStatus>

</RoleInstance></RoleInstanceList><UpgradeDomainCount>1</UpgradeDomainCount><RoleList>

<Role><RoleName>WorkerRole1</RoleName><OsVersion>WA-GUEST-OS-1.16_201109-01</OsVersion></Role><Role><RoleName>WebRole1</RoleName><OsVersion>WA-GUEST-OS-1.16_201109-01</OsVersion></Role></RoleList></Deployment>

 

was i not configuring the deployment name properly?

Regards

Shruthi

 

Answer 17

Yes when you see this it means all works. You may use c1d27beb-0370-44bc-8514-099e699f7d3b as the deployment name in your original code or change the REST request URL to call .../deploymentslots/production

---

Allen Chen [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 18

Thanks a lot Allen :)

Regards

Shruthi

Answer 19

Hi Shruthi

I'm trying this code as well but not getting as far as you even.

I'm struggeling with:

3. Both deployment files must be available on a blob in a storage account (Deployment files of the service that needs to be scaled)

How did you get the deployment files into Azure storage? Is this standard functionality or do I need to do this seperate on each deploy? What files do I use?

Thanks for your help.

Chris

Answer 20

For autoscaling in Windows Azure, use Wasabi - http://aka.ms/autoscaling

Answer 21

Hello,

I have defined some rules for my cloud app using wasabi but only the constraint rules seem to work. The reactive rules don't have any effect. 

Any idea why this happens or how can i investigate this?

Thanks