none
Worker role problem not helped by Diagnostics issues

    Question

  • This problem has also been posted on Stackoverflow.

    I am using System.Timers.Timer to schedule a number of "jobs" to be executed by a worker role. This works perfectly well in development but now that I have deployed the worker to staging, the jobs don't appear to be running - a number of the jobs retrieve data from a queue and I'm not seeing the dequeue count being incremented.

    Ultimately I want to solve the problem of my jobs not running, but in an effort to do this I have run into another. Azure diagnostics seems to be unreliable or at least the way I have it configured does. I am not seeing much of my debug appear in WADLogsTable, this is true in my development environment as well as staging and as such is making it impossible to troubleshoot my jobs problem (in development I see all of the trace statements appear in the output window).

    My code is below (which is slightly different to the code on SO as it has moved on slightly in an attempt to resolve this issue). With this current version, I am seeing the debug at the begining of the OnStart method and the "Heartbeat" debug. I am not for example, seeing any of the debug in the Initialise() method. I would appreciate any insight into either of these problems.

    Worker Role Code

        public class WorkerRole : RoleEntryPoint
        {
            #region Member variables
    
            private IWindsorContainer _container;
    
            private IJob[] _jobs;
    
            #endregion
    
            #region Methods
    
            public override bool OnStart()
            {
                ConfigureDiagnostics();
    
                Trace.WriteLine("JobProcessor.WorkerRole.OnStart()", "Information");
    
                try
                {
                    Initialize();
    
                    Trace.WriteLine("Resolving jobs...", "Information");
                    _jobs = _container.ResolveAll<IJob>();
    
                    StartJobs();
    
                    return base.OnStart();
                }
                catch (Exception ex)
                {
                    TraceUtil.TraceException(ex);
                    throw;
                }
                finally
                {
                    Trace.WriteLine("JobProcessor.WorkerRole.OnStart - Complete", "Information");
                }
            }
    
            /// <summary>
            /// Sets up diagnostics.
            /// </summary>
            private void ConfigureDiagnostics()
            {
                CloudStorageAccount storageAccount = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue(Constants.DiagnosticsConnectionString));
    
                RoleInstanceDiagnosticManager manager = storageAccount.CreateRoleInstanceDiagnosticManager(RoleEnvironment.DeploymentId, RoleEnvironment.CurrentRoleInstance.Role.Name, RoleEnvironment.CurrentRoleInstance.Id);
                DiagnosticMonitorConfiguration config = manager.GetCurrentConfiguration();
    
                if (config == null)
                    config = DiagnosticMonitor.GetDefaultInitialConfiguration();
    
                config.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
                config.Logs.ScheduledTransferLogLevelFilter = LogLevel.Verbose;
    
                manager.SetCurrentConfiguration(config);
    
                DiagnosticMonitor.Start(Constants.DiagnosticsConnectionString, config);
            }
    
            /// <summary>
            /// Sets up the IoC container etc.
            /// </summary>
            private void Initialize()
            {
                Trace.WriteLine("WorkerRole.Initialize()", "Information");
    
                try
                {
                    Trace.WriteLine("Configuring AutoMapper...", "Information");
                    AutoMapperConfiguration.Configure();
    
                    Trace.WriteLine("Configuring Windsor...", "Information");
                    _container = new WindsorContainer();
    
                    Trace.WriteLine(string.Format("Installing assemblies from directory...{0}", 
                        Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot)), "Information");
    
                    _container.Install(FromAssembly.InDirectory(
                        new AssemblyFilter(Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot))));
    
                    Trace.WriteLine(string.Format("Setting the default connection limit..."), "Information");
                    ServicePointManager.DefaultConnectionLimit = 12;
                }
                finally
                {
                    Trace.WriteLine("WorkerRole.Initialize - Complete", "Information");
                }
            }
    
            /// <summary>
            /// Starts all of the jobs.
            /// </summary>
            private void StartJobs()
            {
                Trace.WriteLine("WorkerRole.StartJobs()", "Information");
    
                try
                {
                    foreach (IJob job in _jobs)
                    {
                        job.Start();
                    }
                }
                finally
                {
                    Trace.WriteLine("WorkerRole.StartJobs - Complete", "Information");
                }
            }
    
            public override void Run()
            {
                Trace.WriteLine("JobProcessor.WorkerRole.Run()", "Information");
    
                try
                {
                    while (true)
                    {
                        Thread.Sleep(10000);
                        Trace.WriteLine("Heartbeat...", "Verbose");
                    }
                }
                catch (Exception ex)
                {
                    TraceUtil.TraceException(ex);
                    throw;
                }
                finally
                {
                    Trace.WriteLine("JobProcessor.WorkerRole.Run() - Complete", "Information");
                }
            }
    
            public override void OnStop()
            {
                Trace.WriteLine("WorkerRole.OnStop()", "Information");
    
                try
                {
                    foreach (IJob job in _jobs)
                    {
                        job.Stop();
                    }
                    _container.Dispose();
                }
                catch (Exception ex)
                {
                    TraceUtil.TraceException(ex);
                    throw;
                }
                finally
                {
                    Trace.WriteLine("WorkerRole.OnStop - Complete", "Information");
                }
            }
    
            #endregion
    
            #region Private util classes
    
            public static class AutoMapperConfiguration
            {
                public static void Configure()
                {
                    Mapper.Initialize(x => x.AddProfile<ModelProfile>());
                }
            }
    
            #endregion
        }
    

    TraceUtil code

        public static class TraceUtil
        {
            /// <summary>
            /// Outputs the exception to the trace listeners.
            /// </summary>
            /// <param name="ex">The exception to output.</param>
            public static void TraceException(Exception ex)
            {
                StringBuilder buffer = new StringBuilder();
    
                while (ex != null)
                {
                    buffer.AppendFormat("{0} : ", ex.GetType());
                    buffer.AppendLine(ex.Message);
                    buffer.AppendLine(ex.StackTrace);
    
                    ex = ex.InnerException;
                }
                Trace.TraceError(buffer.ToString());
    
                // Attempt to ensure that the debug is pushed up to storage.
                Thread.Sleep(2000);
            }
        }
    

    App.config

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      ...
      <system.diagnostics>
        <trace autoflush="true">
          <listeners>
            <add type="Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
                 name="AzureDiagnostics">
              <filter type="" />
            </add>
          </listeners>
        </trace>
      </system.diagnostics>
    </configuration>

    Service Configuration

    <?xml version="1.0" encoding="utf-8"?>
      ...
      <Role name="JobProcessor">
        <Instances count="1" />
        <ConfigurationSettings>
          <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=<my account>;AccountKey=<my key>" />
          <Setting name="DataConnectionString" value="DefaultEndpointsProtocol=https;AccountName=<my account>;AccountKey=<my key>" />
          <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
          <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="<my username>" />
          <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="<my encrypted password>" />
          <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="2013-02-22T23:59:59.0000000+00:00" />
          <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="true" />
        </ConfigurationSettings>
        <Certificates>
          <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption" thumbprint="<my thumbprint>" thumbprintAlgorithm="sha1" />
        </Certificates>
      </Role>
    </ServiceConfiguration>

    Monday, March 05, 2012 5:55 PM

Answers

All replies

  • Do you mean you see “JobProcessor.WorkerRole.OnStart()” in your log, but not “WorkerRole.Initialize()” from the Initialize method? That is strange, please try to move all code from the initialize method to the OnStart method itself to see if you at least see more logs.
    Tuesday, March 06, 2012 7:21 AM
  • Hi,

    Does it work if you use elevated execution context? (At least can you get the File trace log fie?)

    http://msdn.microsoft.com/en-us/library/windowsazure/gg557552.aspx

    And what is the log you get if you use intellitrace?

    http://blogs.msdn.com/b/jnak/archive/2010/06/07/using-intellitrace-to-debug-windows-azure-cloud-services.aspx


    Allen Chen [MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    • Marked as answer by S1mm0t Tuesday, March 06, 2012 3:11 PM
    Tuesday, March 06, 2012 9:13 AM
  • Allen,

    Thank you so much for your answer, thanks to you I have been able to resolve my issue. I am using VS2010 professional so I was unable to use intellitrace, but switching on the elevated execution context, the text file is now produced containing all of my debug.

    The reason why my jobs were not being executed was due to the line below:

    _container.Install(FromAssembly.InDirectory(
                        new AssemblyFilter(Path.Combine(Environment.GetEnvironmentVariable(Constants.RoleRoot), Constants.AppRoot))));

    The role root on staging is E:. Path.Combine() has an obscure implementation which you can read more about here. What this meant was that Castle was searching for assemblies in E:approot rather than E:\approot as I expected. I am now constructing the approot path with the method below:

            private string GetAppRoot()
            {
                string root = Environment.GetEnvironmentVariable(Constants.RoleRoot);
    
                if (root.EndsWith(Path.VolumeSeparatorChar.ToString()))
                    root += Path.DirectorySeparatorChar;
    
                return Path.Combine(root, Constants.AppRoot);
            }

    That has resolved my issue and I am now seeing my jobs work as expected.

    I would still really like to understand why I am not seeing all of my debug appear in the WADLogsTable however as I am bound to have more problems that I need to debug going forward and I would like to avoid having to run my worker role with elevated privileges if I can help it.

    Tuesday, March 06, 2012 3:21 PM
  • I would still really like to understand why I am not seeing all of my debug appear in the WADLogsTable however as I am bound to have more problems that I need to debug going forward and I would like to avoid having to run my worker role with elevated privileges if I can help it.

    It appears that now I am running in the elevated context, I am seeing all of my debug appear in the WADLogsTable as well.
    Tuesday, March 06, 2012 3:53 PM