Strange Diagnostics Errors

Question

Hi,

I tried to configure Azure Diagnostics but I get a lot of error messages and don't know how to get rid of them.
I just want to be able to log messages in the Table Storage, for now I don't need performance counters or event logs.

I've read a dozen blogs, tutorials and MSDN articles but all to no avail. The console is still cluttered with error messages:

[fabric] Role Instance: deployment(198).XXX.XXX.0
[fabric] Role state Busy
[Diagnostics]: UpdateState(Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorStartupInfo, Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorConfiguration, )
[Diagnostics]: Acquired mutex
[Diagnostics] Information: ...
[Diagnostics] Information: ...\MonAgentHost.exe -LocalPath "..." -StaticConfigFile "xxx.xml" -ConfigFile "xxx.xml" -ShutDownEvent WADDM-ShutDown-bc768b148e17401585d8be3a83c1d735 -InitializedEvent WADM-StartUp-bc768b148e17401585d8be3a83c1d735 -parent 8540 -events
[MonAgentHost] Error: MA EVENT: 2011-01-18T12:34:13.898Z
[MonAgentHost] Error:   2
[MonAgentHost] Error:   7840
[MonAgentHost] Error:   3908
[MonAgentHost] Error:   SelfMonitoring
[MonAgentHost] Error:   0
[MonAgentHost] Error:   x:\rd\rd_fun_stable\services\monitoring\agent\dll\selfmon.cpp
[MonAgentHost] Error:   MASelfMon::GetProcCntrs
[MonAgentHost] Error:   1417
[MonAgentHost] Error:   ffffffffc0000bb8
[MonAgentHost] Error:   0
[MonAgentHost] Error:   
[MonAgentHost] Error:   PdhAddCounter(\Process(MonAgentHost#0)\ID Process) failed
[Diagnostics]: Creating config channel server
[MonAgentHost] Output: Agent will exit when WADDM-ShutDown-bc768b148e17401585d8be3a83c1d735 is signaled.
[MonAgentHost] Output: Will signal WADM-StartUp-bc768b148e17401585d8be3a83c1d735 after the agent is initialized.
[MonAgentHost] Output: Registered as an event consumer.
[MonAgentHost] Output: Agent will exit when parent process 8540 exits.
[MonAgentHost] Output: Monitoring Agent Started
[Diagnostics]: Starting configuration channel polling
[runtime] Role entrypoint . CALLING  OnStart()
[WaWorkerHost.exe] Diagnostics initialized and running
[fabric] Role state Started

Can anyone help me with this? I don't even know what the problem is, logging works but the console is full of errors.

Even if this is not a "real" problem (as in: can't deploy the application), I would like to fix this (or at least hide the messages) because I need the console for my own output.

Thanks in advance,
EnCey

 

Answer 1

Hi EnCey,

It looks like an error related to performance counter, could you let me know your azure diagnostic initialization code?

Thanks,

---

Mog Liang
Please mark the replies as answers if they help or unmark if not.
If you have any feedback about my replies, please contact [e-mail].
Microsoft One Code Framework

Answer 2

Hi,

sorry for the late response, I wasn't in the office last week.

Here is my initialization code:

			CloudStorageAccount storageAccount = CloudStorageAccount.Parse(RoleEnvironment.GetConfigurationSettingValue(diagnosticConnectionString));
			RoleInstanceDiagnosticManager roleInstanceDiagnosticManager = storageAccount.CreateRoleInstanceDiagnosticManager(RoleEnvironment.DeploymentId, RoleEnvironment.CurrentRoleInstance.Role.Name, RoleEnvironment.CurrentRoleInstance.Id);
			DiagnosticMonitorConfiguration config = roleInstanceDiagnosticManager.GetCurrentConfiguration();

			config.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
			config.DiagnosticInfrastructureLogs.ScheduledTransferLogLevelFilter = LogLevel.Warning;
			config.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
			config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
			config.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);

			// performance counter
			PerformanceCounterConfiguration perfCounterConfig = new PerformanceCounterConfiguration();
			perfCounterConfig.CounterSpecifier = @"\Processor(*)\% Processor Time";
			perfCounterConfig.SampleRate = System.TimeSpan.FromSeconds(10.0);
			config.PerformanceCounters.DataSources.Add(perfCounterConfig);

			roleInstanceDiagnosticManager.SetCurrentConfiguration(config);
			Trace.TraceInformation("Diagnostics initialized and running");

 

I added a simple performance counter configuration in the hope that it might fix the errors I received, but it didn't help.
Without the above performance counter configuration, the same error messages are logged to the console.

Perhaps I should add that I work in the development environment, I haven't deployed and tested the Logger code yet (due to the errors).

Answer 3

I don't think the problem is with your code, instead maybe how MonAgentHost is running on your machine.

The error code from your logs is ffffffffc0000bb8, the important bit being least c0000bb8 bit:

0xC0000BB8 (PDH_CSTATUS_NO_OBJECT)

The specified object is not found on the system.

http://msdn.microsoft.com/en-us/library/aa373046(v=vs.85).aspx

Looks like the actual counter it's looking for is MonAgentHost specific too: PdhAddCounter(\Process(MonAgentHost#0)\ID Process) failed

Have you upgraded the version of the Azure SDK on your machine recently? It may be a good idea to try another computer or reinstall the Azure Tools SDK.

I hope this may shed some further light on the situation.

Thanks

Andy

---

http://blog.bareweb.eu

Answer 4

Hi Andy,

thanks for the link, at least I've a starting point now.

I have uninstalled and re-installed the newest version of Windows Azure SDK, Tools and AppFabric, but the problem remains.
Do you have another idea of how to resolve this? 

Thanks,
EnCey

Answer 5

Are you running Visual Studio elevated as an Administrator? I'm sure permissions could be an issue when trying to access the counters.

I can't replicate it locally so it's hard to see how it can happen. Does it happen in a brand new Cloud Project?

Andy

---

http://blog.bareweb.eu

Answer 6

Yes, Visual Studio runs with elevated privileges, I restarted and checked it to be sure.

The Cloud Project is new. I started working on it about one month ago, I don't think I installed a newer version of Azure in the meantime (except today).
My OS is a 64bit Windows 7 in case this if of relevance. 

I don't understand why the Diagnostics service even messes around with performance counters when I don't want to use them...

Can I perhaps just turn off these MonAgentHost messages?
The errors seem to have no effect except that they clutter up the console, my log messages are all written to the WADLogsTable.

Thanks,
EnCey

Answer 7

The MonAgent host is going to be the thing collecting all the diagnostics for your role, so if something is amiss with its counters you're probably seeing a symptom that's of a wider concern. 

I've just opened an elevated command prompt and connected to the counters - lets see if you can do the same.

Run any basic Azure project - mine is a single WebRole with no other changes in a brand new application. Open an elevated command prompt, and type: typeperf "\Process(MonAgentHost#0)\ID Process"

 

This is what I get:

 

C:\Windows\system32>typeperf "\Process(MonAgentHost#0)\ID Process"

 

"(PDH-CSV 4.0)","\\BAREWEBDEV2\Process(MonAgentHost)\ID Process"

"01/25/2011 10:35:37.300","7980.000000"

"01/25/2011 10:35:38.303","7980.000000"

"01/25/2011 10:35:39.307","7980.000000"

"01/25/2011 10:35:40.311","7980.000000"

"01/25/2011 10:35:41.314","7980.000000"

"01/25/2011 10:35:42.317","7980.000000"

"01/25/2011 10:35:43.321","7980.000000"

"01/25/2011 10:35:44.324","7980.000000"

"01/25/2011 10:35:45.327","7980.000000" 

---

http://blog.bareweb.eu

Answer 8

I don't get the same output:

C:\Windows\system32>typeperf "\Process(MonAgentHost#0)\ID Process"

Fehler: Keine gültigen Leistungsindikatoren

Translated this means "Error: No valid Performance Indicators"

I tried this with Diagnostics enabled and disabled in the ServiceConfiguration file since I wasn't sure which is the default, but the result is the same in both cases.

Answer 9

I think maybe you've found a bug with how the Azure tools work with the localisation of your version of Windows. Can I ask which culture that is, is it de-DE? 

I wonder if maybe in German the "ID Process" part of the performance counter name is localised. Can you do:

C:\Windows\system32>typeperf -q process

\process(*)\% Processor Time

\process(*)\% User Time

\process(*)\% Privileged Time

\process(*)\Virtual Bytes Peak

\process(*)\Virtual Bytes

\process(*)\Page Faults/sec

\process(*)\Working Set Peak

\process(*)\Working Set

\process(*)\Page File Bytes Peak

\process(*)\Page File Bytes

\process(*)\Private Bytes

\process(*)\Thread Count

\process(*)\Priority Base

\process(*)\Elapsed Time

\process(*)\ID Process

\process(*)\Creating Process ID

\process(*)\Pool Paged Bytes

\process(*)\Pool Nonpaged Bytes

\process(*)\Handle Count

\process(*)\IO Read Operations/sec

\process(*)\IO Write Operations/sec

\process(*)\IO Data Operations/sec

\process(*)\IO Other Operations/sec

\process(*)\IO Read Bytes/sec

\process(*)\IO Write Bytes/sec

\process(*)\IO Data Bytes/sec

\process(*)\IO Other Bytes/sec

\process(*)\Working Set - Private

 

The command completed successfully.

---

http://blog.bareweb.eu

Answer 10

I have the same problem, and it's getting me crazy... I'm following the conversation

Answer 11

"typeperf -q process" yields nothing, however "typeperf -q prozess" works (culture is de-AT)

\prozess(*)\Prozessorzeit (%)
\prozess(*)\Benutzerzeit (%)
\prozess(*)\Privilegierte Zeit (%)
\prozess(*)\Virtuelle Bytes (max.)
\prozess(*)\Virtuelle Größe
\prozess(*)\Seitenfehler/s
\prozess(*)\Arbeitsseiten (max.)
\prozess(*)\Arbeitsseiten
\prozess(*)\Auslagerungsdatei (max. Bytes)
\prozess(*)\Auslagerungsdatei (Bytes)
\prozess(*)\Private Bytes
\prozess(*)\Threadanzahl
\prozess(*)\Basispriorität
\prozess(*)\Verstrichene Zeit
\prozess(*)\Prozesskennung
\prozess(*)\Prozesskennung erstellen
\prozess(*)\Auslagerungsseiten (Bytes)
\prozess(*)\Nicht-Auslagerungsseiten (Bytes)
\prozess(*)\Handleanzahl
\prozess(*)\E/A-Lesevorgänge/s
\prozess(*)\E/A-Schreibvorgänge/s
\prozess(*)\E/A-Datenvorgänge/s
\prozess(*)\Andere E/A-Vorgänge/s
\prozess(*)\E/A-Bytes gelesen/s
\prozess(*)\E/A-Bytes geschrieben/s
\prozess(*)\E/A-Datenbytes/s
\prozess(*)\Andere E/A-Bytes/s
\prozess(*)\Arbeitsseiten - privat

Well I guess that means I'm screwed... unless changing the culture without reinstalling Windows works. I knew it would cause only problems to have the German version of OS and Visual Studio :/

Answer 12

I have the same problem with the spanish SO

Answer 13

So in theory you should be able to do:

typeperf "\Prozess(MonAgentHost#0)\Prozesskennung"

That's my guess at which one is "ID Process", sorry my Austrian isn't so great ;-) 

And you'd get the same sort of results that I got earlier with typeperf "\Process(MonAgentHost#0)\ID Process"

Andy

---

http://blog.bareweb.eu

Answer 14

Yes that works, you guessed right ;). 
The question is if I can change the current culture in code or OS-wide and if it works then, I will try this now.

Answer 15

Yes, I get:

C:\Windows\system32>typeperf "\Proceso(MonAgentHost#0)\Id. de proceso

"(PDH-CSV 4.0)","\\PORTATIL\Proceso(MonAgentHost)\Id. de proceso"
"01/25/2011 12:10:41.471","19952.000000"
"01/25/2011 12:10:42.478","19952.000000"
"01/25/2011 12:10:43.483","19952.000000"
"01/25/2011 12:10:44.489","19952.000000"
"01/25/2011 12:10:45.494","19952.000000"
"01/25/2011 12:10:46.498","19952.000000"
"01/25/2011 12:10:47.505","19952.000000"
"01/25/2011 12:10:48.512","19952.000000"

Answer 16

I think it's going to be OS wide, but would be happy to be proven wrong :-)

 

 

Just another point, "The errors seem to have no effect except that they clutter up the console, my log messages are all written to the WADLogsTable." That's pretty great news. It means you can ignore the errors for now.

Since the logs aren't written to the Console as their primary destination you can ignore these warnings. All the instances of Windows I've remote desktop'd onto in Azure have been the USA edition of Windows - and they've been in the EU West. 

So my point is hopefully the MSFT guys will get on to fixing this localisation issue, but until then you shouldn't stop doing wonderful fun stuff with Azure.

 

 

 

---

http://blog.bareweb.eu

Answer 17

Since we still need the console for test output I can't just ignore the messages, they make working with the console very frustrating.

But I'll try to either change the culture or filter the messages. Although the errors didn't seem to have a negative effect, I didn't want to check in the Logger code without knowing the source of the problem and the possible side effects. Now that i know what caused them, I'm more confident that they won't affect the rest of the application.

Thanks a lot Andy, you saved me days of reserach and testing!

@Xavi and all others with the same problem: I will report back here when I've found a solution / workaround for the problem.

 

Answer 18

No worries EnCey.

I have been trying to filter out the messages, but since it's not your apps raising them (MonAgentHost is) it's quite hard to find the right config file to put the filters into! 

I'm going to have a bit more of a go at it when I have a little more time, and I'll post back here if I can find a solution for you.

Andy

---

http://blog.bareweb.eu

Answer 19

I will write too if I find a solution / workaround...

Answer 20

Hi guys,

Do either of you (Xavi or EnCey) want to try something out for me? 

I created a simple Console App that will loop every second and copy an existing performance counter to a new performance counter. The idea is that you could use command line arguments to specify which one is which, and you could configure it with arguments for any Localisation.

http://blog.bareweb.eu/wp-content/uploads/2011/01/LocalisedPerfCounterMapper.zip

The Zip file is the full source code so you can see what I'm doing. 

The arguments I have at the moment are:

LocalisedPerfCounterMapper.exe process "ID Process" "MonAgentHost" Andy "ID Process" MonAgentHost

EnCey you could use 

LocalisedPerfCounterMapper.exe prozess Prozesskennung "MonAgentHost" process "ID Process" "MonAgentHost" 

Xavi you could use

LocalisedPerfCounterMapper.exe Proceso "Id. de proceso" MonAgentHost" process "ID Process" "MonAgentHost" 

 

It works for me but I don't know if anything like the "Process" counter name is reserved.

Note that also I don't know whether you'd like to use this in the long term, but it should give you an idea as to whether something like this is possible and a confirmation that this is the real error.

Thanks
Andy

---

http://blog.bareweb.eu

Answer 21

Hi Andy,

here's what happens when I run your app:

 

C:\..\>LocalisedPerfCounterMapper.exe prozess Prozesskennung "MonAgentHost" process "ID Process" "MonAgentHost"

Unbehandelte Ausnahme: System.InvalidOperationException: Der angeforderte Leistungsindikator muss als ReadOnly initialisiert werden, da er nicht benutzerdefiniert ist.
  bei System.Diagnostics.PerformanceCounter.InitializeImpl()
  bei System.Diagnostics.PerformanceCounter..ctor(String categoryName, String counterName, String instanceName, Boolean readOnly)
  bei LocalisedPerfCounterMapper.Program.Main(String[] args) in c:\dev\ForumQuestions\LocalisedPerfCounterMapper\LocalisedPerfCounterMapper\Program.cs:Zeile 37.

 

Basically it says that the performance counter must be declared read-only because it is not a custom (user-defined) counter.

 

Answer 22

Yeah it looks like the "Process" counter must be reserved on all Windows OS flavours then. What a shame! Thanks for trying. 

Have you read this document: http://support.microsoft.com/kb/287159 It really explains what's going on and talks about how the registry works underlying this PDH api, albeit not for the latest versions of Windows.

I found another document here: http://www.hardtoc.com/archives/88 This one is a really fun read.

Unfortunately, there's very little we can do without MSFT's help on this one. 

 

---

http://blog.bareweb.eu

Answer 23

Unfortunately, there's very little we can do without MSFT's help on this one. 

Just wanted to highlight this one. 

Can we please get official response on this topic or is marking the identification of the problem as an answer the only input we'll get from MSFT?
Right now, all I know is that you can't use Diagnostics on any non-English Windows PC... and that doesn't solve my problem.

I will leave the post marked as answer so others with the same problem know what is causing it, but I sincerly hope that there will be another MSFT repsonse... perhaps about a workaround or whether this is being looked at and when/if it'll be fixed.