Azure Stack TP3 tracing logs for troubleshooting

Question

็Hi All,

Now, i'm deployed azure stack tp3 complete. but, i stuck to step create resource group. 

i try to creating resource group via powershell and azure stack portal but not working both. it's show "creating resource group" status and never finish.

how can i trace the log for trobleshoting?

if i have any error or problem how can find logs for investigate my stuck.

Thanks,

Answer 1

Hi there,

What resources are you trying to create within your resource Group? or are you deploying an empty resource group?

Thanks

--

shri

Answer 2

Hi,

This looks like an ARM deadlock initializing the location data provider cache.

You can try the following workaround to restart ARM app pool (or do an iisreset) on was01 VM:

 

Enter-PSSession mas-was01

Import-Module WebAdministration

Restart-WebAppPool AzureStack-ResourceManager

 

Let us know how it goes,

 

TP3 Azure Stack Docs:

https://docs.microsoft.com/en-us/azure/azure-stack/

We apologize for any inconvenience and appreciate your time and interest in Azure Stack.

https://azure.microsoft.com/en-us/blog/hybrid-application-innovation-with-azure-and-azure-stack/

If you continue experience any issues with TP3 release, feel free to contact us.

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-troubleshooting

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-diagnostics

https://aka.ms/GetAzureStackLogs

Thanks,

---

Gary Gallanes

Answer 3

Hello,

Please see https://aka.ms/GetAzureStackLogsfor instructions and detailed information on running traces and collecting logs.

 

The Trace Collector

The Trace Collector is enabled by default. It continuously runs in the background and collects all Event Tracing for Windows (ETW) logs from component services on Azure Stack and stores them on a common local share.

The following are important things to know about the Trace Collector:+

• The Trace Collector runs continuously with default size limits. The default maximum size allowed for each file (200 MB) is not a cutoff size. A size check occurs periodically (currently every 10 minutes) and if the current file is >= 200 MB, it is saved and a new file is generated. There is also an 8 GB (configurable) limit on the total file size generated per event session. Once this limit is reached, the oldest files are deleted as new ones are created.
• There is a 5-day age limit on the logs. This limit is also configurable.
• Each component defines the trace configuration properties through a JSON file. The JSON files are stored in C:\TraceCollector\Configuration. If necessary, these files can be edited to change the age and size limits of the collected logs. Changes to these files require a restart of the Microsoft Azure Stack Trace Collector service for the changes to take effect.

The Log Collection Tool

The PowerShell command Get-AzureStackLogs can be used to collect logs from all the components in an Azure Stack environment. It saves them in zip files in a user defined location. If our technical support team needs your logs to help troubleshoot an issue, they may ask you to run this tool.+

Caution

These log files may contain personally identifiable information (PII). Take this into account before you publicly post any log files.+

We currently collect the following log types:+

• Azure Stack deployment logs
• Windows event logs
• Panther logs

To troubleshoot VM creation issues.

• Cluster logs
• Storage diagnostic logs
• ETW logs

These are collected by the Trace Collector and stored in a share from where Get-AzureStackLogs retrieves them.

 

To identify all the logs that get collected from all the components, refer to the <Logs> tags in the customer configuration file located at C:\EceStore\<Guid>\<GuidWithMaxFileSize>.

 

The Get-AzureStackLogs.ps1 script

To run Get-AzureStackLogs

1. Log in as AzureStack\AzureStackAdmin on the host.
2. Open a PowerShell window as an administrator.
3. Run the following commands to import the PowerShell modules:
   • cd C:\CloudDeployment\AzureStackDiagnostics\Microsoft.AzureStack.Diagnostics.DataCollection
   • Import-Module .\Microsoft.AzureStack.Diagnostics.DataCollection.psd1
4. Run Get-AzureStackLogs.

Examples

• Collect all logs for all roles:

Get-AzureStackLogs -OutputPath C:\AzureStackLogs

• Collect logs from VirtualMachines and BareMetal roles:

Get-AzureStackLogs -OutputPath C:\AzureStackLogs -FilterByRole VirtualMachines,BareMetal

• Collect logs from VirtualMachines and BareMetal roles, with date filtering for log files for the past 8 hours:

Get-AzureStackLogs -OutputPath C:\AzureStackLogs -FilterByRole VirtualMachines,BareMetal -FromDate (Get-Date).AddHours(-8) -ToDate (Get-Date)

 

If the FromDate and ToDate parameters are not specified, logs are collected for the past 4 hours by default.

Currently, you can use the FilterByRole parameter to filter log collection by the following roles:+

 

ACSMigrationService	ACSMonitoringService	ACSSettingsService
ACS	ACSFabric	ACSFrontEnd
ACSTableMaster	ACSTableServer	ACSWac
ADFS	ASAppGateway	BareMetal
BRP	CA	CPI
CRP	DeploymentMachine	DHCP
Domain	ECE	ECESeedRing
FabricRing	FabricRingServices	FRP
Gateway	HealthMonitoring	HRP
IBC	InfraServiceController	KeyVaultAdminResourceProvider
KeyVaultControlPlane	KeyVaultDataPlane	NC
NonPrivilegedAppGateway	NRP	SeedRing
SeedRingServices	SLB	SQL
SRP	Storage	StorageController
URP	UsageBridge	VirtualMachines
WAS	WASPUBLIC	WDS

 

A few things to note:

 

• This command takes some time for log collection based on which role logs are collected. Contributing factors include the time duration specified for log collection, and the numbers of nodes in the Azure Stack environment.
• After log collection completes, check the new folder created in the -OutputPath parameter specified in the command.
• A file called Get-AzureStackLogs_Output.log is created in the folder containing the zip files and includes the command output, which can be used for troubleshooting any failures in log collection.
• Each role has its logs inside an individual zip file.
• To investigate a specific failure, logs may be needed from more than one component.
  • System and Event logs for all infrastructure VMs are collected in the VirtualMachines role.
  • System and Event logs for all hosts are collected in the BareMetal role.
  • Failover Cluster and Hyper-V event logs are collected in the Storage role.
  • ACS logs are collected in the Storage and ACS roles.
• For more details, you can refer to the customer configuration file. Investigate the <Logs> tags for the different roles.

 

We apologize for any inconvenience and appreciate your time and interest in Azure Stack.

If you continue experience any issues with TP3 release, feel free to contact us.

TP3 Azure Stack Docs:

https://docs.microsoft.com/en-us/azure/azure-stack/

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-troubleshooting

https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-diagnostics

https://aka.ms/GetAzureStackLogs

 

 Thanks,

---

Gary Gallanes