Intermittent "System.Xml.XmlException: Data at the root level is invalid" while reading data with CloudTableQuery<T>

Question

Hi, my setup is similar to following (1.7 Azure SDK):

var cloudStorageAccount = CloudStorageAccount.Parse(storageConnectionString);
var context = new TableServiceContext(cloudStorageAccount.TableEndpoint.AbsoluteUri, cloudStorageAccount.Credentials)
{
    MergeOption = MergeOption.NoTracking, //we just read the data, no need in tracking
    RetryPolicy = RetryPolicies.RetryExponential(5, TimeSpan.FromSeconds(1))
}

string tableQuery = "PartitionKey lt '20120704'"; // just a sample query

CloudTableQuery<MyEntity> query = context.CreateQuery<GenericEntity>("TableName")
    .AddQueryOption("$filter", tableQuery);
    .AsTableServiceQuery(); //to use pagination processing implemented in CloudTableQuery<T>

foreach (var entity in query)
{
  //do my work
}

This code works perfectly on my local environment, processing about 8 million rows in "foreach" loop.

But when I run exactly the same code on production environment I sometimes get the following exception (after processing several million rows successfully):

   Description: System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
   at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.ExecuteAndWait()
   at Microsoft.WindowsAzure.StorageClient.TaskImplHelper.ExecuteImplWithRetry(Func`2 impl, RetryPolicy policy)
   at Microsoft.WindowsAzure.StorageClient.ResultSegment`1.GetNext()
   at Microsoft.WindowsAzure.StorageClient.CommonUtils.<LazyEnumerateSegmented>d__0`1.MoveNext()
   at MyProgram.TestRun()

Obviously there is something wrong with response received from Azure Table Storage, but I could not find the way to get to the raw response data to see what's going on there. And strangely enough RetryPolicy looks like doesn't handle XmlExceptions so the very first error stops my processing, without a chance to restart it from latest ContinuationToken.

So my questions are:

1. Is there any way to enable logging of raw responses (before XML deserialization) for CloudTableQuery<T> (or underlying DataServiceQuery<T>)?
2. What can you suggest to fix this issue? I'm thinking about using CloudTableQuery<T>.BeginExecuteSegmented/EndExecuteSegmented (I wonder why there is no synchronous method) to get hold of continuation tokens, so that at least I can resume processing later.

Answer 1

Hi,

I suggest you connect to Azure VM and use Fiddler to capture the request from the cloud application, fiddler can help you get the raw responses. If you got the message, would you like to share it? I believe these info will help us fix the problem.

BR,

Arwind

---

Please mark the replies as answers if they help or unmark if not. If you have any feedback about my replies, please contact [e-mail] Microsoft One Code Framework

Answer 2

Just to update on this thread: I've tried to detect issues with fiddler (ran processing about ten times with different settings trying to replicate the error), but surprisingly everything worked correctly even when I disabled fiddler... will post another update if/when I capture the error again

Answer 3

To summarise the thread for those who will search on this topic.

I've seen this exception many times, but never when Fiddler was intercepting the data flow. So somehow Fiddler "fixed" the data flow... maybe ordering of data packets, delays... don't know.

I've solved the issue by adding a proper exception handling (retries) for XmlException to Azure SDK and compiling my own Microsoft.WindowsAzure.StorageClient.dll (you can grab and compile my version from github, it also includes a fix for another similar issue with StorageClientException been thrown during processing)