none
Possibly a bug with deleting and updating single entity where key contains escaped characters

    Question

  • Hi,

    Today we discoved an issue with deleting and updating a single entity where key (either PartitionKey or RowKey) contained escaped characters. To give you an example, let's say I have an entity where I want to save PartitionKey as "Url" and RowKey as "http://www.microsoft.com". Now "/" is an invalid character in RowKey value, what we do is we URL encode it before saving. So my RowKey for the saved entity would be "http%3a%2f%2fwww.microsoft.com". Entity is saved fine and I can retrieve the entity as well.

    Now I want to delete this entity. Based on the documentation, I should be escaping the RowKey as it is passed in the URL. The request for deleting this entity would be something like:

    http://myaccount.table.core.windows.net/MyTable(PartitionKey='Url',RowKey='http%253a%252f%252fwww.microsoft.com')?

    However when I perform this operation, I get a 400 error (One of the request inputs is not valid). I even tried to pass the RowKey as is i.e. "http%3a%2f%2fwww.microsoft.com" and still the same error.

    Interesting thing is that when I try and delete this single entity in a batch (consisting of this one entity only), delete works. I haven't tried update in batch yet but my suspicion is that it would work as well in the batch.

    Can somebody please look into it.

    Thanks

    Gaurav Mantri

    Cerebrata Software

    http://www.cerebrata.com

     

     

    Friday, June 10, 2011 7:43 AM

Answers

  • Hi Gaurav,

    After tried, I find out this issue occurs in querying entities when PartitionKey or RowKey contains special characters. The document Query Entities says these characters must be encoded but does not tell us which encoding method we should use.

    As you have used url encoding and it throws 404 error, so I think we cannot use that method. Instead, converting it to a base64 string works.

    public void Run()
    {
        CloudStorageAccount account = CloudStorageAccount.DevelopmentStorageAccount;
        CloudTableClient tableClient = account.CreateCloudTableClient();

        string tableName = "MyTable";
        tableClient.CreateTableIfNotExist(tableName);

        string partitionKey = "Url";
        string rawRowKey = "http://www.microsoft.com" + "_" + DateTime.Now.Ticks.ToString();
        string rowKey = HttpUtility.UrlEncode(rawRowKey);

        // Workaround.
        rowKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(rawRowKey));

        TableServiceContext context = tableClient.GetDataServiceContext();
        context.AddObject(tableName, new MyItem() { PartitionKey = partitionKey, RowKey = rowKey, Content = "Hello" });
        context.SaveChanges();

        MyItem item = context.CreateQuery<MyItem>(tableName)
                .Where(p => p.PartitionKey == partitionKey && p.RowKey == rowKey)
                .Single();

    }
    [DataServiceKey("PartitionKey", "RowKey")]
    public class MyItem
    {
        public string PartitionKey { get; set; }
        public string RowKey { get; set; }
        public DateTime Timestamp { get; set; }
        public string Content { get; set; }
    }

    Also you may want to contact our azure support to confirm which method we should use to encode the special characters and whether using url encoding is recommended.

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    • Marked as answer by Wenchao Zeng Monday, June 20, 2011 8:54 AM
    Monday, June 13, 2011 5:08 AM

All replies

  • Hi Gaurav,

    After tried, I find out this issue occurs in querying entities when PartitionKey or RowKey contains special characters. The document Query Entities says these characters must be encoded but does not tell us which encoding method we should use.

    As you have used url encoding and it throws 404 error, so I think we cannot use that method. Instead, converting it to a base64 string works.

    public void Run()
    {
        CloudStorageAccount account = CloudStorageAccount.DevelopmentStorageAccount;
        CloudTableClient tableClient = account.CreateCloudTableClient();

        string tableName = "MyTable";
        tableClient.CreateTableIfNotExist(tableName);

        string partitionKey = "Url";
        string rawRowKey = "http://www.microsoft.com" + "_" + DateTime.Now.Ticks.ToString();
        string rowKey = HttpUtility.UrlEncode(rawRowKey);

        // Workaround.
        rowKey = Convert.ToBase64String(Encoding.UTF8.GetBytes(rawRowKey));

        TableServiceContext context = tableClient.GetDataServiceContext();
        context.AddObject(tableName, new MyItem() { PartitionKey = partitionKey, RowKey = rowKey, Content = "Hello" });
        context.SaveChanges();

        MyItem item = context.CreateQuery<MyItem>(tableName)
                .Where(p => p.PartitionKey == partitionKey && p.RowKey == rowKey)
                .Single();

    }
    [DataServiceKey("PartitionKey", "RowKey")]
    public class MyItem
    {
        public string PartitionKey { get; set; }
        public string RowKey { get; set; }
        public DateTime Timestamp { get; set; }
        public string Content { get; set; }
    }

    Also you may want to contact our azure support to confirm which method we should use to encode the special characters and whether using url encoding is recommended.

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    • Marked as answer by Wenchao Zeng Monday, June 20, 2011 8:54 AM
    Monday, June 13, 2011 5:08 AM
  • Hi Wengchao,

    Thanks. Per your suggestion, I have opened a support ticket. Let's see what they come up with.

    Regards

    Gaurav

     

    Monday, June 13, 2011 10:11 AM
  • Url encoding does not solve the problem when the key contains characters that are disallowed. As Wengchao suggested, base64 is a better choice. Some users have introduced their own escape logic too if base64 does not work for some reason.

     

    Thanks,

    Jai

    Monday, June 13, 2011 8:41 PM
  • I read somewhere that Base64 encoding introduces the issue that / is one of the 64 codes used. However, I haven't looked at it. However, the Patterns & Practices Windows Azure books do use Base64 encoding so my worry may be overblown.
    Monday, June 13, 2011 9:57 PM
  • Thanks guys for responding. I tried to use the solution suggested by WengChao and it is not working for me. This is my code for creating the request URI (I am using REST API):

                 string PartitionKeyBase64Encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(AzureEntity.PartitionKey));
                string RowKeyBase64Encoded = Convert.ToBase64String(Encoding.UTF8.GetBytes(AzureEntity.RowKey));
                string requestUrl = string.Format("{0}/{1}(PartitionKey='{2}',RowKey='{3}')?", EndPoint, AzureEntity.TableName, PartitionKeyBase64Encoded.Replace("'", "''"), RowKeyBase64Encoded.Replace("'", "''"));

    Using this code, following request URI is created: https://cerebrataqa.table.core.windows.net/a002(PartitionKey='QmluZ3xJbWFnZXJ5U2VydmljZXxHZXRJbWFnZXJ5TWV0YWRhdGF8MS0wLTAtMA==',RowKey='aHR0cCUzYSUyZiUyZmRldi52aXJ0dWFsZWFydGgubmV0JTJmd2Vic2VydmljZXMlMmZ2MSUyZmltYWdlcnlzZXJ2aWNlJTJmaW1hZ2VyeXNlcnZpY2Uuc3ZjJTJmYmluYXJ5SHR0cHxDdXN0b21CaW5kaW5nX0lJbWFnZXJ5U2VydmljZXxC')?

    When I try to delete entity this way, I get a 404 (ResourceNotFound) error. I get the following response headers back:

    x-ms-request-id: 77b007e5-e622-4da9-971d-2a4c5cef0f2b
    x-ms-version: 2009-09-19
    Date: Tue, 14 Jun 2011 01:28:53 GMT
    Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0

    Can somebody see why this is happening. The entity in question does exist in the table. I even tried URLEncoding the Base64 encoded keys but in that case I get 400 error.

    Thanks

    Gaurav

     

     

    Tuesday, June 14, 2011 1:34 AM
  • Hi Gaurav,

     Did you insert the entity using the same encoding? Note that the table service does not decode - so what was suggested is that when keys have special characters that are invalid, it is best to encode/escape them (for insert and other operations). Another property can hold the original value.

     

    Thanks,

    Jai

    Tuesday, June 14, 2011 2:51 AM
  • Thanks Jai. Somehow I misinterpreted Wengchao's response and assumed that Table Storage service is doing the conversion from Base64 encoded string. Unfortunately this solution will not work as the key when inserted was not Base64 encoded.

    Gaurav

    Tuesday, June 14, 2011 6:21 AM
  • Hi Gaurav,

    If the key has been inserted, I think we don't have a solution for this as the same key cannot be used for querying or deleting the object. I will consider it as an issue of table storage. If a key cannot be used in querying or deleting, it should not be accepted when adding.

    According to Table Service API: When inserting entity, the PartitionKey and RowKey is sent in request body so they are not url encoded. But when querying and deleting entity, they are embedded in the url and url encoded. So maybe that is why we can use the same key to insert entity but cannot use it in querying and deleting. The automatic url encoding is causing this issue.

    A workaround may be to recreate the table (that means deleting all existing entities), and insert new entities using the Base64 encoded PartitionKey and RowKey if they contain special characters.

    Thanks,


    Wengchao Zeng
    Please mark the replies as answers if they help or unmark if not.
    If you have any feedback about my replies, please contact msdnmg@microsoft.com.
    Microsoft One Code Framework
    Tuesday, June 14, 2011 11:15 AM
  • I also discovered this problem while simply returning a single entity. I had encoded a PartitionKey with %23 instead of # - it works for inserting, and it works for returning a range query (in C#). It does NOT work correctly for returning a single entity by setting both PartitionKey and RowKey. By experimentation, I noticed that replacing %23 with %2523 (only on the call that was failing) works. (%25 is the escape for %, so I assume something somewhere is decoding %23 as # when it shouldn't.)

    While this is annoying, this at least provides a workaround for someone who has already stored data and can't go back and re-save everything!

    If you have the choice, I recommend not using % in either PartitionKey or RowKey.

    Hope that is useful to someone.


    ISV Architect Evangelist

    Thursday, August 23, 2012 2:06 AM