none
ADLS Gen2 Rest API - continuation parameter

    Question

  • Hi,

    I wrote a simple program that uses Rest API to query an Azure Data Lake Gen2. 

    When I try to GET the list of the paths in my filesystem, the API return only the first 5000 paths. I've read the documentation (https://docs.microsoft.com/it-it/rest/api/storageservices/datalakestoragegen2/path/list) and I've found that there is this URI parameter:

    continuation: "The number of paths returned with each invocation is limited. If the number of paths to be returned exceeds this limit, a continuation token is returned in the response header x-ms-continuation. When a continuation token is returned in the response, it must be specified in a subsequent invocation of the list operation to continue listing the paths"..

    So I get this value from the first invocation and pass to the subsequent invocations (for each subsequent invocation I take the value from the previous and pass it). And here is the problem: sometimes, randomly, the invocation return a response equal to a response already obtained previously. 

    This is an example code:

    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(string.Format("https://{0}/{1}?directory={2}&resource=filesystem&recursive=true", accountName, filesystem, storagepath));
    
    req.Method = "GET";
    req.Headers.Add("Authorization", "Bearer " + token);
    
    HttpWebResponse res = (HttpWebResponse)req.GetResponse();
    
    using (var streamRes = new StreamReader(res.GetResponseStream()))
            {
                response = streamRes.ReadToEnd();
            }
    
    token_continuation = res.Headers["x-ms-continuation"];
    
    while (token_continuation != null)
    {
          req = (HttpWebRequest)WebRequest.Create(string.Format("https://{0}/{1}?directory={2}&continuation={3}&resource=filesystem&recursive=true", accountName, filesystem, storagepath, token_continuation));
               
          req.Method = "GET";
          req.Headers.Add("Authorization", "Bearer " + token);
    
          res = (HttpWebResponse)req.GetResponse();
    
          using (var streamRes = new StreamReader(res.GetResponseStream()))
          {
               response = streamRes.ReadToEnd();
    
          }
    
          token_continuation = res.Headers["x-ms-continuation"];
    }

    What am I doing wrong? How can I solve this problem?

    G.

    Thursday, February 21, 2019 11:04 AM

All replies

  • Hi gcocci,

    Have you tried changing the “maxResults” in the URI Parameters?

    Note: MaxResults: An optional value that specifies the maximum number of items to return. If omitted or greater than 5,000, the response will include up to 5,000 items.

    Hope this helps.

    Friday, February 22, 2019 6:20 AM
    Moderator
  • Hi,

    Thank you for your answer. Unfortunately this does not solve the problem because my items are more than 5000 and the maximum value that the parameter MaxResults can handle is 5000. If I set it with a higher value, he still considers 5000.

    G.

    Friday, February 22, 2019 8:34 AM
  • Hi gcocci,

    I’m working with the product team and get back to you when I have more information.

    Tuesday, February 26, 2019 5:40 AM
    Moderator
  • Thanks very much, I am looking forward to the response!
    Tuesday, February 26, 2019 10:47 AM
  • Hi Gcocci,

    If there are more entries, the API returns a continuation token which the client needs to send in a subsequent request to continue listing files. 

    Thursday, February 28, 2019 6:42 AM
    Moderator
  • Hi,

    Thank you for your answer. I know this behavior, I also described it in my question. The problem is that, with the code I reported, the successive calls to which I pass the continuation token sometimes return the same answer already given previously (this behavior does not seem to be cyclical).

    Is there anything wrong with my code?

    G.

    Friday, March 8, 2019 4:02 PM
  • Hi gcocci,

    In order, to investigate this issue further. Could you please provide an account name and a request ID or time frame when this occurred?

    Tuesday, March 12, 2019 6:22 AM
    Moderator
  • Did you figure this out? I'm having trouble: getting 403 Forbidden when I try to include continuation tokens (I get the first 5000 items, and then the 403). Thanks!
    Monday, May 13, 2019 5:07 PM