none
NotFound error on call to Data Lake Store Create

    Question

  • When this block of code is executed:

    using (var stream = new MemoryStream())
    {
        var writer = new StreamWriter(stream);
        using (var csv = new CsvWriter(writer))
        {
            foreach (var row in records)
            {
                foreach (var cell in row)
                {
                    csv.WriteField(cell ?? "");
                }
                csv.NextRecord();
            }
    
            writer.Flush();
            stream.Position = 0;
            _adlsFileSystemClient.FileSystem.Create("ACCOUNTNAME", destinationFilePath, stream, true);
        }
    }

    I get the error:

    "Microsoft.Azure.Management.DataLake.Store.Models.AdlsErrorException: Operation returned an invalid status code 'NotFound'\r\n   at Microsoft.Azure.Management.DataLake.Store.FileSystemOperations.<CreateWithHttpMessagesAsync>d__16.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.Azure.Management.DataLake.Store.FileSystemOperationsExtensions.<CreateAsync>d__23.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServi
    ces.TaskAwaiter.GetResult()\r\n   at Microsoft.Azure.Management.DataLake.Store.FileSystemOperationsExtensions.Create(IFileSystemOperations operations, String accountName, String directFilePath, Stream streamContents, Nullable`1 overwrite, Nullable`1 syncFlag)\r\n   at Submission#0.<>c__DisplayClass5_1.<Run>b__5() in C:\\Source\\Repos\\...\\run.csx:line 191"

    First of all, a NotFound error is weird since I'm creating a file. This code works for other files and destinations as well as if I cut down the data getting fed into the stream. Any ideas what's going wrong and how I can get around this issue?


    Thursday, December 22, 2016 8:06 PM

Answers

All replies

  • Does your destination path refer to any folder that doesn't exist?

    Jason Chen, Windows Azure PM

    Thursday, December 22, 2016 11:39 PM
  • Yep, the path exists. If the file is smaller, it will copy successfully.
    Friday, December 23, 2016 3:18 AM
  • How big is your file when the problem occurred? Have you tried to create destination file initially with no data, then append each row to it?

    Jason Chen, Windows Azure PM

    Friday, December 23, 2016 8:33 PM
  • How big is your file when the problem occurred? Have you tried to create destination file initially with no data, then append each row to it?

    Jason Chen, Windows Azure PM

    Jason is on to something here. For performance reasons, the SDK is using a fixed Content-Length request body. IIS has a default limit on how big this request body can be, which results in a 404.13 status code if it is exceeded:

    https://support.microsoft.com/en-us/kb/943891

    https://www.iis.net/configreference/system.webserver/security/requestfiltering/requestlimits

    Specifically, the limit is 30000000 bytes or about 28.6MB. Please try to limit your stream sizes to < 28MB (ideally 4MB chunks are the best size in terms of performance and retryability) and see if that helps resolve your issue.


    Wednesday, January 4, 2017 12:12 AM