none
PUT and PATCH requests not working in Azure Data Lake Storage Gen2

    Question

  • Get API calls are working, when I invoke Gen2 Rest API calls using Shared key authorization.  But I am getting error for PUT and PATCH API calls. Below is the code for PATCH requests to append the data to a file :

    1) I created  Test1 folder in Gen2 and uploaded an empty file using Storage Explorer

    2) When I try to append data to the file using following code, I got 202 response code for append action and 403 response code for flush action.

          
      byte[] bytesArray = null;        
            file = new File("C:\\Work\\hello.txt");
            inputStream = new FileInputStream(file);
            bytesArray = new byte[(int) file.length()];
            inputStream.read(bytesArray);
            inputStream.close();
            
            int fileLength=bytesArray.length;        
            String accountName = "xxx";
            String fileSystem = "yyyy";
            String accessKey = "zzzz";
    
            Client client = ClientBuilder.newClient().property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true);
            String uri = "https://"+accountName+".dfs.core.windows.net/"+fileSystem+"/Test1/hello2.txt";
            
            WebTarget target = client.target(uri);
            String gmDate1 = getStorageDate();
    
            String StringToSign = "PATCH\n"
                    + "\n" // content encoding
                    + "\n" // content language
                    + bytesArray.length + "\n"// content length
                    + "\n" // content md5
                    + "application/octet-stream\n" // content type
                    + "\n" // date
                    + "\n" // if modified since
                    + "\n" // if match
                    + "\n" // if none match
                    + "\n" // if unmodified since
                    + "\n" // range
                    + "x-ms-date:" + gmDate1 + "\n" + "x-ms-version:2018-11-09" + "\n" // headers
                    + "/mftdatalakegen2/datalakefs1/Test1/hello2.txt\naction:append\n"; // resources
            
            String authorizationStringToHash = null;
            authorizationStringToHash = getAuthenticationString(accountName, accessKey, StringToSign);
            
            Response response = target.queryParam("position", "0")
                    .queryParam("action", "append")
                    .request().header("Authorization", authorizationStringToHash)
                    .header("content-length", "" + bytesArray.length)
                    .header("content-type", "application/octet-stream")
                    .header("x-ms-date", gmDate1)
                    .header("x-ms-version", "2018-11-09")
                    .method("PATCH", Entity.entity(bytesArray, MediaType.APPLICATION_OCTET_STREAM));
                    
            String gmDate2 = getStorageDate();
            StringToSign = "PATCH\n"
                    + "\n" // content encoding
                    + "\n" // content language
                    + "0\n"// content length
                    + "\n" // content md5
                    + "\n" // content type
                    + "\n" // date
                    + "\n" // if modified since
                    + "\n" // if match
                    + "\n" // if none match
                    + "\n" // if unmodified since
                    + "\n" // range
                    + "x-ms-date:" + gmDate2 + "\nx-ms-version:2018-11-09\n"//"x-ms-client-request-id:"+requestId
                    + "/mftdatalakegen2/datalakefs1/Test1/hello2.txt\naction:flush"
                    + "\nposition:"+fileLength; // resources
    
            authorizationStringToHash = getAuthenticationString(accountName, accessKey, StringToSign);
            
            Client client1 = ClientBuilder.newClient().property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true);
            client1.property(ClientProperties.SUPPRESS_HTTP_COMPLIANCE_VALIDATION, true);
            
            String uri1 = "https://mftdatalakegen2.dfs.core.windows.net/datalakefs1/Test1/hello2.txt";
            WebTarget target1 = client1.target(uri1);        
            response = target1
                    .queryParam("action", "flush")
                    .queryParam("position", ""+fileLength)
                    .request()
                    .header("Authorization", authorizationStringToHash)
                    .header("Content-Length", "0")
                    .header("x-ms-date", gmDate2)
                    .header("x-ms-version", "2018-11-09")
                    .method("PATCH");

    3) I am using below method to get Shared Key authorization and Date


    public static String getStorageDate() {
            Date d = new Date(System.currentTimeMillis());
            SimpleDateFormat f = new SimpleDateFormat("EEE, dd MMM yyyy HH:mm:ss zzz");
            f.setTimeZone(TimeZone.getTimeZone("GMT"));
            String dateString = f.format(d);
            return dateString;
        }
    
        private static String getAuthenticationString(String accountName, String accessKey, String stringToSign)
                throws Exception {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(Base64.decodeBase64(accessKey), "HmacSHA256"));
            String authKey = new String(Base64.encodeBase64(mac.doFinal(stringToSign.getBytes("UTF-8"))));
            String auth = "SharedKey " + accountName + ":" + authKey;
            return auth;
        }


    Please let me know the issue with the above code.

    Also, if I update the file with 1 character and when I change the position to 1 for append action and 1+fileLength for flush action. I get the below error for append action also :

    403 - Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

    Thanks,

    Ramya





    • Edited by Ramya_K Tuesday, April 23, 2019 9:53 AM
    Monday, April 22, 2019 3:44 PM

Answers

  • Issue is resolved after removing the content-length (from the header and the signature) and setting the content-type.
    We cannot set the content-length as Jersey automatically sets the content-length header based on the payload data.

    StringToSign = "PATCH\n"
                    + "\n" // content encoding
                    + "\n" // content language
                    + "\n"// content length
                    + "\n" // content md5
                    + MediaType.TEXT_PLAIN+"\n" // content type
                    + "\n" // date
                    + "\n" // if modified since
                    + "\n" // if match
                    + "\n" // if none match
                    + "\n" // if unmodified since
                    + "\n" // range
                    + "x-ms-date:" + gmDate2 + "\nx-ms-version:2018-11-09\n"//"x-ms-client-request-id:"+requestId
    	        + "/"+accountName+"/datalakefs1/Test1/hello2.txt" // resources				+ 
    		+ "\naction:flush\nposition:"+fileLength; // resources
    
      authorizationStringToHash = getAuthenticationString(accountName, accessKey, StringToSign);
            
       Client client1 = ClientBuilder.newClient().property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true);
       client1.property(ClientProperties.SUPPRESS_HTTP_COMPLIANCE_VALIDATION, true);
    	        
        Entity<?> empty = Entity.text("");
        WebTarget target1 = client1.target(uri);        
        response = target1
    					.queryParam("action", "flush")
    					.queryParam("position", ""+fileLength)
    					.request()
    					.header("Authorization", authorizationStringToHash)
    					.header("Content-Type", MediaType.TEXT_PLAIN)
    					.header("x-ms-date", gmDate2)
    					.header("x-ms-version", "2018-11-09")				
    					.method("PATCH", empty);

    • Marked as answer by Ramya_K Friday, April 26, 2019 7:08 AM
    Friday, April 26, 2019 7:08 AM

All replies

  • Hi Ramya_K,

    This error occurs if the service principal you are using for Azure Data Lake Storage Gen2 is not assigned Delegated permissions for the service principal app registered in Azure AD. 

    For more details, refer “Access control in Azure Data Lake Storage Gen2”.

    Hope this helps.

    Thursday, April 25, 2019 6:48 AM
    Moderator
  • Issue is resolved after removing the content-length (from the header and the signature) and setting the content-type.
    We cannot set the content-length as Jersey automatically sets the content-length header based on the payload data.

    StringToSign = "PATCH\n"
                    + "\n" // content encoding
                    + "\n" // content language
                    + "\n"// content length
                    + "\n" // content md5
                    + MediaType.TEXT_PLAIN+"\n" // content type
                    + "\n" // date
                    + "\n" // if modified since
                    + "\n" // if match
                    + "\n" // if none match
                    + "\n" // if unmodified since
                    + "\n" // range
                    + "x-ms-date:" + gmDate2 + "\nx-ms-version:2018-11-09\n"//"x-ms-client-request-id:"+requestId
    	        + "/"+accountName+"/datalakefs1/Test1/hello2.txt" // resources				+ 
    		+ "\naction:flush\nposition:"+fileLength; // resources
    
      authorizationStringToHash = getAuthenticationString(accountName, accessKey, StringToSign);
            
       Client client1 = ClientBuilder.newClient().property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true);
       client1.property(ClientProperties.SUPPRESS_HTTP_COMPLIANCE_VALIDATION, true);
    	        
        Entity<?> empty = Entity.text("");
        WebTarget target1 = client1.target(uri);        
        response = target1
    					.queryParam("action", "flush")
    					.queryParam("position", ""+fileLength)
    					.request()
    					.header("Authorization", authorizationStringToHash)
    					.header("Content-Type", MediaType.TEXT_PLAIN)
    					.header("x-ms-date", gmDate2)
    					.header("x-ms-version", "2018-11-09")				
    					.method("PATCH", empty);

    • Marked as answer by Ramya_K Friday, April 26, 2019 7:08 AM
    Friday, April 26, 2019 7:08 AM
  • Hi Ramya_K,

    Glad to know that your issue has resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread. 

    Friday, April 26, 2019 7:16 AM
    Moderator