none
Secure string type Pipeline parameter RRS feed

  • Question

  • Hello Team,

    I need to pass a password as a pipeline parameter but that needs to be secured; so i had defined the type as secure string.

    But after updating the type to secure string from string, I am not able to pass the parameter value to a pipeline variable.

    I am getting the below error:

    "errorCode": "BadRequest", "message": "The variable 'Test' of type 'String' cannot be initialized or updated with value '{\r\n \"type\": \"SecureString\",\r\n \"value\": \"Test\"\r\n}' of type 'Object'. The variable 'Test' only supports values of types 'String'.", "failureType": "UserError", "target": "Set Variable1"

    So how can one pass a secure string type pipeline parameter to a pipeline variable (I am using set variable activity)

    Tuesday, July 16, 2019 7:01 AM

Answers

  • Thank you for this interesting problem.  At first this seems to be difficult, but I found a way to extract the value from the secureString.  In a set variable (string type) I used:

    @{json(string(pipeline().parameters.password)).value}

    I was inspired by the error code you provided.  The error code contained the JSON object of the password.  I noticed that this is what i got when I tried using the secureString as input to my own Web activity.  Before it can be used by web activity, it must be changed into a string, or so I reasoned.  Therefore, why not try changing to string, then changing back to JSON, grabbing the value, and then reconverting to string.

    Since this is now an obvious security hole, I don't know how long it will work.

    Wednesday, July 17, 2019 6:26 PM
    Moderator
  • I have reached out internally raising 4 points:

    1. How to use with Set Variable activity
    2. Usage in Web activity
    3. The behavior of default parameters
    4. Use case of secure string

    I have gotten some response:

    1. Set Variable does not support SecureString type yet.
    2. Developer is interested in the JSON of Nandan Hegde's pipeline with the Oauth.  Also reccomended using MSI authentication for Azure Resources.
    3. Default Values for secure string parameters in Debug and TriggerNow needs to be manually provided.  This is a known issue.  Please provide the values manually until it is fixed.

    From what I have seen, in most cases you can just use regular string.  SecureString seems useful only for cases where a user will write a value to be used by the machine, but the user must never read the value.

    In regards to point #3, I have tried creating a scheduled trigger, and setting the secureString's value in that.  I found the secureString works with scheduled triggers.  Each triggered run of PriyaJha's test pipeline works.  If the definition of the trigger is changed from the UI Trigger pane, the value is untouched because the parameters are not set from there.  However if the trigger is changed from the pipeline>edit trigger pane, the SecureString value must be re-entered.  Since it works with scheduled triggers, I expect the other triggers to work the same way.

    I am working on a way to fetch secrets from Key Vault for use inside Data Factory variables, etc.


    Thursday, September 12, 2019 12:44 AM
    Moderator

All replies

  • Hello Nandan Hegde, and thank you for your inquiry.

    I have reproduced the error you are getting.  Thank you for the details.  I also found a caveat when trying to use them with Git.

    Could you please explain to me what you are trying to do?  In any case where you need to use the password, you could reference the pipeline parameter directly.  The main use case involving passwords and variables I can think of would be to construct an array of passwords, to be used in a parameterized dataset and linked service, however that would require the Append Variable activity, which I have found the secure string CAN be used in.

    Thank you for your patience,
    Martin Jaffer

    Tuesday, July 16, 2019 6:49 PM
    Moderator
  • Hello Martin,

    I am trying to generate an Outh token via web activity:

    wherein the URL value is:

    @concat('https://login.microsoftonline.com/',pipeline().parameters.TenantID,'/oauth2/token')

    Body is:

    @concat('grant_type=client_credentials&resource=https://management.azure.com&client_id=',pipeline().parameters.ClientID1,'&client_secret=',encodeUriComponent(pipeline().parameters.ClientSecret1))

    wherein I am trying to send the clientId and client secret via pipeline parameters.

    When I have kept the type of the parameter as string and passing the value ,the web activity is running successfully and generating the Oauth token.

    But since client secret should not be visible for others, so I updated the data type as SecureString.

    After updating as secure string ,the web activity is failing with the error:

    { "errorCode": "2108", "message": "{\"error\":\"unauthorized_client\",\"error_description\":\"AADSTS700016: Application with identifier '{\\\"type\\\":\\\"SecureString\\\",\\\"value\\\":\\\"xxxx-xxxx-xxxxx-xxxx-xxxxx\\\"}' was not found in the directory

    }

    where xxxx-xxxx-xxxx-xxxx is the original value which i have scrubbed in this thread.

    So when we define the parameter as secure string, do we need to add some other condition while consuming the parameter in another activity to get the original value?

    Wednesday, July 17, 2019 5:46 AM
  • Thank you for this interesting problem.  At first this seems to be difficult, but I found a way to extract the value from the secureString.  In a set variable (string type) I used:

    @{json(string(pipeline().parameters.password)).value}

    I was inspired by the error code you provided.  The error code contained the JSON object of the password.  I noticed that this is what i got when I tried using the secureString as input to my own Web activity.  Before it can be used by web activity, it must be changed into a string, or so I reasoned.  Therefore, why not try changing to string, then changing back to JSON, grabbing the value, and then reconverting to string.

    Since this is now an obvious security hole, I don't know how long it will work.

    Wednesday, July 17, 2019 6:26 PM
    Moderator
  • Was this able to help you @Nandan Hegde?
    Thursday, July 18, 2019 9:01 PM
    Moderator
  • Thank you very much Martin.

    It was very helpful.:)

    Friday, July 19, 2019 2:59 AM
  • Hi @MartinJaffer

    I used the above mentioned code provided by you and had the following observations:

    1) I created one parameter and specified it as SecureString and published the pipeline.

    

    After that i used your method and stored the parameter value in a variable by using the code: 

    After that i have an if Condition with the expression: 

    And 2 wait conditions in True and False activity.

    When i run the pipeline i got the following output: 

    And the Set Variable1 output is as follows: 

    And this is the expected output.

    2) But when i refresh the page, the SecureString Parameter is masked as follows:

    And after running the pipeline, i am getting wrong output, instead of wait1 activity in If Condition i am getting wait2 activity: 

    And the output of Set Variable1 is as follows: 

    Can you please explain why after refreshing the page, output is varying and how can i get the original value of the parameter back.


    • Edited by PriyaJha Monday, September 9, 2019 12:17 PM
    Monday, September 9, 2019 12:13 PM
  • Oh wow, that is interesting.  Thank you PriyaJha.  I see what you run into.

    I publish, then do two 'trigger now' runs.  One time I leave the default value ('*******'), the other time I type in '1234'.  One fails , the other succeeds.

    I will escalate the issue internally, but I expect they will tell me it is because of the hack I provided.

    Until I have a solid answer, may I suggest using Azure Key Vault?  There are some benefits in using Azure Key Vault over SecureString.  Using Azure Key Vault with Azure Data Factory (v2) allows you to have all the places you would enter your credentials, point to the same secret in Key Vault.  Then, when you need to change password, you can just change it once, in Key Vault, instead of dozens of times in Data Factory.

    I also noticed, that the 'Rerun' option in the in the pipeline monitoring refuses attempts to rerun when there is a secureString involved.

    Tuesday, September 10, 2019 9:15 PM
    Moderator
  • Thank you Martin for the reply.

    But using of keyvault is like creation of an extra activity of azure function to pull in the value from the key vault.

    So if there are so many issues w.r.t secure string , can you tell me what exactly is the use case for the secure string in ADF v2.(meaning why was it created)

    Note: The above scenario is not w.r.t linked service creation wherein we can use key vault but to assign the secret value to a variable.

    Wednesday, September 11, 2019 3:08 AM
  • I have reached out internally raising 4 points:

    1. How to use with Set Variable activity
    2. Usage in Web activity
    3. The behavior of default parameters
    4. Use case of secure string

    I have gotten some response:

    1. Set Variable does not support SecureString type yet.
    2. Developer is interested in the JSON of Nandan Hegde's pipeline with the Oauth.  Also reccomended using MSI authentication for Azure Resources.
    3. Default Values for secure string parameters in Debug and TriggerNow needs to be manually provided.  This is a known issue.  Please provide the values manually until it is fixed.

    From what I have seen, in most cases you can just use regular string.  SecureString seems useful only for cases where a user will write a value to be used by the machine, but the user must never read the value.

    In regards to point #3, I have tried creating a scheduled trigger, and setting the secureString's value in that.  I found the secureString works with scheduled triggers.  Each triggered run of PriyaJha's test pipeline works.  If the definition of the trigger is changed from the UI Trigger pane, the value is untouched because the parameters are not set from there.  However if the trigger is changed from the pipeline>edit trigger pane, the SecureString value must be re-entered.  Since it works with scheduled triggers, I expect the other triggers to work the same way.

    I am working on a way to fetch secrets from Key Vault for use inside Data Factory variables, etc.


    Thursday, September 12, 2019 12:44 AM
    Moderator