none
Can we execute shell script on HDI cluster edge node using Azure Data Factory pipeline activity.

    Question

  • Hi Team,

    I am using ADF v2 and building pipeline. I have custom shell script deployed on HDI cluster and i want to execute them as part of pipeline. Is there any way i can execute shell scripts from a data factory pipeline activity.

    Thanks in advance. Looking forward for your response.

    Regards,

    Anikesh

    Monday, May 28, 2018 1:48 PM

All replies

  • Hi Anikesh,

    You could try to execute a shell script on an HDI edge node using a custom activity:

    https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-dotnet-custom-activity

    This section of the above link describes running a command with a custom activity:

    https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-dotnet-custom-activity#executing-commands

    Your custom activity would be an SSH command to your HDI edge node to execute the script located on the edge node.  The SSH command itself would be executed on an Azure Batch node.  You would have to figure out a way of getting your SSH key on the Azure Batch node.  You may be able to do this by creating a custom virtual machine image with the SSH key stored inside it, which Azure Batch would use to create the Batch node:

    https://docs.microsoft.com/en-us/azure/batch/batch-linux-nodes#virtual-machine-image-reference

    Thursday, May 31, 2018 10:26 PM
    Moderator