none
How to run shell script in Azure datafactory V2? RRS feed

All replies

  • Hello Vignesh,

    You can now directly run commands, scripts, and your own custom code, compiled as an executable. 

    You can directly execute a command using Custom Activity. The following example runs the "echo hello world" command on the target Azure Batch Pool nodes and prints the output to stdout.

    {
      "name": "MyCustomActivity",
      "properties": {
        "description": "Custom activity sample",
        "activities": [{
          "type": "Custom",
          "name": "MyCustomActivity",
          "linkedServiceName": {
            "referenceName": "AzureBatchLinkedService",
            "type": "LinkedServiceReference"
          },
          "typeProperties": {
            "command": "cmd /c echo hello world"
          }
        }]
      }
    }

    Note: The custom activity runs your customized code logic on an Azure Batch pool of virtual machines.

    For more details, refer “Use custom activities in an Azure Data Factory pipeline”.

    You could use Hadoop hive/pig scripts activity in Azure Data Factory.

    Hive activity in ADF:

    {
        "name": "Hive Activity",
        "description": "description",
        "type": "HDInsightHive",
        "linkedServiceName": {
            "referenceName": "MyHDInsightLinkedService",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "scriptLinkedService": {
                "referenceName": "MyAzureStorageLinkedService",
                "type": "LinkedServiceReference"
            },
            "scriptPath": "MyAzureStorage\\HiveScripts\\MyHiveSript.hql",
            "getDebugInfo": "Failure",
            "arguments": [
                "SampleHadoopJobArgument1"
            ],
            "defines": {
                "param1": "param1Value"
            }
        }
    }

    Pig activity in ADF:

    {
        "name": "Pig Activity",
        "description": "description",
        "type": "HDInsightPig",
        "linkedServiceName": {
            "referenceName": "MyHDInsightLinkedService",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "scriptLinkedService": {
                "referenceName": "MyAzureStorageLinkedService",
                "type": "LinkedServiceReference"
            },
            "scriptPath": "MyAzureStorage\\PigScripts\\MyPigSript.pig",
            "getDebugInfo": "Failure",
            "arguments": [
                "SampleHadoopJobArgument1"
            ],
            "defines": {
                "param1": "param1Value"
            }
        }   
    }

    For more details, refer the below links:

    Transform data using Hadoop Hive activity in ADF

    Transform data using Hadoop Pig activity in ADF

    Hope this helps.      

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Tuesday, September 3, 2019 9:51 AM
    Moderator
  • Hi Vignesh,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.


    Wednesday, September 4, 2019 10:39 AM
    Moderator
  • Hi Vignesh,

    Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

    Friday, September 6, 2019 9:11 AM
    Moderator