none
Error while trying to execute a hive script in data factory v2

    Question

  • I'm having some trouble trying to execute hive scripts using data factory. I'm creating pipelines to copy data to a data lake and process that data using hive.

    Here we have a cluster on azure so I'm not using an "On-demand HDInsight", but instead I'm trying to configure a linked service for that specific cluster.

    For example, I'm trying to execute an "Insert" command, so I create a .hql file and put it on a storage blob.

    The code I'm trying to execute is this: (I changed the object names, but it's exactly like that) I've already ran that code on hive so I know it's working. 

    use DB_NAME;
    
    INSERT OVERWRITE TABLE TARGET_TABLE
    SELECT 
    FIELD1,
    FIELD2,
    FIELD3,
    FIELD4,
    FIELD5,
    FIELD6,
    FIELD7
    FROM
    SOURCE_TABLE
    WHERE FIELD1 IS NOT NULL
    AND FIELD2 IS NOT NULL
    AND FIELD3 IS NOT NULL
    AND FIELD4 IS NOT NULL
    AND FIELD5 IS NOT NULL
    AND FIELD6 IS NOT NULL
    AND FIELD7 IS NOT NULL;

    The error I get when I try to run the pipeline is the error bellow:

    {
        "errorCode": "2300",
        "message": "Hadoop job submission failed. Job: 51540c3c-3e1d-4c92-b46b-c68b1039b834, Cluster: https://<clustername>.azurehdinsight.net/. Error: A task was canceled..",
        "failureType": "UserError",
        "target": "Hive1"
    }

    I just can't find any reference on the error above. 

    Obs.: I tried uploading images with more information about the configurations and the pipeline I'm using but apparently I can't til they verify my account...

    Tuesday, August 21, 2018 4:40 PM

All replies

  • Please upload those information once your account is verified. It will be useful for us to identify the issue with those information.
    Thursday, August 30, 2018 10:34 PM
    Moderator