Unable to See result of hive script when transforming data in Azure data lake with Hdinisght cluster having Azure data lake as primary storage in ADF (The pipleine result is showing as success) RRS feed

  • Question

  • We are trying to transform data in Azure data lake folder and move it to another azure data lake folder through ADF

    We have created a Hdinsight cluster with Data Lake storage as primary Storage.

    We created a pipeline in ADF with a Hive Activity and an outputdataset

    As per Azure documentation,Since Any HDinisght cluster cannot be linked to a Azure data Lake store in ADF ,We have linked the hdinsight cluster(which is shaving ADL as primary storage) to AzureBlobLinkedService in Pipeline

    We have placed our transformation script .hql file in Azure blob Storage,and that particular .hql file consists of queries related to azure data lake store-creating tables,loading data into table from azure data lake..etc

    When we try to run this pipeline,the ADF shows a success scenario,but we cannot see any table or data loaded into hive

    As it is a success scenario we are not getting any error logs,we doubt if the query is not getting executed at all,even though its showing success in pipeline

    The same thing is working if we try to create a HDinsight cluster with Azure blob and we can see the tables getting created as output

    can you please help with the above scenario with an example of how to execute a transformation job of Hive queries in Hdinsight cluster which is having Azure data lake as primary storage in ADF

    Thursday, August 3, 2017 1:52 PM

All replies

  • Send me Activity Code and Hive Script file



    Tuesday, August 8, 2017 1:15 PM