Which Azure Data Factory resource to use with HDInsights Jupyter Notebook (Pyspark 3) RRS feed

  • Question

  • My team and I are trying to configure an azure data factory (ADF) pipeline that runs some code I wrote for an HDInsight job. The code runs just fine when I open the Jupyter Notebook in my HDInsight cluster, but when I try to run it from ADF, it fails. It tells me I am getting the following error code:

    "errorCode": "2303", "message": "Hadoop job failed with exit code '64'.

    It also tells me to check the logs in ambari, but I can't find where those logs are. Can someone tell me how they have done this? I've tried using "Hive" and "Spark" resources in ADF and pointing the script file path at my code. Not sure what I'm doing wrong?

    Tuesday, September 3, 2019 5:50 PM

All replies

  • Hi Justin,

    Azure Data Factory supports the following transformation activities that can be added to pipelines either individually or chained with another activity.

    For more details, refer “Transform data in ADF”.

    To debug:

    Use getDebugInfo which Specifies when the log files are copied to the Azure storage used by HDInsight cluster (or) specified by YourLinkedService. Allowed values: None, Always, or Failure. Default value: None.

    For more details, refer “Transform data using Hive activity in Azure Data Factory”.


    You can track and debug Hadoop jobs running on HDInsight cluster using YARN UI:

    The YARN ResourceManager UI runs on the cluster head node, and is accessed through the Ambari web UI. Use the following steps to view the YARN logs:

    1. In a web browser, navigate to Replace CLUSTERNAME with the name of your HDInsight cluster.
    2. From the list of services on the left, select YARN.
    3. From the Quick Links dropdown, select one of the cluster head nodes and then select ResourceManager logs. You are presented with a list of links to YARN logs.

    For more details, refer “Manage logs for an HDInsight cluster”.

    Hope this helps.      


    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Wednesday, September 4, 2019 4:49 AM
  • Hi Justin,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Friday, September 6, 2019 9:04 AM
  • Hi Justin,

    Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

    Tuesday, September 10, 2019 4:23 AM