locked
Using JsonSerDe in a hive script in an on-demand hdinsight cluster RRS feed

  • Question

  • I have written a hive script that works fine in an existing HDInsight cluster. But when I instantiate the script via the Azure Data Factory in an on demand cluster, I get the following error:

    Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2214)
    at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143)
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:316)
    ... 19 more
    I use JsonSerDer to load data from a JSON file into a hive table. In my existing cluster the class is present without any additional configuration. Is it necessary to specify the JsonSerDer JAR file explicitly for an on demand cluster and if so how is this possible?

    Saturday, May 2, 2020 8:20 PM

Answers

  • Hello,

    You will receive this error message, when you are missing the Jar file.

    You can add the Jar file in the ADF Hive activity as shown.

    Go to the activity => File Path => Browse Local and add the Jar file.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Monday, May 4, 2020 6:29 AM

All replies

  • Hello,

    You will receive this error message, when you are missing the Jar file.

    You can add the Jar file in the ADF Hive activity as shown.

    Go to the activity => File Path => Browse Local and add the Jar file.

    Hope this helps. Do let us know if you any further queries.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Monday, May 4, 2020 6:29 AM
  • Hello,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Tuesday, May 5, 2020 1:11 PM
  • Hello,

    Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

    Thursday, May 7, 2020 12:10 PM