none
Azure Data Factory error in executing Data Lake Analytics activity: Job exists and can't be modified

    Question

  • Hello,

    I am using Data Factory v1 and I have several pipelines which have a Azure Data Lake Analytics (ADLA) activity. They have been working for several weeks with daily executions. The loads for today have failed all with the same error:

    Data Lake Analytics service threw cloud exception: 'JobExists: Job exists and can't be modified.'.

    When I have checked the ADLA jobs, was throwing an error at preparation time (not runtime!). The issue was because a partition with the same label was already existing:

    ALTER TABLE [mydb].[dbo].[mytable] DROP IF EXISTS PARTITION(@partitionDate);
    ALTER TABLE [mydb].[dbo].[mytable] ADD PARTITION(@partitionDate);

    Apparently Microsoft might have changed the "pre-compiler", and since yesterday is not smart enough to figure out there is a drop if exist before, and also  the variable @partitionDate is not being changed between both lines (they are one after the other). Nevertheless I have change it to be:

    ALTER TABLE [mydb].[dbo].[mytable] DROP IF EXISTS PARTITION(@partitionDate);
    ALTER TABLE [mydb].[dbo].[mytable] ADD IF NOT EXISTS PARTITION(@partitionDate);

    With this change what is happening is the following. I rerun the activity from the Data Factory Monitor and the activity fails after 3 seconds with the message Data Lake Analytics service threw cloud exception: 'JobExists: Job exists and can't be modified.'. But if I check the ADLA jobs, the job keeps executing and running finishing properly after 1m 30s!

    This is happening for all the pipelines I have with ADLA activities. I have tried to rerun older slices from the previous week and now they are running into the same issue explained.

    Is this an issue between ADF and ADLA internal APIs or I can do something from my end? I have already tried creating new pipelines with an ADLA activity but they throw the same error as well.

    Thank you for your support!



    Friday, March 16, 2018 11:27 AM

Answers

  • All,

    Issue is mitigated by Azure Data Lake Analytics hotfix.  Please re-try else reach out to Microsoft Support

    Thank you,

    Sundar - MSFT


    This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by AzureBI Monday, March 19, 2018 7:48 AM
    Friday, March 16, 2018 7:49 PM

All replies

  • I have the same issue since yesterday (2018-03-16)
    Friday, March 16, 2018 11:54 AM
  • Hello,

    I also experience that issue.

    From yesterday evening, all ADLA pipelines end with that error after 1-2 second but when I checked ADLA jobs, they finished. Also, re-run of historical slice ends with that error in ADF but ADLA job was submitted and was running.

    So, only ADF cannot catch the proper state of ADLA jobs.

    Friday, March 16, 2018 12:09 PM
  • Also same issue here since 16th March

    Microsoft BI / .NET developer

    Friday, March 16, 2018 2:00 PM
  • Same here

    And also using v1 
    • Edited by RasmusRas Friday, March 16, 2018 3:04 PM
    Friday, March 16, 2018 2:23 PM
  • I have also the same issue since 2018-03-16. Anyone an idea how to solve this issue?
    Friday, March 16, 2018 3:14 PM
  • All,

    Issue is mitigated by Azure Data Lake Analytics hotfix.  Please re-try else reach out to Microsoft Support

    Thank you,

    Sundar - MSFT


    This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by AzureBI Monday, March 19, 2018 7:48 AM
    Friday, March 16, 2018 7:49 PM