Can't update a pipeline in the UI that was deployed as an ARM template


  • Hi,

    I have a number of pipelines that I created as ARM templates and then deployed into Data Factory v2.  They deploy fine, and execute fine.

    But when I make a change (*any* change, no matter how trivial) I get the "Publishing failures" message "document creation or update failed because there were duplicate properties in the json payload"

    If they've deployed fine, and executed fine.... what could these duplicates possibly be?

    Is there a way to get more detail on this error message?


    Tuesday, April 10, 2018 3:30 PM

All replies

  • Hi JohnNev,

    You might be able to find some more information in Azure Monitor activity logs as outlined here:

    If you can find the specific action which failed, you can find a description of the action here:

    You might also find more information on failed deployment operations:

    I tried to replicate your issue by creating duplicate key:values in my JSON template for my ADFv2 pipelines then deploying them, but each time it seems the ARM template parser automatically took them out and launched them successfully.  Even so, it is possible you have duplicate entries in your template which are not getting caught.  I would double check all the entries in your JSON template and make certain there are no unnecessary or duplicate blocks.

    Monday, April 16, 2018 11:31 PM
  • Hi JohnNev,

    I just reread your question and I think I misunderstood it.  I was able to deploy a pipline in with a template and make changes to it in the UI.  I'll keep trying to replicate your issue and will report back if I am successful.

    Tuesday, April 17, 2018 12:00 AM
  • Any updates on the issue. I am also facing the same issue.
    Sunday, May 6, 2018 3:45 AM
  • Hiya,

    Here is a more detailed breakdown of what I do and what I get:

    1. Start with an empty Data Factory
    2. Deploy a pipeline from an ARM template (-> it deploys without issue)
    3. Run the pipeline (-> it Succeeds, and I can see the data arrive in my db)
    4. Make a change to the Pipeline in the UI; it can be any change no matter how trivial
    5. Click 'Publish All' in the UI
    6. The publish fails with the error message "The document creation or update failed because there were duplicate properties in the json payload."
    7. Double check your original ARM template for possible duplicates
    8. Hang head in hands because there are none

    The specific pipeline I am working on at the moment uses Blob source to Azure SQL db destination with almost no transformations in between, so it's small enough to scan by eye to see if there is some subtle duplicate somthing-or-other tucked away.  But there really isn't.

    @Jason_J thanks for the link to the resource manager deployment operations page.  I've followed those instructions and can see the error in the Activity Log.  But at the end of the log there is only:

        "properties": {
            "statusCode": "BadRequest",
            "statusMessage": "{\"code\":\"BadRequest\",\"message\":\"The document creation or update failed because there were duplicate properties in the json payload.\",\"target\":\"/subscriptions/7ae30bbe-e805-494a-803c-3a5d12c9ef2d/resourceGroups/issbdazbhbrgIngestion-123star/providers/Microsoft.DataFactory/factories/issbdazbhbdf1123star/pipelines/issbdazbhbpiASE123star\",\"details\":null,\"error\":null}",
            "serviceRequestId": null
    Which tells me nothing new.

    How can I find where this alleged duplicate is?



    • Edited by JohnNev Sunday, May 6, 2018 10:04 AM formatting
    Sunday, May 6, 2018 10:04 AM
  • Did you figure this out?  I am having the same issue
    Wednesday, May 30, 2018 8:11 PM