none
Copy Activity has performance issues above a certain volume and also writes records multiple times into Sink RRS feed

  • Question

  • Using Copy Activity from an on-premise Basis database (SQL) via ODBC to Azure SQL Database. Some tables  work fine with 100,000+ records. Two of the 11 tables have issues. You can load up to approximately 50,000 records cleanly and in less than a minute. Once you go above this volume, runs for hours and actually starts writing redundant records into Azure SQL Database (so a single record has now been written 8 times as opposed to once into the Sink). Not sure how to debug from here.
    Tuesday, November 27, 2018 4:51 PM

All replies

  • Hi TexasJAPatterson,

    I think a good place to start looking would be the "writeBatchSize" parameter.

    For more information, you can check the documentation :

    https://docs.microsoft.com/en-us/azure/data-factory/connector-odbc

    There's a note in the documentation that says :

    For "writeBatchSize", if it's not set (auto-detected), copy activity first detects whether the driver supports batch operations, and set it to 10000 if it does, or set it to 1 if it doesn’t. If you explicitly set the value other than 0, copy activity honors the value and fails at runtime if the driver doesn’t support batch operations.

    Please let us know if this helps or else we can gladly continue the dialogue.
    Wednesday, November 28, 2018 10:37 AM
    Owner