none
How to find records missing from import? -I used Azure Cosmos DB Data migration tool RRS feed

  • Question

  • I just imported 10 million records into Azure Cosmos DB but 78 didn't end up getting imported.

    I used Azure Cosmos DB Data migration tool and the error csv file has 160  "Message: Channel is closed" messages.

    I did a count of the items and that's how it came out that 78 records didn't get imported because the total is 78 shy of ten million.

    The data set is sequentially numbered. I converted the data to JSON (not compressed) before uploading it to my container. 

    I provisioned my container to 10,000 Ru/S before doing the import and from my home computer it took ~20 hours to upload the records.

    Wednesday, September 4, 2019 6:04 PM

All replies

  • Hi

    Channel is closed message come is the Connection between client and server is interrupted. This can happen in-cases either party terminates connection (directly or indirectly due to crash/shutdown) or connectivity between them is interrupted. These failures are typically transient and are/can be retried. So, the Migration tool must have picked up from where it lost connection and restarted the migration.

    Additionally, here are some performance tips that are useful for such events in Cosmos Db in general –

    https://docs.microsoft.com/en-us/azure/cosmos-db/performance-tips

    You can use the Change Feed in Azure Cosmos DB to check the state of the documents you migrated. The error that 78 items didn’t get imported must have been logged in Change feed with the reason. Please check the following link –

    https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed

    Hope this helps and you can resolve your issue. Please get back to us for any questions.

    Thanks
    Navtej S


    Thursday, September 5, 2019 10:54 PM
    Moderator
  • Hi 

    We have proposed the last reply as the answer. Please mark it as an answer if it helped you or get back to us if you have any other question.

    Thanks
    Navtej S

    Monday, September 9, 2019 3:07 PM
    Moderator
  • I appreciate the links and I've tried to go into working with the change feed with azure functions but it's slow going. 
    Tuesday, September 10, 2019 12:29 AM