locked
Out of memory error in ADF copy RRS feed

  • Question

  • Hi,

    I am copying 11 GB table from On-prem SQL Server to Azure SQL through ADF, and job is failing after around 35 minutes with below error message.

    Copy activity met an internal service error. For more information, provide this message to customer support. ErrorCode: 8000 ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=mscorlib,'.4

    I'am using Staged Copy approach, and in the job I can see 11 GB read is completed...however above error shows up somewhere when 4 GB is written to stage.

    Any thoughts what could be wrong and resolution?


    Monday, July 25, 2016 4:52 PM

Answers

  • Hi Pradip,

    You could set a smaller writeBatchSize property to mitigate the OOM issue.

    If you still met such issue. Could you provide the runId of your failed copy? It would help us to find the root cause.

    • Marked as answer by Pradip Thoke Friday, July 29, 2016 4:33 AM
    Thursday, July 28, 2016 4:05 AM

All replies

  • Hi Pradip,

    You could set a smaller writeBatchSize property to mitigate the OOM issue.

    If you still met such issue. Could you provide the runId of your failed copy? It would help us to find the root cause.

    • Marked as answer by Pradip Thoke Friday, July 29, 2016 4:33 AM
    Thursday, July 28, 2016 4:05 AM
  • Thanks for reply.

    Yes it was exact same issue, but the problem was direct from on prem to azure sql having stage copy in single pipeline it was not picking up writeBatchSize property.

    So what I did was splitted pipeline into two 1) On Prem -> Azure blob 2) Azure blob -> Azure SQL and this time in second pipeline given writeBatchSize to 50 and it worked though it took little longer but it was due to low memory on azure sql.

    Friday, July 29, 2016 4:33 AM