ADF Pipeline Hangs (ForEach Loop) RRS feed

  • Question

  • Hi have a pipeline that connects to on-premise data source using on-premise hosted Self Hosted Integration Runtime. The pipeline worked fine in Dec 2019 but it stopped working in Jan. When run, it just hangs and eventually timesout.  The pipleline uses ForEach look to query a set of views from on-premise DB and create files in ADLS Gen 2 with the file name same as the on-premise view it is reading. I stripped the pipeline down to just one view and removed the ForEach loop and it works just fine. So, the issue only appears to be with the usage of ForEach loop.

    Anythoughts or know bugs/issues with ForEach loop? Thanks.

    Friday, February 28, 2020 4:55 PM

All replies

  • Thanks IndyDB , 

    As you mentioned  you are using SHIR , Can you please check and see how is the SHIR doing ? 

    Did the data which you are pulling from in In premise service increased in Dec ? Are you using the sequential option in the foreach activity ?

    Thanks Himanshu

    Monday, March 2, 2020 6:32 PM
  • Thanks Himanshu. SHIR connectivity is just fine. No issues found there. The data has not changed, infact the data size is relatively much small, just in thousands of records. I had tried both sequential and the default parallel option by changing the degree.


    Tuesday, March 3, 2020 2:19 AM
  • Hello IndyDBA,

    Can you please check if you are the latest version of the SHIR , if not please try to upgrade to SHIR and see if that helps . I have a strong feeling that the something is not right on the SHIR side . Also please do share the ADF json .

    Please do let us know how it goes .

    Thanks Himanshu

    Tuesday, March 3, 2020 5:04 PM
  • Thanks Himanshu.  The SHIR is uptodate. It is setup to update automatically. The issue is with ForEach Lookup activity. So, when we isolated these above to, we are noticing that the pipeline hangs in the LookUp activity before it gets to ForEach.


    Tuesday, March 17, 2020 8:20 PM
  • Can you please share the JSON of the adf ? What is that you are doing in lookup ? Are you querying SQL / BLOB etc 

    Thanks Himanshu

    Tuesday, March 17, 2020 11:12 PM
  • The ADF pipeline basically connects to an on-premise Sql data source via a SHIR hosted on a DMZ on-premise. The pipeline queries a series of views and for each view queried, it creates a csv with the same name as the view name and write the csv file to Azure Data Lake Storage, Gen2 (blob storage). Tested the data retreival without ForEach Lookup activity and it works. Tested writing to ADLS blob storage directly with-in Azure, no problem there either. Once you add the look up, it hangs up. Thank you.

    Wednesday, March 18, 2020 2:41 PM
  • Himanshu,

                  Thanks for your review of this issue. I see that you work for Microsoft. I have a case currently open with support for about 8 weeks and you may get more specifics regarding this issue from the case (120011726003863). Thank you.

    Wednesday, March 18, 2020 4:40 PM
  • Thanks IndyDBA , 

    I see that somebody is already working with you on that SR . Please do paste if you find any resolution  or you need any help from us .

    Thanks Himanshu

    Thursday, March 19, 2020 6:16 PM