locked
How to make Lookup activity handle >5000 rows? RRS feed

  • Question

  • Hi Techie!

    How to make Lookup activity handle >5000 rows? Is there any other alternatives available. 

    My source is Blob .csv file, which will have around 1lac record, i want to do Lookup on that.

    Thanks

    Thursday, October 31, 2019 8:08 AM

All replies

  • What do you want to do with the data that you have from the lookup?
    Thursday, October 31, 2019 8:17 AM
  • You are mistaking lookup "transformation" in dataflow to lookup activity. The activity does not do any sort of data lookup
    Thursday, October 31, 2019 2:43 PM
  • Hi SanthoshKumar09,

    Could you please elaborate more on your business requirement as requested by mkro?



    Note: Please check here for Lookup activity limitations and workaround.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 1, 2019 6:36 PM
  • Hi SanthoshKumar09,

    Following up to see if you still need assistance on this query. If so, could you please share the details requested in above comment? 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, November 4, 2019 6:42 PM
  • Sorry for the delay!

    My source is DataWarehouse having 1lac + records, i have to load this data into D365 CRM. Before loading the data into CRM, if the record already exists in CRM, then, i need to get it's corresponding Guid, OptionSet id etc., for upsert.

    To get the Guid & Option Set id's, i'm using Lookup & ForEach activities which will fetch the data from CRM and then i'm using Copy Data to insert into CRM.

    Now, the problem is Lookup will support only 5,000 or 2MB records. In my scenario the records are >1lacs. How to make Lookup activity to handle >5,000 records?


    Santhosh Kumar K

    Wednesday, November 6, 2019 3:08 AM
  • Hi Santosh,

    Thanks for sharing the additional details. I am reaching out to internal team to know about the possible workarounds about your query. Will keep posted once I have an update from the team.

    Also one thing I would like to confirm: In the initial post you have mentioned the source as Blob.csv ("My source is Blob .csv file, which will have around 1lac record, i want to do Lookup on that.") and in the most recent reply you have mentioned DataWarehouse ("My source is DataWarehouse having 1lac + records,")- could you please confirm what exactly is your source? 


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered"Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.



    Thursday, November 7, 2019 7:38 PM
  • Hi Kranthi,

    I have multiple pipelines, in one pipeline the source is blob and in other it's DataWarehouse. In both the cases i will have 1lac+ records. Awaiting your reply on this, as i'm waiting on this for a long time now.

    Thanks


    Santhosh Kumar K

    Friday, November 8, 2019 3:11 AM
  • Hi Santhosh,

    Thanks for sharing the details. As your business requirement is related to product limitation and for immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you please send an email to AzCommunity@Microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. 

    Subject of the email: ADF: How to make Lookup activity handle >5000 rows?
    Thread URL: https://social.msdn.microsoft.com/Forums/en-US/c7fec374-2242-4264-8744-354a5489f662/how-to-make-lookup-activity-handle-gt5000-rows?forum=AzureDataFactory
    Subscription ID: <Your subscription ID>

    Please let me know once you have done the same


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 8, 2019 4:29 AM