locked
looking for how a datalake sctivity can kickoff a adf pipeline RRS feed

  • Question

  • I've been reading all weekend about ADF but what I don't see is the details on how to set up a trigger when a transfer to a datalake file has been completed so I can fire off an ADf pipeline.

    Can someone outline the details to set this kind of activity up or point me to a more relevant documentation?Does this type of process use log analytics notifications to cause such a trigger event for ADF?


    Contract Developer Frog Haven Enterprises

    Monday, March 4, 2019 4:59 PM

Answers

  • Hi elbilo,

    You are correct, event-based triggers supports only on version 2 Storage accounts (General purpose).

    Azure Event Grid doesn't receive events from Azure Data Lake Gen2 accounts because those accounts don't yet generate them.

    For more details, refer “Known issues with Azure Data Lake Storage Gen2”.

    Hope this helps.

    • Marked as answer by elbilo Friday, March 8, 2019 9:58 PM
    Thursday, March 7, 2019 8:28 AM

All replies

  • Hi elbilo,

    Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Data Factory customers to trigger pipelines based on events. Data Factory is now integrated with Azure Event Grid, which lets you trigger pipelines on an event.

    This article describes the event-based triggers that you can create in your Data Factory pipelines.

    Kindly go through a ten-minute introduction and demonstration of this feature, watch the following video: Event-based data integration with Azure Data Factory.

    Hope this helps.
    Tuesday, March 5, 2019 8:50 AM
  • I saw this article but it seems the events are tied only to blob storage. Somewhere else I saw that data lake storage was based on blob storage. Does this mean that the blob event also works with data lake store elements as well or is it limited to only blob storage?


    Contract Developer Frog Haven Enterprises

    • Marked as answer by elbilo Friday, March 8, 2019 9:57 PM
    • Unmarked as answer by elbilo Friday, March 8, 2019 9:58 PM
    Tuesday, March 5, 2019 3:20 PM
  • Hi elbilo,

    You are correct, event-based triggers supports only on version 2 Storage accounts (General purpose).

    Azure Event Grid doesn't receive events from Azure Data Lake Gen2 accounts because those accounts don't yet generate them.

    For more details, refer “Known issues with Azure Data Lake Storage Gen2”.

    Hope this helps.

    • Marked as answer by elbilo Friday, March 8, 2019 9:58 PM
    Thursday, March 7, 2019 8:28 AM