none
Alerts for Data Lake Store

    Question

  • I want to get a message when a folder in ADLS is either created or deleted. 

    We ingest data from all kinds of external and internal sources, and store these raw data in separate folders. I want to monitor whenever a new data source is added or deleted, so that I can update the Data Catalog accordingly.

    Wednesday, February 7, 2018 8:52 AM

All replies

  • I'd be happy to help.  First have you looked at the code samples for ADLS?  What language are you using? 

    https://github.com/Azure-Samples/data-lake-store-adls-dot-net-samples 

    https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-operations-net-sdk

    Guy

    Thursday, February 8, 2018 5:42 PM
  • Thanks for your answer, Guy.

    Yes, I have looked into the code samples, but I cannot find what I'm looking for.

    I'm not an expert on coding, but I've used C# (Visual Studio) to perform some testing. With a lot of help from YouTube on publishing to Azure from VS. 

    I was hoping there was a function in Azure I could use to avoid coding, or an example for how to do it. I see there are functions for creating folders in ADLS, and I've tested that, and it worked fine. Maybe there is something similar for checking if "new folder created" or "folder deleted" in ADLS for "the last 24 hours", and if so, "send message to..".

    Soffi

    Friday, February 9, 2018 10:42 AM
  • Hi Soffi,

    You can use Azure Log Analytics to track events from ADLS; this way you should not need to develop any custom code. Here are the steps:

    1. Connect your ADLS account to Log Analytics -- you can follow the steps I present in this blog post: Struggling to get insights for your Azure Data Lake Store? Azure Log Analytics can help!
    2. In Log Analytics, go to the Alerts (Preview) section and click on New Alert Rule
    3. Choose the alert target (typically the Log Analytics accountt that is tracking the logs)
    4. Create the alert criteria:

      - Click on Add Criteria

      - Select Custom Log Search:


      - On the Search Query, paste the following query:
      search *
      | where OperationName == "mkdirs"
      - Set the Alert Logic - the threshold should be greater than 0 (so that each time a folder is created, an alert is sent. The period and frequency can align to any relevant window of time you want to monitor - depending on how often you want the system to check, and how often to be notified. 

      - Define the details for the alert (and enable it on creation):


      - Define the action group details -- in this case, let's send an e-mail (you can add specific accounts/other recipients in New Action Group):


      - Click the create button!
    5. Once the rule is provisioned, it should appear in the list:

    This should deliver an email message (or a fire an event to a webhook) every time the query evaluated is returns the event log for the folder creation. You can check the logs emitted by the ADLS account and create alerts for them. 

    We're working on better ways to get notified on events like these, but these steps can help you solve your current need, Soffi.



    Friday, February 16, 2018 12:48 AM
  • Thank you José!

    I'm now getting an email every time a new folder is created. Thank you for making the guideline so detailed and easy to follow!

    Soffi :-)


    Monday, February 19, 2018 10:56 AM
  • Hi,

    Can we do this for changes to existing folders also? My scenario is to get some kind of alert to trigger a function app whenever a new file is added to any of the existing folders. (Basically change monitoring).

    Thursday, March 29, 2018 9:46 PM