none
Alerts for lake store

    Question

  • I want to generate alerts for lake store every time Read or write happens in the lake store. I tried using Azure metrics but it is not supported for Azure lake store. Can anyone provide me a viable solution to achieve so. Can this be acheived using powershell or any other way?
    Wednesday, December 6, 2017 11:22 AM

All replies

  • I don't believe that there is a simple solution to have an alert for each and every read/ write. The amount of alerts that would be generated in most cases would be huge and not overly useful information to have. 

    Generally the desire is to have a threshold for an alert set such as you have described and see via the metrics tab. However I understand that the metrics option is not available for Lake Store.

    For what purpose are you looking to get an alert for every Read/ Write operation? Maybe I can suggest an alternative solution to what you are trying to acvhive

    Wednesday, December 6, 2017 8:51 PM
  • Everytime a file is opened, or a file is deleted or file is added in my data lake store account , alerts should be generated. This is my requirement. I believe one possible solution is using powershell but i am not sure.
    Thursday, December 7, 2017 10:07 AM
  • I have reached out to some of our Azure Data Lake experts and they have confirmed that there is no way to achieve this out of the box. 

    A suggestion that was given was you could have the audit logs from your data lake written to a storage account then create a script or app that monitors those logs and from there generate alerts for specific events. 

    So it would take some additional components but in theory what you are looking to achieve is possible. 

    Friday, December 8, 2017 7:22 PM
  • Any information whether Microsoft will provide these features for Data lake store in future updates?
    • Edited by Sachin1133 Monday, December 11, 2017 7:31 AM
    Monday, December 11, 2017 7:31 AM
  • No plans to as of now. If you like you can submit this feature request here: 

    https://feedback.azure.com/forums/34192--general-feedback

    That way if others find use for this feature as well they can up vote it and the DataLake team can consider adding it. 

    Monday, December 11, 2017 5:08 PM
  • They why is there a grey out "Add Metric Alert" button when you open Data Lake Store Metrics under Monitoring?

    SERIOUSLY poorly thought out UI if there is not going to be support for metric alerts.
    Come on Microsoft... you're embarrassing yourselves.

    Graham

    Monday, December 18, 2017 5:28 PM
  • You can set alerts for metrics. If you read the above thread you can see this issue is in regards to alerts for a single read/ write/ delete operation not genetic metrics. If you are having issues setting up alerts for your ADL please create a new thread and we can take a look. Thanks, Micah
    Monday, December 18, 2017 5:57 PM
  • Hello Sachin,

    Thanks for reaching out. We are currently working to improve this experience, but in the meantime, one approach you can take is to push the ADLS Diagnostics Logs to an Event Hub, and from Azure Monitor, create an alert for the events registered.

    You can do it the following way:

    1. On your ADLS, go to the Diagnostic Logs and select the option to Stream to an Event Hub (should be in the same region as the ADLS). You can select which type of request to send to the Event Hub. 

    2. On the Event Hub, you can see the incoming notifications based on the requests you chose to send - each creates a type of insight-log.

    3. From Azure Monitor, select Alerts, select the subscription, resource group, and resource name (Event Hub you created earlier), and add a new metric alert.

    4. In the metric alert you can select the specific metric that you're looking for depending on the action from the log, and configure the action (email, webhook, etc). 

    5. Once the diagnostic log that matches your request is detected, the Azure Monitor alert will alert you of the activity. 

    As I mentioned above, we're still improving the experience and hope that this helps you for the time being. Feel free to reach out if you need more details or help. 

    Thanks!


    Tuesday, December 19, 2017 3:33 AM