Searching Security Data in ADLS Gen2 RRS feed

  • Question

  • Presently our enterprise has some 2-3 year old log data compressed and archived in ADLS Gen2.   To search it, we literally pull the data files to a Linux VM and grep for the content.   This is primarily done for compliance reasons, as the data is not being actively used in daily operations.

    If we wanted to expand this solution to be used in our daily operations, we would need to be able to have teams of a couple dozen analysts work with the data quickly and very reliably.   Queries would need to be relatively easy and controlled carefully by individual RBAC.

    From what I can see of the Microsoft solutions:

    Azure Monitor - seems to be oriented as a data collection and search tool, but would duplicate our data storage for the first two years and not be able to retain data beyond that.

    Azure Data Explorer Cluster - seems to require us to choose a sizing and keep it "always on".   It's not clear if it would be able to search data in ADLS Gen2.

    Azure Log Analytics - I may have been mistaken looking down this path, it seems to analyze the security data *of* ADLS, but cannot analyze data *in* ADLS.

    Are there other solutions?   Is there a MS solution which would search arbitrary compressed data in ADLS Gen2 or a solution like Monitor which could handle arbitrary data or log data beyond 2 years?

    I am not very experienced with Azure, so it's very possible I'm missing something obvious.


    Thursday, January 2, 2020 9:48 PM

All replies

  • Hello mgjk2 , 

    Since you never mentioned the data volume , I am assuming that its massive :) . I believe that Azure Data Explorer Cluster  should be something which will help you . Its does support ADLS gen2 also (https://docs.microsoft.com/en-us/azure/data-explorer/data-lake-query-data) . But since you mentioned that the analyst will not be using all the data actively  , I am not sure if you should go this route as it will be expensive . 

    If your schema of the logs is not changing very often , do you think you can use Azure data factory and ingest  the data in a database ? By doing this the logs can be queried from an IDE like SSMS for free . 

    One other option is to use Azure DW for ingestion , the advantage is you can pause a SQL DW when you are not using it , which saves cost .

    Hope this helps .

    Thanks Himanshu

    Friday, January 3, 2020 7:44 PM
  • Hello , 

    We have not received a response from you.   If you found a solution, would you please share it here with the community

    Thanks Himanshu

    Wednesday, January 15, 2020 8:41 PM
  • Hello ,

    Since we have still not heard back from you, we will assume you found your own resolution.  If you found a solution, would you please share it here with the community?

    Thanks Himanshu

    Thursday, January 23, 2020 8:36 PM