none
Azure Data Lake Storage - Create SQL like views RRS feed

  • Question

  • Hi, 

    I am working on a solution design for a PoC/spike around Azure Data Lake Storage (ADLS), where client's existing data from relational and file based systems will be moved to ADLS. 

    No issues with this as ADF fits well here.

    What I m not sure of is the requirements to create SQL like VIEWs which will hold come business logic. The plan is to use these for ad-hoc querying, if possible, in the downstream front end user apps.

    Since, there is no specific choice guidelines between Gen 1 or Gen 2 from end client, I am not sure which would be the one to go with. Especially from a data access point of view; 

    ADLS Gen1 supports:

    • Data Lake Analytics & U-SQL
    • SQL DW Polybase
    • Event Hub
    • Stream Analytics
    • Power BI
    • ADF, etc

    ADLS Gen 2 supports:

    • Databricks
    • Power BI
    • SQL DW Polybase
    • ADF

    I have used other Azure Data services but this is my first time using ADLS. I would really appreciate some help and insights in making this choice. 


    Cheers!! Sumit Bhatnagar

    Friday, November 8, 2019 12:53 AM

All replies

  • Hi Sumit,

    Hope you know that there should be a extra layer required to store and expose your data in a relational format if end users are looking for any as such.

    However, with the current shared information - You can store the Processed Data into Azure Data Lake again with a kind of refined layer.

    I would recommend using ADLS Gen2 rather than Gen1 as it comes with more features (Like Granular Permissions -ACL's, Blob API's etc.,) along with Supportive Analytical tools (Except ADLA/U-SQL) which would not be a great idea to go with as things are getting matured in Azure and currently no-new implementation are considering U-SQL as Databricks is one of the Primary approach being Cloud agnostic.

    Hope this is helpful !!

    Thank you 


    If this post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 8, 2019 1:39 PM
  • Hi Sumit.Bhatnagar,

    Just checking to see if the above suggestion from nkumar230 was helpful. If this answers your query, please consider to click “Mark as Answer” and "Up-Vote" as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Monday, November 11, 2019 11:44 PM
    Moderator
  • Hi Sumit.Bhatnagar,

    Following up to see if the above suggestion from nkumar230 was helpful. If you have any further query let us know.


    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Tuesday, November 12, 2019 3:50 PM
    Moderator
  • KranthiPakla - could you please clarify your statement here:

    "I would recommend using ADLS Gen2 rather than Gen1 as it comes with more features (Like Granular Permissions -ACL's, Blob API's etc.,) along with Supportive Analytical tools (Except ADLA/U-SQL) which would not be a great idea to go with as things are getting matured in Azure and currently no-new implementation are considering U-SQL as Databricks is one of the Primary approach being Cloud agnostic."

    We are using ADLA for existing processes and are creating additional new ADLA procedures and scripts, as well.

    I've understood the DataBricks popularity of recent, but what is the clear direction of ADLA for the development community which is using ADLA today?

    Thanks!


    Bill Blakey

    Thursday, November 14, 2019 1:25 PM
  • Hi Bill Blakey FA,

    Apologies for the late reply. I have reached to the internal team to get more info about your query, will keep posted once I have update from the team. 

    Thank you

    If a post helps to resolve your issue, please click the "Mark as Answer" of that post and/or click Answered "Vote as helpful" button of that post. By marking a post as Answered and/or Helpful, you help others find the answer faster.

    Friday, November 22, 2019 1:30 AM
    Moderator