none
azure data lake architecture RRS feed

  • Question

  • Our company have on-premises DW, data source is just LOB, max data size could be 100GB. We jusr run reports using SSRS and SSAS. Currently on very old stack of SQL 2008, so getting performance problems.

    Some how thought process going to move in data lake. Idea is move structural data in to files then move it Azure data store using Azure Data Factory. Then process that data and make ready for reports again move that in SQL storage or On-premises  using Azure Data Factory and then run reports on it using Power BI .

    Some how I personally do not like this design and thinking it is over engineering, and I think if data is relational and just need to run reports then why not just move 2016 and redesign.

    As many of you are expert you can advice?

    If we move in ADL with above option with SQL storage how much will be cost?


    • Edited by dnyanbhar Thursday, February 2, 2017 10:30 PM
    Thursday, February 2, 2017 6:58 PM

Answers

  • >max data size could be 100GB

    If is not a "big data" problem, you wont save any time or money using "big data" tools. 

    At this scale SQL Server 2016 is going to be much simpler, faster to implement, and give you faster ETL and query response time than using Data Factory and Azure Data Lake.

    SQL Server : Azure Data Lake :: A Ferrari : A Semi Truck

    David


    Microsoft Technology Center - Dallas

    My Blog


    Thursday, February 2, 2017 10:35 PM
  • >is Data Lake solution will cost effective for such solution in long term or on-premise SQL 2016 is cost effective?

    At that scale, neither solution is going be terribly expensive.  The costs of design, development, maintenance, and likelihood of failure are the bigger concerns.

    In data warehousing and analytics, technology is no longer the hard part.  Focus on engaging with business stakeholders, scoping delivery to show incremental business value, and on helping the business through the transitions. 

    David


    Microsoft Technology Center - Dallas

    My Blog


    Friday, February 3, 2017 8:06 PM

All replies

  • >max data size could be 100GB

    If is not a "big data" problem, you wont save any time or money using "big data" tools. 

    At this scale SQL Server 2016 is going to be much simpler, faster to implement, and give you faster ETL and query response time than using Data Factory and Azure Data Lake.

    SQL Server : Azure Data Lake :: A Ferrari : A Semi Truck

    David


    Microsoft Technology Center - Dallas

    My Blog


    Thursday, February 2, 2017 10:35 PM
  • I totally agree with you, I try to explain same thing to team but really tough time. 

    Finally thinking let them face it. BTW is Data Lake solution will cost effective for such solution in long term or on-premise SQL 2016 is cost effective?

    Friday, February 3, 2017 3:55 PM
  • >is Data Lake solution will cost effective for such solution in long term or on-premise SQL 2016 is cost effective?

    At that scale, neither solution is going be terribly expensive.  The costs of design, development, maintenance, and likelihood of failure are the bigger concerns.

    In data warehousing and analytics, technology is no longer the hard part.  Focus on engaging with business stakeholders, scoping delivery to show incremental business value, and on helping the business through the transitions. 

    David


    Microsoft Technology Center - Dallas

    My Blog


    Friday, February 3, 2017 8:06 PM