none
Do we still need Hadoop , HDinsight and Spark now that we have Azure Data lake Gen2 and Azure stream analytics with the modern Azure datawarehouse architecture? RRS feed

  • Question

  • Do we still need Hadoop , HDinsight and Databricks Spark now that we have Azure Data lake Gen2 and Azure stream analytics with the modern Azure datawarehouse architecture?




    • Edited by MLTC Friday, November 8, 2019 6:12 PM
    Friday, November 8, 2019 6:12 PM

All replies

  • I will answer to that as "yes" we do . Lets revisit the terms which you mentioned above again let me add a bit more details to that . 

    Azure Data lake Gen2 : This is storage which is more designed to help the big data technology to consume it . Previously we used to write the blob storage and now this , but its just a storage . 

    Stream analytics : This is the tool to run analysis on the incoming data , if you ask get all the signals which is coming from IOT device in the last 1 hour , we have the data queries whcih basically work on some datatime field and gives you the output .  

    Hadoop , HDinsight and Databricks : There are computation engines and have the map -reduce implementation inside them ,  if you ask question "Get me the ratio of car which runs on gas vs diesel  in all the countries in the world " , then I think these technology come in to play and process the data in parallel and supplies the output . 
    These technologies had better security implementation also .

    HTH .



    Thanks Himanshu

    Friday, November 8, 2019 9:33 PM
    Moderator