locked
Azure Data Factory Kafka Connector RRS feed

  • Question

  • Hi All, 

    is there a connector for Kafka or Event hub in Azure Data Factory ?

    Thanks

    Sri 


    Sri.Tummala

    Sunday, May 12, 2019 1:35 AM

Answers

  • looks like a half baked product compared with GCP (Data Fusion) I hope microsoft works on it and make below improvements.

    Spark SQL Task component --> similar to sql task in SQL Server --> Spark sql task lets developers write sql instead of writing up notebook etc.... also developers have a chance to fine tune bad data using sql for example cross products, data skew using distribute by few columns.

    there is no way right now developers will be able to fine tune the sql or data frame code what data flow creates in data factory.

    Code generation should be one option so developers should be able to fine tune the code generated by data flow component inside data factory.

    please look into GCP data fusion and several other spark GUI based ETL tools which are super mature, Microsoft should take these as examples moving forward and improving data factory.

    https://cdap.io/

    https://www.youtube.com/watch?v=kehG0CJw2wo&t=684s

    https://www.bitwiseglobal.com/innovations/hydrograph/

    https://github.com/BitwiseInc/Hydrograph

    https://www.trifacta.com/


    Sri.Tummala

    • Marked as answer by kali786 Tuesday, May 14, 2019 5:02 PM
    Tuesday, May 14, 2019 5:02 PM

All replies