locked
Processing data without transferring data to the cloud RRS feed

  • Question

  • Hello all,

    The company I work for processes medical data and therefore prohibited from using cloud technologies; data must always stay on premise. This does hurt our development process, especially for a company which is very much 'big data'. We have very large data sets which are run through monthly 'ETL' processes that take a couple of weeks to run. To improve our process we are looking at many technologies; one being Azure Data Factory. Data Factory does sound exactly what we need and would give us the excuse to redesign parts of our process to get the most efficient process.

    Our problem is that we can not allow data to flow through the 'cloud'. I know that we can use on premise data sources and on premise data destinations but what is unclear is how the process and transforms take place.

    Could someone give some insight into an all on premise data factory? Can we restrict data from ever making its way into the cloud while still taking advantage of Azure? I have a feeling the answer will be no; me and my team are just another group of developers looking amazing Azure solutions wishing we weren't held back by data privacy and UK laws. We would love to take advantage of the cloud as much as we can!


    • Edited by Edward1421 Wednesday, May 27, 2015 3:18 PM
    Wednesday, May 27, 2015 3:01 PM

All replies

  • Hey Edward,

    While I am reading your words, my interpretation is you would like to use Azure Data Factory as orchestration from the Cloud, to kick off copy between two on-prem data sources, and the data is flowed only at on-prem? Is that correct?

    If true, can you share with us what on-prem data sources here you want?

    Today, for example, if copy is between two on-prem SQL Server, the data flow is only at on-prem, not go through cloud.

    Regards,

    -oliver


    Oliver Yao - MSFT

    Thursday, May 28, 2015 9:32 PM