Azure Resources in various data centers RRS feed

  • Question

  • Hi was wondering... we are creating an ETL pipeline using Data Factory and databricks to move data from azure data lake to azure data warehouse. What are the ramifications of having an original source, onprem in Oregon, databricks on the Est US 2 and then our data warehouse on West US 2. Wouldn't this cause the data to transverse the continent to complete the pipeline? Is it not more efficient to just have our databricks on the West US 2 also? 
    Wednesday, March 13, 2019 3:00 PM

All replies

  • Hello,

    Azure data factory can access data stores and compute services in other Azure regions to move data between data stores or process data using compute services.

    Azure Data Factory itself does not store any data. It lets you create data-driven workflows to orchestrate the movement of data between supported data stores and the processing of data using compute services in other regions or in an on-premises environment. It also allows you to monitor and manage workflows by using both programmatic and UI mechanisms.

    Although Data Factory is available only in certain regions, the service that powers the data movement in Data Factory is available globally in several regions. If a data store is behind a firewall, then a Self-hosted Integration Runtime that's installed in your on-premises environment moves the data instead.

    For an example, let's assume that your compute environments such as Azure HDInsight cluster and Azure Machine Learning are running out of the West Europe region. You can create and use an Azure Data Factory instance in East US or East US 2 and use it to schedule jobs on your compute environments in West Europe. It takes a few milliseconds for Data Factory to trigger the job on your compute environment, but the time for running the job on your computing environment does not change.

    Hope this helps.

    Thursday, March 14, 2019 10:24 AM
  • Hello,

    Just checking in to see if the above answer helped. If this answers your query, do click “Mark as Answer” and Up-Vote for the same. And, if you have any further query do let us know.

    Monday, March 18, 2019 5:40 AM