Azure Synapse Analytics Questions - Data Factory

Question

I have lots of questions regarding Azure Synapse Analytics (There doesn't appear to be a Forum for it yet)

We have all our data Factories is a specific service. Should be be moving all our data factory information into Azure Synapse Analytics if we go for a data Warehouse Option?

If so, how easy is this to do?

Does it cost the same?

Is it OK to do all the ETL Orchestration within a Data Factory Service or must it be added into  Azure Synapse Analytics?

---

Debbie

Answer 1

Hi Debbie

Here is the consolidated list of questions that you have for Azure Synapse.

1. As mentioned above We have all our data Factories is a specific service. Should be moving all our data factory information into Azure Synapse Analytics if we go for a data Warehouse Option? If so, how easy is this to do? Does it cost the same? And Is it OK to do all the ETL Orchestration within a Data Factory Service or must it be added into Azure Synapse Analytics?
   
2. Is it ok to simply use this as the data warehouse and have all the other services as separate services? Or are you paying for everything within Azure Synapse Analytics (Data Factory, Data Lake etc. This would mean you could treat this in the same way as you would normally do if you had a data Warehouse set up.
   
3. We tend to get started with a data base when we are just doing development and proof of concept testing. Then we would decide to move into a data warehouse. With Azure Synapse Analytics, how easy is it to move your database to a data warehouse?
   
4. If you use a data Lake outside of Azure Synapse Analytics service, can you still add it and look at the data against your data warehouse using Polybase? Or can you only do this with a data lake inside Another Azure Synapse Analytics?
   
5. You can now do all your ETL Processing etc. within the one service. What’s the best practice? If you use a Warehouse does that mean everything should be within the one service? What happens if you use Logic Apps with Data Factory? Logic Apps doesn't seem to be included. I’m not sure if it’s still fine to use Azure Synapse Analytics as data warehouse with external services or you should always have everything in the one solution? Which is the cheapest option?
   
6. A final Question, If you set up all your Data bricks, note books, Data science services up within  Azure Synapse Analytics, is this as cost effective as setting them up as external services?
   

We will try to help you with all these questions. For more information, please check my next reply.

Thanks
Navtej S

Answer 2

Hi Debbie

Thanks for your interest in Azure Synapse Analytics. Before I get responses from our Product team for your questions. You can get started immediately with the features in General Availability by clicking this link.  

Also there are a plethora of documents, tutorials and additional resources available here. I recommend:

What is Azure Synapse Analytics?

Azure Synapse Architecture

Cheat sheet for Azure Synapse Analytics

Building a Data Warehouse

Connect to Power Bi

Mechanics deep-dive video

Also you can register for the free virtual event to get more information about the Azure Synapse.

December 10, 2019 / 10:00–11:00 am PST

 Join us for this free virtual event to learn how to simplify your data lake and data warehousing solutions with Azure Synapse Analytics. Query data on your terms using on-demand or provisioned resources. Experience a limitless analytics service built to ingest, prep, manage, and serve data, accelerating your time to insight.​

Why attend?​

Come find out how Azure Synapse Analytics will help you:​

• Query petabyte-scale data on demand from the data lake, or provision elastic compute resources for demanding workloads.​
• Build a modern data warehouse enhanced with streaming analytics, machine learning, BI, and AI capabilities.​
• Reduce project development time for machine learning, BI, and AI.​
• Easily optimize petabyte-scale workloads and automatically prioritize critical jobs.​
• Help safeguard your data with Azure Active Directory integration, dynamic data masking, column-level and row-level security, and automated threat detection.​

The presentation also includes a hands-on demo of Azure Synapse Analytics and a live Q&A. 

Where can I register?​

https://aka.ms/analytics-virtual-event-2019​

​

Where can I view the event?​

Click the “join the virtual event” link in your registration confirmation email.​

 

When is the Analytics in Azure virtual event?​

Tuesday, December 10 from​

10:00–11:00 AM Pacific Time.​

 

Who should attend?​

All interested customers and partners who want a deeper look at Azure Synapse Analytics and what the launch really means for them.

Regards
Navtej S

Answer 3

thanks,

Im already registered for that even. although I'm in the UK so not sure how timings will go for me.

Ive been reading the above documentation, thanks for that. I look forward to getting a reply back on my specific questions. I cant get them answered with the documentation provided unfortunately. But they have answered a lot of other questions and given more structure for me

---

Debbie

Answer 4

Hi Any response to these questions yet

---

Debbie

Answer 5

Hi Debbie

Here is the consolidated list of questions that you have for Azure Synapse.

• As mentioned above We have all our data Factories is a specific service. Should be moving all our data factory information into Azure Synapse Analytics if we go for a data Warehouse Option? If so, how easy is this to do? Does it cost the same? And Is it OK to do all the ETL Orchestration within a Data Factory Service or must it be added into Azure Synapse Analytics?
  
• Is it ok to simply use this as the data warehouse and have all the other services as separate services? Or are you paying for everything within Azure Synapse Analytics (Data Factory, Data Lake etc. This would mean you could treat this in the same way as you would normally do if you had a data Warehouse set up.
  
• We tend to get started with a data base when we are just doing development and proof of concept testing. Then we would decide to move into a data warehouse. With Azure Synapse Analytics, how easy is it to move your database to a data warehouse?
  
• If you use a data Lake outside of Azure Synapse Analytics service, can you still add it and look at the data against your data warehouse using Polybase? Or can you only do this with a data lake inside Another Azure Synapse Analytics?
  
• You can now do all your ETL Processing etc. within the one service. What’s the best practice? If you use a Warehouse does that mean everything should be within the one service? What happens if you use Logic Apps with Data Factory? Logic Apps doesn't seem to be included. I’m not sure if it’s still fine to use Azure Synapse Analytics as data warehouse with external services or you should always have everything in the one solution? Which is the cheapest option?
  
• A final Question, If you set up all your Data bricks, note books, Data science services up within  Azure Synapse Analytics, is this as cost effective as setting them up as external services?
  

We will try to help you with all these questions. For more information, please check my next reply.

Thanks
Navtej S

Hi Debbie 

Here are the answers.

As mentioned above We have all our data Factories is a specific service. Should be moving all our data factory information into Azure Synapse Analytics if we go for a data Warehouse Option? If so, how easy is this to do? Does it cost the same? And Is it OK to do all the ETL Orchestration within a Data Factory Service or must it be added into Azure Synapse Analytics.

We will provide the capability to export all your work ADF to Synapse. If you need a DW option now, it will be also associated to the Synapse workspace by the time we GA. In a nutshell, all the work you do on ADF or DW will be valid in the synapse workspace.
Is it okay to simply use this as the data warehouse and have all the other services as separate services? Or are you paying for everything within Azure Synapse Analytics (Data Factory, Data Lake etc. This would mean you could treat this in the same way as you would normally do if you had a data Warehouse set up.

You are paying for the capabilities within Azure Synapse Analytics but the business model of each capability that exist outside the workspace will be the same. There should be no price difference.

We tend to get started with a data base when we are just doing development and proof of concept testing. Then we would decide to move into a data warehouse. With Azure Synapse Analytics, how easy is it to move your database to a data warehouse.

For a Data Warehousing concept, you should be using what was called SQL DW. This is also why we renamed the capability SQL DW as SQL Analytics Pool (and On-Demand). Our message if you want to use SQL to run analytics, use Synapse. A blog post like that  should not be valid anymore.
If you use a data Lake outside of Azure Synapse Analytics service, can you still add it and look at the data against your data warehouse using Poly base? Or can you only do this with a data lake inside Another Azure Synapse Analytics.

The data lake is not part of Azure Synapse workspace. Polybase still works to access and copy data. A data lake is just associated to a workspace. In the case of the workspace, you can also use SQL On-Demand if you want to query and analyze the data directly.
You can now do all your ETL Processing etc. within the one service. What’s the best practice? If you use a Warehouse does that mean everything should be within the one service? What happens if you use Logic Apps with Data Factory? Logic Apps doesn't seem to be included. I’m not sure if it’s still fine to use Azure Synapse Analytics as data warehouse with external services or you should always have everything in the one solution? Which is the cheapest option?
For Data analytics operations and orchestration, ADF is key in Synapse. I would use it. I cannot comment on the price. It really depends on your workloads. 
A final Question, If you set up all your Data bricks, note books, Data science services up within  Azure Synapse Analytics, is this as cost effective as setting them up as external services
Azure Databricks is not included in Azure Synapse. There are integrations that we build with Databricks though.  Today, you can share the same data lake between synapse and ADB, Synapse also has support for Delta Lake. As a rule of thumb my comments on pricing hold.

Hope this helps.

Thanks
Navtej S

Answer 6

Hi Debbie

Please let us know if you have any further question in this regard. Last reply has been proposed as an answer. If it has helped you, please mark it as an answer.

Thanks
Navtej S

Answer 7

thank you. thats great!

---

Debbie