Azure Data Factory announcement
-
Link
We have noticed customers post their questions on Azure Databricks on incorrect forums on MSDN, since there is a new home for Azure Databricks.
Forum Link: https://social.msdn.microsoft.com/Forums/en-US/home?forum=AzureDatabricks
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
-
0 VotesEvent Trigger is not working
Hi, The event trigger is not working in Data Factory. We have created data factory event trigger on the creation of blob. The trigger works in the lower environment but it is not ...Unanswered | 4 Replies | 210 Views | Created by Ajinkya2210 - Wednesday, December 11, 2019 9:40 AM | Last reply by MartinJaffer-MSFT - Friday, December 13, 2019 1:30 AM
-
0 Votes
Fault Tolerance not working
Hi, The Fault Tolerance option in the Copy Data component is not working. We had our pipelines running fine till 8-Dec-2019 but since that it is failing. We have selected the Skip ...Proposed | 3 Replies | 250 Views | Created by Ajinkya2210 - Tuesday, December 10, 2019 11:58 AM | Last reply by HimanshuSinha-msft - Friday, December 13, 2019 12:26 AM -
0 Votes
ADF Expression Split?
Hi again I am stumbling on an expression syntax here which is critical to our App The original source file can ...Proposed | 2 Replies | 218 Views | Created by EMKISER - Friday, December 6, 2019 4:58 PM | Last reply by HimanshuSinha-msft - Thursday, December 12, 2019 8:08 PM -
2 Votes
Event Trigger - Error when getting container list
Hi I have a working Event Trigger against our test blob storage (regular blob storage v2 for our test environment), but when I try to create a new trigger against out Production ...Answered | 13 Replies | 419 Views | Created by Stranded Software - Thursday, December 5, 2019 11:18 AM | Last reply by Cesar Berard - Thursday, December 12, 2019 7:27 PM -
0 Votes
Data type mapping for SAP BW - how to handle Null
We are bringing SAP Bex query data into ADLS. There is a default data type mapping as per document BW data type mapping. However, we occasionally get the ...Proposed | 3 Replies | 162 Views | Created by SAP to ADLS - Tuesday, December 10, 2019 6:10 PM | Last reply by SAP to ADLS - Thursday, December 12, 2019 5:48 PM -
0 Votes
How to acess the trigger property(frequency,interval) to pass dynamically to adf pipeline
"type": "ScheduleTrigger", "typeProperties": { "recurrence": { "frequency": ...Proposed | 2 Replies | 149 Views | Created by SubhadipRoy - Thursday, December 12, 2019 12:04 AM | Last reply by SubhadipRoy - Thursday, December 12, 2019 2:31 PM -
0 Votes
Parameter to dynamically retrieve 'Activity Name' of a pipeline
Hi Everyone, I am trying to store details of Pipeline Name,Pipeline Id and Activity Name in SQL Table by executing SP in pipeline which is followed by an ...Unanswered | 1 Replies | 74 Views | Created by sai_09 - Thursday, December 12, 2019 11:51 AM | Last reply by Zekaryah - Thursday, December 12, 2019 2:03 PM -
0 Votes
DataFlow Can't Connect To SQL Source Dataset
I am working in Azure Data factory (v2) on moving data from CSV files to Azure SQL Database. The Copy data is working fine for moving data, but I need to do some modification on the data ...Unanswered | 15 Replies | 565 Views | Created by Joe Tannouri - Monday, December 2, 2019 8:51 AM | Last reply by Joe Tannouri - Thursday, December 12, 2019 1:05 PM -
1 Votes
Developments tools other than Author/monitor portal in ADF
Hi, I am exploring Azure Data Factory, i see Azure is providing Author/monitor portal but I am looking for any other development tools such as Visual studio, Visual code or any ...Proposed | 3 Replies | 139 Views | Created by Cloud learner - Wednesday, December 11, 2019 12:07 PM | Last reply by Zekaryah - Wednesday, December 11, 2019 4:58 PM -
0 Votes
How to write the activity status to a parameter
I need to capture the system variables (Pipeline Name, etc. ) in addition to some other pieces of information about an activity (Status, Name, Rows Read) and pass it from the Copy task to a Stored ...Proposed | 3 Replies | 607 Views | Created by Shannon Royer - Wednesday, June 13, 2018 3:21 PM | Last reply by Andries van der Walt - Wednesday, December 11, 2019 12:17 PM -
0 Votes
Importing Daily snapshots of data to Data Lake Storage Gen 2
I have a source table in a SQL Database that contains daily snapshots of whats going on. What I want to do is get daily files of each snapshot out into a Data Lake Gen 2 Data store ...Proposed | 1 Replies | 104 Views | Created by Debbie Edwards - Wednesday, December 11, 2019 9:25 AM | Last reply by Vaibhav-Chaudhari - Wednesday, December 11, 2019 10:07 AM -
0 Votes
Order in foreach
I've made some pipelines; Fill staging tablesExecute all stored proceduresRefresh Power BI datasets (in progress)Master pipeline which first executes the fill staging ...Proposed | 4 Replies | 270 Views | Created by MiKeHendriks - Friday, December 6, 2019 8:49 PM | Last reply by MiKeHendriks - Wednesday, December 11, 2019 8:24 AM -
0 Votes
How to process match and Nomatch data in ADF data flow?
Hi, I am working on to transfer data from one database to another using ADF dataflow (Mapping dataflow). There are some scenario where I need to look for lookup ...Proposed | 6 Replies | 319 Views | Created by AmitBDesai - Friday, December 6, 2019 1:20 AM | Last reply by AmitBDesai - Wednesday, December 11, 2019 3:47 AM -
0 Votes
Issues in self hosted integration runtime connectivity
Hi Team , I have created a self hosted IR and I am trying to connect to on-prem oracle database . However I am facing below error ...Unanswered | 2 Replies | 266 Views | Created by Lopamudra Kanungo - Wednesday, November 27, 2019 4:49 PM | Last reply by HimanshuSinha-msft - Tuesday, December 10, 2019 7:37 PM -
0 Votes
Failed activity does not fail the pipeline
Hello, I have a pipeline with few dependent activities and one of them is a Stored Procedure activity. When this SP activity failes (for some reasons) the activity is marked as ...Proposed | 8 Replies | 390 Views | Created by paljon - Thursday, November 21, 2019 11:58 AM | Last reply by HimanshuSinha-msft - Tuesday, December 10, 2019 7:35 PM -
0 Votes
Gen2 to SQL Database Table adding derived column during Copy Activity
How to Add derived column using the copy activity from source Gen2 to target SQL Database Table. I come across option to define explicit schema in configuration table but in my ...Proposed | 3 Replies | 215 Views | Created by Sam cit - Wednesday, December 4, 2019 6:41 PM | Last reply by MartinJaffer-MSFT - Tuesday, December 10, 2019 5:50 PM -
0 Votes
Your Azure-SSIS IR has failed to start due to custom setup failure.
Hi, I am working on the Azure SSIS IR. Where i need to deploy the SSIS package on the Azure so i am creating IR(Integration Runtime) in the Azure Data Factory but i am facing below ...Answered | 3 Replies | 758 Views | Created by Sam D'Souza - Tuesday, September 4, 2018 6:51 AM | Last reply by TariqMAwan - Tuesday, December 10, 2019 12:28 PM -
2 Votes
Azure Data Factory Pagination with offset & limit paramters
Hello, I have a Rest API that I want to copy data from into a data lake however the api uses offset and limit query parameters for Pagination. I was wonder how I would do this with Pagination ...Answered | 2 Replies | 132 Views | Created by BryBry02 - Monday, December 9, 2019 7:33 PM | Last reply by ChiragMishra-MSFT - Tuesday, December 10, 2019 9:41 AM -
0 Votes
if exists
In my Azure Data Factory pipeline, there is a copy activity with an input parameter. which copies data from on prem sql server to Azure sql server... The destination (Sink) table ...Proposed | 2 Replies | 240 Views | Created by arkiboys - Saturday, December 7, 2019 3:01 PM | Last reply by ChiragMishra-MSFT - Tuesday, December 10, 2019 9:16 AM -
1 Votes
Connecting to Oracle DB which is in On Premise Server
Hi All, I have a requirement where I need to get data from Oracle DB which is on premise server. I have got the solution of Self Hosted ...Proposed | 5 Replies | 317 Views | Created by Pankaj Mahra - Thursday, November 28, 2019 1:14 PM | Last reply by ChiragMishra-MSFT - Tuesday, December 10, 2019 9:09 AM
Azure Data Factory announcement
-
Link
We have noticed customers post their questions on Azure Databricks on incorrect forums on MSDN, since there is a new home for Azure Databricks.
Forum Link: https://social.msdn.microsoft.com/Forums/en-US/home?forum=AzureDatabricks
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
