Azure Data Factory announcement
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
-
0 VotesADF v1 : Integration run time set up
Hello All, As per my understanding , only 1 integration run time can be installed on a virtual machine which can be connected to only 1 ADF v1 via its key. SO ...Proposed | 3 Replies | 122 Views | Created by Nandan Hegde - Wednesday, August 14, 2019 1:09 PM | Last reply by Nandan Hegde - Monday, August 19, 2019 3:17 AM
-
0 Votes
Azure Data Factory - Trigger and Tumbling Window
Hi, I have a pipeline that has two pipeline string parameters.... Paramter Name - Type - DefaultValue windowStart - string - ...Unanswered | 1 Replies | 0 Views | Created by Caz71 - Sunday, August 18, 2019 2:44 PM | Last reply by Wang Zhang - Monday, August 19, 2019 2:10 AM -
0 Votes
@Substring With Unknown Length
Hello. I don't know why these ADF Expressions are so hard to get right :-) I have a ComanyID where I use the ... -
0 Votes
What are the ways to migrate data from MongoDB to Azure SQL database?
Hello, I have a MongoDB database (not on Azure) and I want to migrate the data from MongoDB to Azure SQL database. Can someone please suggest ...Unanswered | 1 Replies | 70 Views | Created by Harsh Jariwala - Saturday, August 17, 2019 6:00 AM | Last reply by Srihad - Sunday, August 18, 2019 3:44 PM -
0 Votes
Azure Data Factory - Copy Folder Causing Error
Hi , I am using the latest version of Data Factory and have created one pipeline. The pipeline does the following... 1) Creates a list if ...Answered | 3 Replies | 0 Views | Created by Caz71 - Saturday, August 17, 2019 10:00 AM | Last reply by Caz71 - Saturday, August 17, 2019 7:20 PM -
0 Votes
Handling transient errors
I have ADF pipelines that are using Dataflows to copy data into Azure SQL DB tables. On occasion I am getting transient errors being raised by the Azure SQL database, which is ...Unanswered | 1 Replies | 78 Views | Created by xhead - Thursday, August 15, 2019 5:06 PM | Last reply by MartinJaffer-MSFT - Saturday, August 17, 2019 1:15 AM -
0 Votes
Copy Files from Azure blob storage to Azure VM Share Folder
At regular intervals, i get data from third part sources and then data is stored in Azure blob. Using azure function/web jobs, i want to copy blob files to the shared folder on azure VM. Is ... -
1 Votes
How to use backslash in Azure Data Factory?
Hi all. I have an 'Execute SSIS Package' activity in my ADF, that requires from me passing the parameter for Environment. In my case the paramater value is ".\Dev". ...Proposed | 2 Replies | 192 Views | Created by jangcy - Tuesday, June 11, 2019 11:11 AM | Last reply by MounikaPisati - Friday, August 16, 2019 10:20 PM -
0 Votes
Spark activity fails when submitting two jobs concurrently
Hi, I build a simple WordCount Spark job and try to run it in Spark ADF pipelines. I notice that if I submit two jobs concurrently to the same Spark Linked Service, both jobs will ... -
0 Votes
@CONCAT Question
Hello. I have the following Dynamic Content; it seems simple: @concat('azfunc-out\',variables('CompanyID')) but I am getting the following ...Answered | 6 Replies | 130 Views | Created by EMKISER - Thursday, August 15, 2019 10:29 PM | Last reply by MartinJaffer-MSFT - Friday, August 16, 2019 9:15 PM -
0 Votes
Error in ADF clean up script
Hi, I have configured Azure DevOps Release pipeline based on the automated CI/CD documentation provided and one of the task in the ...Proposed | 2 Replies | 84 Views | Created by VijayVish - Tuesday, August 13, 2019 3:09 PM | Last reply by MartinJaffer-MSFT - Friday, August 16, 2019 5:50 PM -
0 Votes
Err deploying to SSIDB on Azure MI
Trying to deploy package from VS 2017 to Azure MI SSISDB. But it failed with the below err msg. Works from VS2015 Failed to deploy the project. Try again later. ...Unanswered | 2 Replies | 158 Views | Created by willtwc - Tuesday, August 13, 2019 11:47 PM | Last reply by HimanshuSinha-msft - Friday, August 16, 2019 5:40 PM -
0 Votes
Pipeline's error output could not be read completely
Hi, I am executing a child pipeline (pipeline_child) in a pipeline (pipeline_main) and try to catch error message of child pipeline, but the error.message is being cut when ...Answered | 1 Replies | 88 Views | Created by MustafaYalcin - Thursday, August 15, 2019 3:32 PM | Last reply by KranthiPakala-MSFT - Friday, August 16, 2019 4:50 PM -
0 Votes
Error while creating metric alerts in ADF using .NET SDK
Hi, I am trying to create metric alerts in Azure Data Factory using .Net SDK Below is my Json Request : { "location": ...Unanswered | 3 Replies | 123 Views | Created by Ramandeep123 - Wednesday, August 14, 2019 7:33 AM | Last reply by Ramandeep123 - Friday, August 16, 2019 9:47 AM -
0 Votes
Deploy Triggers automatically through ARM templates
Hi, I want to deploy the triggers for a pipeline automatically based on ARM template. The definition is as below However, I notice two things that impede a full ...Answered | 3 Replies | 131 Views | Created by enjoyear - Tuesday, August 13, 2019 6:26 PM | Last reply by enjoyear - Friday, August 16, 2019 1:33 AM -
1 Votes
ADF V2 : "Copy Data" activity from MySQL to Blob errors due to null values in Source MySQL table
I started having a similar issue starting late yesterday doing a "Copy Data" activity from MySQL to Blob. The error I receive is: "errorCode": ...Unanswered | 6 Replies | 141 Views | Created by Mik Cheez - Thursday, August 15, 2019 12:30 AM | Last reply by KranthiPakala-MSFT - Friday, August 16, 2019 12:23 AM -
0 Votes
Azure Data factory
"errorCode": "2200", "message": "ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An ...Unanswered | 3 Replies | 147 Views | Created by v.c.pawan theja - Tuesday, August 13, 2019 11:24 AM | Last reply by KranthiPakala-MSFT - Thursday, August 15, 2019 10:39 PM -
0 Votes
Dataflow: How can I pass a parameter to the dataset used by the source?
Hi I created a parameter "FileName" in a dataflow which I would like to pass to the dataset I use in my source. I also created a parameter in this dataset but I haven't ...Proposed | 3 Replies | 107 Views | Created by Paul Pelletier - Monday, August 12, 2019 7:18 PM | Last reply by KranthiPakala-MSFT - Thursday, August 15, 2019 10:29 PM -
0 Votes
Windows Updates and patches breaks our SSIS runtime environment twice
Windows Updates and patches breaks our SSIS runtime Production twice. Please does anyone knows how Microsoft communicate updates to PASS environment such as Azure Data factory ...Unanswered | 2 Replies | 86 Views | Created by Abioye - Monday, August 12, 2019 3:07 PM | Last reply by HimanshuSinha-msft - Thursday, August 15, 2019 7:35 PM -
0 Votes
Region conflict
> Load Data) Thanks, JagProposed | 1 Replies | 96 Views | Created by jagdish.chichria - Thursday, August 15, 2019 5:37 AM | Last reply by MartinJaffer-MSFT - Thursday, August 15, 2019 7:06 PM
Azure Data Factory announcement
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
