Azure Data Factory announcement
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
-
0 VotesAzure Data Factory - Copy Activity Json Array
I'm trying to pull data from MongoDB in Azure Data Factory V2. The data structure is some thing like below. { "Username":"Joe", ...Proposed | 2 Replies | 56 Views | Created by Joe Samraj - Thursday, June 13, 2019 6:30 AM | Last reply by Joe Samraj - Thursday, June 13, 2019 11:38 AM
-
0 Votes
Access SharePoint list data using ADF
Hi, This seem to be the most discussed issue all over, however could not find any solution or relevant documentation to get it working. The set up is quite ... -
2 Votes
Azure Data Factory Alerting
Hi, I have created an Azure Data Factory to copy data from Azure CosmosDB to Azure Blob Storage. The copy activity pipeline is working as expected, currently set to run on a weekly ...Answered | 7 Replies | 156 Views | Created by Gus_7 - Monday, June 10, 2019 7:48 PM | Last reply by MartinJaffer-MSFT - Wednesday, June 12, 2019 9:51 PM -
0 Votes
Please help! I have a question on the Source for the Copy Activity inside a ForEach for ADFv2
Here is what I want to do: 1) I have a lookup activity working with is looking up data on which connection string to a DB goes with which ...Answered | 3 Replies | 101 Views | Created by EMKISER - Tuesday, June 11, 2019 11:43 PM | Last reply by MartinJaffer-MSFT - Wednesday, June 12, 2019 9:21 PM -
0 Votes
Data Factory set pipeline in arm template
I'm trying to set my data factory pipeline via the arm template, I do it as follow: { "name": "[variables('DataFactoryName')]", ...Proposed | 2 Replies | 171 Views | Created by LironLevy - Monday, June 10, 2019 5:11 PM | Last reply by LironLevy - Wednesday, June 12, 2019 1:59 PM -
1 Votes
How to use backslash in Azure Data Factory?
Hi all. I have an 'Execute SSIS Package' activity in my ADF, that requires from me passing the parameter for Environment. In my case the paramater value is ".\Dev". ...Proposed | 1 Replies | 75 Views | Created by jangcy - Tuesday, June 11, 2019 11:11 AM | Last reply by MartinJaffer-MSFT - Wednesday, June 12, 2019 1:15 AM -
0 Votes
Can Azure Data Factory constantly upload data from a Blob to process?
Can Azure Data Factory constantly upload data from a Blob to process? Noticed that in the below link the upload of the raw data from the Blob is manual or at most scheduled in ...Proposed | 3 Replies | 249 Views | Created by kimdav111 - Tuesday, June 4, 2019 9:04 PM | Last reply by KranthiPakala-MSFT - Tuesday, June 11, 2019 9:36 PM -
0 Votes
Event trigger not firing in Data Factory
I've created an event trigger in DataFactory but its not executed when my blob file is being created. If I click on 'Trigger now', everything execute correctly. I ...Answered | 4 Replies | 115 Views | Created by Sylvain Dorion - Monday, June 10, 2019 11:57 AM | Last reply by KranthiPakala-MSFT - Tuesday, June 11, 2019 9:03 PM -
0 Votes
ADF not executing loops sequentially for Trigger Now
I have a loop that has the sequential button checked. In Debug the Pipeline operates sequentially like it's supposed to; however, when clicking Trigger Now, it runs all loops in ...Proposed | 1 Replies | 79 Views | Created by Ward Morris - Monday, June 10, 2019 3:39 PM | Last reply by HimanshuSinha-msft - Tuesday, June 11, 2019 7:25 PM -
0 Votes
ADFv2 User Property setting
User Properties were recently added into ADFv2 but I can't find any documentation on how to use them. Is there something available?Unanswered | 2 Replies | 1777 Views | Created by FrankMn - Tuesday, June 19, 2018 3:09 PM | Last reply by Daniel VK - Tuesday, June 11, 2019 5:20 PM -
1 Votes
Dynamic schema issue in Aggregate activity in Data flow
Hi, I want to perform aggregation grouping by some column in ADF Data Flow. But the schema is getting passed dynamically. So while performing group by on ...Proposed | 4 Replies | 116 Views | Created by prachee_g_ - Thursday, June 6, 2019 11:47 AM | Last reply by Mingyang Liu (MSFT) - Tuesday, June 11, 2019 3:46 PM -
2 Votes
Data Flow Dynamic File Name - Recursively Process Files
Is it possible to iterate through sub-folders, so rather than processing hourly - we can process a days worth of data. Its straight forward to do in a "CopyData" function as there ...Proposed | 5 Replies | 143 Views | Created by Rippll - Thursday, June 6, 2019 10:21 AM | Last reply by Mingyang Liu (MSFT) - Tuesday, June 11, 2019 3:43 PM -
0 Votes
Auditing/Logging Famework for ELT/ETL using ADF v2
Hello great community people, I Know the question is bit broader in scope and cannot be limited purely to ADF , but i think it still makes more sense to pitch it in ...Proposed | 3 Replies | 292 Views | Created by Viswa is here - Friday, March 1, 2019 10:18 AM | Last reply by Anil Bangaru - Tuesday, June 11, 2019 2:35 PM -
2 Votes
How to view lineage of pipeline in ADF
I created simple pipeline with one copy activity copying table from datasource to ADLS using ADF. Here I want to view data lineage for the pipeline I created. I am new to ADF. Can you explain me how ...Proposed | 3 Replies | 1152 Views | Created by Annepu - Friday, June 8, 2018 9:32 PM | Last reply by Naveen Kumar Thummalapenta - Tuesday, June 11, 2019 10:00 AM -
0 Votes
ADF Copy to Blob is ok, but how to copy to File Directory?
I try to Bulk Copy from SQL Server to Data Lake gen2. I have succesfully bulk copied to Block Blob, then I tried to modify data set to copy to File System. Data Set still copy files ...Answered | 3 Replies | 97 Views | Created by Kenny_I - Sunday, June 9, 2019 2:39 AM | Last reply by ChiragMishra-MSFT - Tuesday, June 11, 2019 9:37 AM -
0 Votes
White-list SSIS or Auto resolved integration runtime IP for data access
Hi, looking for the recommendation to address the scenario Assume that your source database system is hosted somewhere outside of Azure ecosystem (could be My SQL hosted on AWS) and ...Unanswered | 3 Replies | 142 Views | Created by Bhushan Gawale - Thursday, June 6, 2019 9:01 AM | Last reply by Chase-MSFT - Tuesday, June 11, 2019 8:19 AM -
3 Votes
Data Factory UI is not loading
Hi team, I have a MSDN license (<label aria-hidden="true" class="msportalfx-property-name fxs-portal-title msportalfx-tooltip-overflow ...Proposed | 15 Replies | 3992 Views | Created by bijeshjoseph - Monday, February 5, 2018 10:13 AM | Last reply by T. Buckingham - Monday, June 10, 2019 7:24 PM -
0 Votes
Error in REST API Call
In REST API call, I am getting bearer token(as POST operation) in POSTMAN & in Logic Apps after passing Content-Type & apikey in ...Proposed | 3 Replies | 113 Views | Created by Preyash Vrat Azure - Friday, June 7, 2019 4:24 AM | Last reply by HimanshuSinha-msft - Monday, June 10, 2019 7:08 PM -
1 Votes
Activity failed - property 'value' doesn't exist, available properties are 'firstRow, effectiveIntegrationRuntime'.
I try to do bulk data copy from SQL Server to Data Lake by following tutorial of https://docs.microsoft.com/en-us/azure/data-factory/tutorial-bulk-copy-portal I'm getting error ...Answered | 2 Replies | 171 Views | Created by Kenny_I - Friday, June 7, 2019 2:41 PM | Last reply by KranthiPakala-MSFT - Monday, June 10, 2019 5:19 PM -
0 Votes
Issue with Relative URL and HTTP LS
I have an HTTP LS that I'm trying to call from a data set that has a Relative URL. The DS works fine if I hard code a string into Relative URL, but not when I try to dynamically produce the ...Proposed | 2 Replies | 248 Views | Created by Ward Morris - Friday, May 17, 2019 8:18 PM | Last reply by Ward Morris - Monday, June 10, 2019 3:47 PM
Azure Data Factory announcement
-
Link
Azure Data Factory allows you to manage the production of trusted information by offering an easy way to create, orchestrate, and monitor data pipelines over the Hadoop ecosystem using structured, semi-structures and unstructured data sources.
You can connect to your on-premises SQL Server, Azure database, tables or blobs and create data pipelines that will process the data with Hive and Pig scripting, or custom C# processing. The service offers a holistic monitoring and management experience over these pipelines, including a view of their data production and data lineage down to the source systems.
The outcome of Data Factory is the transformation of raw data assets into trusted information that can be shared broadly with BI and analytics tools.
The MSDN forum will be used for general discussions for Getting Started, Development, Management, and Troubleshooting using Azure Data Factory.
