none
Data Factory V1 Custom Activity from S3 source to Azure SQL

    Question

  • Hi,

    I am trying to create a custom activity with excel S3 as source and Azure SQL as destination. In between in the custom activity I need to process few columns using a API call from S3 source and save it back to Azure SQL.

    How well this process is achievable?

    Also, I am following this tutorial (https://azure.microsoft.com/en-us/documentation/articles/data-factory-use-custom-activities/#see-also) but it has both input and output as blobs. Can I also get some sample code for S3 storage as input and Azure SQL as output.

    This example i got from the stackoverflow post: ****AzureSqlDatabaseLinkedService sqlInputLinkedService; AzureSqlTableDataset sqlInputLocation;****

    But I wanted one for S3 as Linked service.

    Do we have for S3 as as a Linked Server?

    And how do i create a data set. We have only this in the sample code.

            **Dataset outputDataset = datasets.Single(dataset => dataset.Name == activity.Outputs.Single().Name);**
    

    Please help me with some sample data sets so that i can call the column and bind the vale to the API.

    Kindly let me know if all these are achievable or not with DF.

    Requesting for some help.

    Thanks, Surendar

    Friday, May 18, 2018 6:02 PM

All replies

  • If you are trying to get data from s3, transform data using Rest API and store in azure SQL. Please do the following steps

    1. Create copy activity in pipeline to retrieve data from s3(Link)

    2. Create a web activity to do the transformation.

    3. Store the transformed data in azure SQL

    Please let me know if you face any issues


    Friday, May 25, 2018 8:32 PM
    Moderator