none
ADF loading data from Amazon Redshift

    Question

  • Hi,

    I need to load data from Amazon Redshift using ADF V2. I am new to ADF V2. Let me explain the scenario. 

    I have a bunch of values in a category table, the table (Category) contains two fields (1) Category Name (2) Date Created. Let's say this table contains two rows

    (1) Cat 1 , 2018-01-02

    (2) Cat 2 , 2018-04-01

    This Category table is in Azure SQL Server. I need to read each value from the Category table and load the data from Redshift using the 'Category Name' and the 'Date Created' filters in a loop. In the first loop the Cat 1 and date 2018-01-02 would be used to filter the source data on Redshift, on the second run the next row would be used.

    Let me know if you have any questions and Many thanks your time and help in advance.

    Regards

    Azvi

    Wednesday, April 4, 2018 5:47 PM

Answers

  • From my understanding, You can do the following activities in the pipeline:

    1. Load both the data sets.

    2. In the transformation step, you can do an inner join on category and on the date created field. if you prefer U-SQL, below is snippet you can modify by adding an AND on the inner join for your case.

    https://msdn.microsoft.com/en-us/azure/data-lake-analytics/u-sql/inner-join-u-sql?f=255&MSPPError=-2147217396

    3. Store the results in the output data store.

    Please let me know if that helps.


    Thursday, April 5, 2018 12:07 AM
    Moderator

All replies

  • From my understanding, You can do the following activities in the pipeline:

    1. Load both the data sets.

    2. In the transformation step, you can do an inner join on category and on the date created field. if you prefer U-SQL, below is snippet you can modify by adding an AND on the inner join for your case.

    https://msdn.microsoft.com/en-us/azure/data-lake-analytics/u-sql/inner-join-u-sql?f=255&MSPPError=-2147217396

    3. Store the results in the output data store.

    Please let me know if that helps.


    Thursday, April 5, 2018 12:07 AM
    Moderator
  • Thanks Vairavan for the detailed response and suggesting the u-sql query!
    Thursday, April 5, 2018 5:07 PM