How should I slice this dataset in Azure Data Factory? RRS feed

  • Question

  • I'm trying to move our SSIS tasks over to the new Azure Data Factory. My first goal is to create a report showing the hours each person spent on a project over a certain time period.

    There are some datasets that don't deal with time, so I'm not sure how to slice them. One example is the list of projects. I get a list of projects, but it just shows me the projects at the particular point in time, not a history or anything. But the pipeline requires a slice and a schedule, so I'm not sure how to specify that the list should be pulled in every time from the dataset...

    The dataset itself is coming from a web service, which is using a custom activity I created and then am putting into an Azure Storage Table.

    Any advice for me on this one?

    I posted this on SO as well, but didn't get any responses

    • Moved by Girish Prajwal Friday, February 26, 2016 4:27 PM related to Datafactory
    Thursday, February 25, 2016 10:45 PM

All replies

  • ADF's slicing model is purely Time driven at the moment.

    So the only solution to your scenario that I can think of is to use an array of DotNetActivity and each activity is responsible for a specific partition of all projects, i.e. you need to be able to maintain a manually prepared partition list or you can use customized codes to partition all projects automatically (by date/project name etc) in your DotNetActivity. Use "extendedProperties" to pass in any rule or clue that what each partition is for each custom activity.

    Scheduling can be based on how often you want to pull the data from your web service.

    Hope that helps.


    Sunday, February 28, 2016 6:06 AM
  • So two questions come to mind:

    1. Is there a reason I need to slice this data? A project list doesn't sound like it necessarily needs slicing at all.
    2. Is ADF the correct tool for what I am trying to do? Is there another Azure tool that I should be using?
    Thursday, March 3, 2016 5:10 PM