Failed ADF time slices (how to restart in batch) RRS feed

  • Question

  • Hi,

    We currently have pipelines that are run hourly in ADF. The pipelines are configured to restart up to 10 times on failure. 

    The Azure Datawarehouse server was down late on Friday and through the weekend. So all timeslices failed over the weekend.

    I would like to know if there is a way of programatically restarting failed time slices (in BATCH). I am thinking some kind powershell process scheduled to run all the time and restart the failed slices.

    Any thoughts very much appreciated.


    Peter Gadsby

    Monday, October 10, 2016 9:24 AM

All replies

  • If we are doing it through powershell then it can be achievable.

    Get the dataset slice with had failed after 10 attempt by the below command.



    Then run the last dataset for that particular slice by the below command with UpdateType as "UpstreamInPipeline".



     Let me know if you need more inputs regarding this.

    Monday, October 10, 2016 10:17 AM
  • Many thanks Prasant I have now implemented a PowerShell script that identifies all failed time slices and sets the status to restart them. I haven't used PowerShell before, so if I have missed a trick on the code enclosed please let me know.

    Module connects to Azure Data Factory using the Azure profile given and sets all failed
    data slices to "Waiting" to trigger the timeslices to run again.
    1. Azure Profile - This can be generated by logging into Azure using the cmdlet Login-AzureRmAccount
                    then saving the profile using cmdlet Save-AzureRmProfile
    1. StartDatetime - Start of timeslice range to check for failure 
                default "" if defaulted will use the Daysback from today to generate the date
                Format: yyyy-mm-ddThh:MM:SSZ
    2. EndDatetime - End of timeslice range to check for failure
                default "" if defaulted will use now date
                Format: yyyy-mm-ddThh:MM:SSZ
    3. Rootdirectory - The directory that this script resides in and where the profile file is held
    4. AzureProfile - Filename that stores the azure profile information (within the RootDirectory)
    5. ADFName - Name of the Azure Data Factory server
    6. ADFGroup - Name of th eAure group that the Azure Data Factory resides in in Azure
    7. AzureRootDir - Root directory that holds the Powershell cmdlets for Azure
    8. Daysback - Number of days back from today to generate StartDatetime
    Amendment History
    Date		By				Description
    ----		---				-----------
    12/10/16    Peter Gadsby    Initial Version
    param (
        [string]$startDateTime = "",        #"2016-10-09T00:00:00Z",
        [string]$endDateTime =   "",        #"2016-10-11T23:59:59Z",
        [string]$RootDirectory = "C:\PRG",
        [string]$AzureProfile = "AzureProfile",
        [string]$AzureRootDirectory="C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\",
        [int]$DaysBack = -1
     $env:PSModulePath=$env:PSModulePath + ";" +  
    		$AzureRootDirectory  + "ServiceManagement\;" + 
    		$AzureRootDirectory  + "ResourceManager\AzureResourceManager\;" +
    		$AzureRootDirectory  + "ServiceManagement\;" + 
    		$AzureRootDirectory  + "Storage\"
    IF ($startDateTime -eq "")
         $startDateTime = (get-date (Get-Date).AddDays($DaysBack).ToUniversalTime() -format s) + "Z"
    IF ($endDateTime -eq "")
         $endDateTime = (get-date (Get-Date).ToUniversalTime() -format s) + "Z"
    # Output parameters
    write-host " "
    write-host "-------------------------------------------------------------"
    write-host "Parameters:-"
    write-host "1. Start Date    : "$($startDateTime)
    write-host "2. End Date      : "$($endDateTime)
    write-host "3. RootDirectory : "$($rootDirectory)
    write-host "4. AzureProfile  : "$($AzureProfile)
    write-host "5. ADFName       : "$($ADFName)
    write-host "6. ADFGroup      : "$($ADFGroup)
    write-host "7. AzureRootDir  : "$($AzureRootDirectory)
    write-host "-------------------------------------------------------------"
    write-host " " 
    $Profile=$RootDirectory + "\" + $AzureProfile
    Select-AzureRmProfile -Path $Profile
    $tableNames = Get-AzureRMDataFactoryDataset `
                                -DataFactoryName $ADFName `
                                -ResourceGroupName $ADFGroup | ForEach {$_.DatasetName}
    $TableCount = @($tableNames).Count
    foreach ($tableName in $tableNames)
            $slices += Get-AzureRMDataFactorySlice `
                               -DataFactoryName $ADFName `
                               -ResourceGroupName $ADFGroup `
                               -DatasetName $tableName  `
                               -StartDateTime $startDateTime `
                               -EndDateTime $endDateTime | Where {$_.State -eq "Failed"}
     $failedSlices = $slices
     $failedSlicesCount = @($failedSlices).Count
     if ( $failedSlicesCount -gt 0 ) 
        foreach ($Failed in $failedSlices)
            write-host "-----------------------------------------------"
            write-host "Dataset Name (FAILED): "$($failed.Datasetname) 
            write-host "Start: "$($failed.Start) " End: "$($failed.End)
            write-host "Status: "$($failed.State)
            write-host "Setting status to Waiting...."
            Set-AzureRMDataFactorySliceStatus `
                               -UpdateType Individual `
                               -Status Waiting `
                               -DataFactoryName $ADFName `
                               -DatasetName $failed.Datasetname `
                               -ResourceGroupName $ADFGroup `
                               -StartDateTime $failed.Start `
                               -EndDateTime $failed.End        
    	write-host "There are no Failed slices."


    Wednesday, October 12, 2016 2:52 PM