locked
Failed ADF time slices (how to restart in batch) RRS feed

  • Question

  • Hi,

    We currently have pipelines that are run hourly in ADF. The pipelines are configured to restart up to 10 times on failure. 

    The Azure Datawarehouse server was down late on Friday and through the weekend. So all timeslices failed over the weekend.

    I would like to know if there is a way of programatically restarting failed time slices (in BATCH). I am thinking some kind powershell process scheduled to run all the time and restart the failed slices.

    Any thoughts very much appreciated.

    Thanks

    Peter Gadsby

    Monday, October 10, 2016 9:24 AM

All replies

  • If we are doing it through powershell then it can be achievable.

    Get the dataset slice with had failed after 10 attempt by the below command.

    $DataSetSliceDetails=Get-AzureRmDataFactoryRun-ResourceGroupName$ResourceGroupName`

    -DataFactoryName$DataFactoryName-DatasetName$FinalDataset-StartDateTime$StartDateTime|Select-Object-Last1

    Then run the last dataset for that particular slice by the below command with UpdateType as "UpstreamInPipeline".

    Set-AzureRmDataFactorySliceStatus-ResourceGroupName$ResourceGroupName-DataFactoryName$DataFactoryName-DatasetName$Dataset`

    -StartDateTime$StartDateTime-EndDateTime$EndDateTime-Status$DatasetStatus-UpdateType$UpdateType 

     Let me know if you need more inputs regarding this.

    Monday, October 10, 2016 10:17 AM
  • Many thanks Prasant I have now implemented a PowerShell script that identifies all failed time slices and sets the status to restart them. I haven't used PowerShell before, so if I have missed a trick on the code enclosed please let me know.

    <#
    ===============================================================================================
    Description
    -----------
    Module connects to Azure Data Factory using the Azure profile given and sets all failed
    data slices to "Waiting" to trigger the timeslices to run again.
    
    Dependancies
    ------------
    1. Azure Profile - This can be generated by logging into Azure using the cmdlet Login-AzureRmAccount
                    then saving the profile using cmdlet Save-AzureRmProfile
    
    Parameters
    ----------
    1. StartDatetime - Start of timeslice range to check for failure 
                default "" if defaulted will use the Daysback from today to generate the date
    
                Format: yyyy-mm-ddThh:MM:SSZ
    
    2. EndDatetime - End of timeslice range to check for failure
                default "" if defaulted will use now date
    
                Format: yyyy-mm-ddThh:MM:SSZ
    3. Rootdirectory - The directory that this script resides in and where the profile file is held
    
    4. AzureProfile - Filename that stores the azure profile information (within the RootDirectory)
    
    5. ADFName - Name of the Azure Data Factory server
    
    6. ADFGroup - Name of th eAure group that the Azure Data Factory resides in in Azure
    
    7. AzureRootDir - Root directory that holds the Powershell cmdlets for Azure
    
    8. Daysback - Number of days back from today to generate StartDatetime
    
    
    Amendment History
    -----------------
    
    Date		By				Description
    ----		---				-----------
    12/10/16    Peter Gadsby    Initial Version
    ===============================================================================================
    
    
    
    #>
    
    param (
        [string]$startDateTime = "",        #"2016-10-09T00:00:00Z",
        [string]$endDateTime =   "",        #"2016-10-11T23:59:59Z",
        [string]$RootDirectory = "C:\PRG",
        [string]$AzureProfile = "AzureProfile",
        [string]$ADFName="XXXXXXXXXX",
        [string]$ADFGroup="YYYYYYYYY",
        [string]$AzureRootDirectory="C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\",
        [int]$DaysBack = -1
        )
    
    
     $env:PSModulePath=$env:PSModulePath + ";" +  
    		$AzureRootDirectory  + "ServiceManagement\;" + 
    		$AzureRootDirectory  + "ResourceManager\AzureResourceManager\;" +
    		$AzureRootDirectory  + "ServiceManagement\;" + 
    		$AzureRootDirectory  + "Storage\"
    
    IF ($startDateTime -eq "")
    {
         $startDateTime = (get-date (Get-Date).AddDays($DaysBack).ToUniversalTime() -format s) + "Z"
    }
    
    IF ($endDateTime -eq "")
    {
         $endDateTime = (get-date (Get-Date).ToUniversalTime() -format s) + "Z"
    }
    
    
    # Output parameters
    
    write-host " "
    write-host "-------------------------------------------------------------"
    write-host "Parameters:-"
    write-host "1. Start Date    : "$($startDateTime)
    write-host "2. End Date      : "$($endDateTime)
    write-host "3. RootDirectory : "$($rootDirectory)
    write-host "4. AzureProfile  : "$($AzureProfile)
    write-host "5. ADFName       : "$($ADFName)
    write-host "6. ADFGroup      : "$($ADFGroup)
    write-host "7. AzureRootDir  : "$($AzureRootDirectory)
    write-host "-------------------------------------------------------------"
    write-host " " 
    
    $Profile=$RootDirectory + "\" + $AzureProfile
    
    $tableNames=@() 
    $slices=@()
    
    Select-AzureRmProfile -Path $Profile
    
    
    $tableNames = Get-AzureRMDataFactoryDataset `
                                -DataFactoryName $ADFName `
                                -ResourceGroupName $ADFGroup | ForEach {$_.DatasetName}
    
    
    $TableCount = @($tableNames).Count
    
    foreach ($tableName in $tableNames)
    {
            $slices += Get-AzureRMDataFactorySlice `
                               -DataFactoryName $ADFName `
                               -ResourceGroupName $ADFGroup `
                               -DatasetName $tableName  `
                               -StartDateTime $startDateTime `
                               -EndDateTime $endDateTime | Where {$_.State -eq "Failed"}
    
    }
    
     $failedSlices = $slices
    
     $failedSlicesCount = @($failedSlices).Count
    
     if ( $failedSlicesCount -gt 0 ) 
     {
        foreach ($Failed in $failedSlices)
        {
    
            write-host "-----------------------------------------------"
            write-host "Dataset Name (FAILED): "$($failed.Datasetname) 
            write-host "Start: "$($failed.Start) " End: "$($failed.End)
            write-host "Status: "$($failed.State)
            write-host "Setting status to Waiting...."
    
    
            Set-AzureRMDataFactorySliceStatus `
                               -UpdateType Individual `
                               -Status Waiting `
                               -DataFactoryName $ADFName `
                               -DatasetName $failed.Datasetname `
                               -ResourceGroupName $ADFGroup `
                               -StartDateTime $failed.Start `
                               -EndDateTime $failed.End        
    
    
        }
    
     }
     else
     {
    	write-host "There are no Failed slices."
     }
    

    Pete 

    Wednesday, October 12, 2016 2:52 PM