locked
Azure Data Factory not picking up input blob RRS feed

  • Question

  • Hello,

    I have a ADF that uses Azure BLOB as input and applies machine learning model to it as below:

        "activities": [
          {
            "name": "SentimentAnalysisMLActivity",
            "description": "Get Sentiment and Confident Level of Tweets using a ML Model",
            "inputs": [
              {
                "name": "ContosoTweetsBlob"
              }
            ],
            "outputs": [
              {
                "name": "ContosoTweetsSentimentBlob"
              }
            ],
            "linkedServiceName": "AzureMachineLearningLinkedService",
            "type": "AzureMLBatchScoring",
            "policy": {
              "concurrency": 1,
              "executionPriorityOrder": "NewestFirst",
              "retry": 1,
              "timeout": "02:00:00"
            }
          }

    The input Blob is configured for 15 mins availability as below:

    {
      "name": "ContosoTweetsBlob",
      "properties": {
        "type": "AzureBlob",
        "linkedServiceName": "StorageLinkedService",
        "typeProperties": {
          "folderPath": "twitteranalysis/twitter/aggregate/TweetsBlob/",
          "fileName": "Tweets.csv",
          "format": {
            "type": "TextFormat",
            "columnDelimiter": ","
          }
        },
        "availability": {
          "frequency": "minute",
          "interval": 15
        }
      }
    }

    Now, the input blob is available at specified location and the pipeline is started so that the Blob's Modified time falls between the activity's StartDateTime and EndDateTime. However, the activity always stays in waiting state.

    Can someone please clarify the logic of an activity dependent on input Blob to start processing? What exactly is the business rule to be satisfied for activity to start?

    Regards

    Tuesday, August 2, 2016 9:48 AM

Answers

  • Well i finally found that the input dataset was not being generated within the data factory and hence required the External property to be set to true.

    Doing that made the factory start again.

    I feel the error messages are not really helpful in debugging and need a lot of improvement.

    Regards

    • Marked as answer by Nirav M Thursday, August 4, 2016 11:32 PM
    Thursday, August 4, 2016 11:32 PM