locked
ADF stopped producing dataset - all upstream datasets are ready RRS feed

  • Question

  • Hi,

    two (or three) days ago and ADF I had running stopped working. At first just all SQL Activities failed with a timeout (seems the AZURE SQL was not available at that day - ‎11‎/‎26‎/‎2015‎ ‎01‎:‎03‎:‎12‎ ‎AM UTC ). There have been 14 such datasets. Based on this 14 datasets one dependent dataset is produced. Since this day I cannot get this dependent dataset to get produced. The Slice says it is pending execution, but there is no upstream dataset pending...

    Why does it not get produced?

    I deleted the Pipeline and recreated it, but still no success.

    Any Ideas?

    the last think I did not try so far is deleting all the datasets, but since I then lose the log and status of all the past slices I do not want to go this way...

    Any hints what could be wrong? If it would be on premise I would think something stopped working and I need a reboot next, but what to do in the cloud?

    How could an active pipeline with all input datasets beeing produced not change from "pending execution" to "in progress"?

    Thank for your advice.

    Hannes


    http://www.hmayer.net/


    • Edited by hmayer1980 Saturday, November 28, 2015 7:57 AM
    Friday, November 27, 2015 3:06 PM

Answers

  • Finally I deleted the Pipeline and all datasets and deployed them again.

    It is working now....


    http://www.hmayer.net/

    • Marked as answer by hmayer1980 Tuesday, December 1, 2015 6:56 AM
    Tuesday, December 1, 2015 6:56 AM

All replies

  • Everything is getting more and more confusing...

    I still think the system needs are reboot....

    Source DS:

    Target DS:

    Look at the delay.. The source was produced at 1 am, but the target DS reports sucess at 7 pm - but also has in "internal Error" while reporting success....

    This is the activity Code

                {
                    "type": "Copy",
                    "typeProperties": {
                        "source": {
                            "type": "BlobSource"
                        },
                        "sink": {
                            "type": "SqlSink",
                            "sqlWriterStoredProcedureName": "Stage.uspLoadWeblog",
                            "storedProcedureParameters": {
                                "LogDate": {
                                    "value": "$$Text.Format('{0:yyyy-MM-dd}',SliceStart)"
                                },
                                "Website": {
                                    "value": "http://www.hmayer.net"
                                }
                            },
                            "sqlWriterTableType": "Stage.WeblogType",
                            "sqlWriterCleanupScript": "$$Text.Format('exec Stage.uspDeleteWeblog @Website=\\'http://www.hmayer.net\\', @LogDate=\\'{0:yyyy-MM-dd}\\'',SliceStart)",
                            "writeBatchSize": 10000,
                            "writeBatchTimeout": "00:00:00"
                        }
                    },
                    "inputs": [
                        {
                            "name": "weblog-File-net-hmayer-http"
                        }
                    ],
                    "outputs": [
                        {
                            "name": "weblog-Sql-Stage-net-hmayer-http"
                        }
                    ],
                    "policy": {
                        "timeout": "03:00:00",
                        "delay": "01:00:00",
                        "concurrency": 1,
                        "retry": 1
                    },
                    "scheduler": {
                        "frequency": "Day",
                        "interval": 1,
                        "style": "EndOfInterval"
                    }

    and the source DS

    {
        "name": "weblog-File-net-hmayer-http",
        "properties": {
            "structure": [
                {
                    "name": "Weblog",
                    "type": "String"
                }
            ],
            "published": false,
            "type": "AzureBlob",
            "linkedServiceName": "AzureStorageLinkedService",
            "typeProperties": {
                "folderPath": "hmayernet-weblog/hmayer.net/http/{Year}/{Month}/{Day}/access.log.gz",
                "format": {
                    "type": "TextFormat",
                    "columnDelimiter": "Ԓ"
                },
                "partitionedBy": [
                    {
                        "name": "Year",
                        "value": {
                            "type": "DateTime",
                            "date": "SliceStart",
                            "format": "yyyy"
                        }
                    },
                    {
                        "name": "Month",
                        "value": {
                            "type": "DateTime",
                            "date": "SliceStart",
                            "format": "MM"
                        }
                    },
                    {
                        "name": "Day",
                        "value": {
                            "type": "DateTime",
                            "date": "SliceStart",
                            "format": "dd"
                        }
                    }
                ],
                "compression": {
                    "type": "GZip",
                    "level": "Optimal"
                }
            },
            "availability": {
                "frequency": "Day",
                "interval": 1,
                "style": "EndOfInterval"
            },
            "external": true,
            "policy": {
                "externalData": {
                    "dataDelay": "3.01:00:00"
                }
            }
        }
    }

    Target DS

    {
        "name": "weblog-Sql-Stage-net-hmayer-http",
        "properties": {
            "published": false,
            "type": "AzureSqlTable",
            "linkedServiceName": "AzureSqlLinkedService",
            "typeProperties": {
                "tableName": "Weblog"
            },
            "availability": {
                "frequency": "Day",
                "interval": 1,
                "style": "EndOfInterval"
            }
        }
    }

    Something strange is happening....


    http://www.hmayer.net/

    Saturday, November 28, 2015 8:47 PM
  • Finally I deleted the Pipeline and all datasets and deployed them again.

    It is working now....


    http://www.hmayer.net/

    • Marked as answer by hmayer1980 Tuesday, December 1, 2015 6:56 AM
    Tuesday, December 1, 2015 6:56 AM