none
Error triggering pipeline RRS feed

  • Question

  • I created a Data Factory as per the documentation from the GitHub repo: https://github.com/microsoftgraph/msgraph-training-dataconnect/blob/master/Lab.md.

    However when the pipeline is executed/run, it throws the following error every time:

    Activity Copy Data1 failed: ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. There is an unexpected internal error. Please contact us for support.,Source=Microsoft.DataTransfer.ClientLibrary,'

    Below are the details of the elements within the factory:

    {
        "name": "pipeline1",
        "properties": {
            "activities": [
                {
                    "name": "Copy Data1",
                    "type": "Copy",
                    "policy": {
                        "timeout": "7.00:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "typeProperties": {
                        "source": {
                            "type": "Office365Source"
                        },
                        "sink": {
                            "type": "BlobSink"
                        },
                        "enableStaging": false
                    },
                    "inputs": [
                        {
                            "referenceName": "Office365Table1",
                            "type": "DatasetReference"
                        }
                    ],
                    "outputs": [
                        {
                            "referenceName": "AzureBlob1",
                            "type": "DatasetReference"
                        }
                    ]
                }
            ]
        }
    }

    {
        "name": "Office365Table1",
        "properties": {
            "linkedServiceName": {
                "referenceName": "Office3651",
                "type": "LinkedServiceReference"
            },
            "type": "Office365Table",
            "structure": [
                {
                    "name": "ReceivedDateTime",
                    "type": "DateTime",
                    "description": "The date and time the message was received."
                },
                {
                    "name": "SentDateTime",
                    "type": "DateTime",
                    "description": "The date and time the message was sent."
                },
                {
                    "name": "HasAttachments",
                    "type": "Boolean",
                    "description": "Indicates whether the message has attachments."
                },
                {
                    "name": "InternetMessageId",
                    "type": "String",
                    "description": "The internet message id"
                },
                {
                    "name": "Subject",
                    "type": "String",
                    "description": "The subject of the message."
                },
                {
                    "name": "Importance",
                    "type": "String",
                    "description": "The importance of the message: Low = 0, Normal = 1, High = 2."
                },
                {
                    "name": "ParentFolderId",
                    "type": "String",
                    "description": "The unique identifier for the message's parent folder."
                },
                {
                    "name": "Sender",
                    "type": "String",
                    "description": "The account that is actually used to generate the message. Format: STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>"
                },
                {
                    "name": "From",
                    "type": "String",
                    "description": "The mailbox owner and sender of the message. Format: STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>"
                },
                {
                    "name": "ToRecipients",
                    "type": "String",
                    "description": "The To recipients for the message. Format: ARRAY<STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>>"
                },
                {
                    "name": "CcRecipients",
                    "type": "String",
                    "description": "The Cc recipients for the message. Format: ARRAY<STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>>"
                },
                {
                    "name": "BccRecipients",
                    "type": "String",
                    "description": "The Bcc recipients for the message. Format: ARRAY<STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>>"
                },
                {
                    "name": "ReplyTo",
                    "type": "String",
                    "description": "The email addresses to use when replying. Format: ARRAY<STRUCT<EmailAddress: STRUCT<Name: STRING, Address: STRING>>>"
                },
                {
                    "name": "ConversationId",
                    "type": "String",
                    "description": "The ID of the conversation the email belongs to."
                },
                {
                    "name": "UniqueBody",
                    "type": "String",
                    "description": "The body of the message that is unique to the conversation.Format: STRUCT<ContentType: INT32, Content: STRING>"
                },
                {
                    "name": "IsDeliveryReceiptRequested",
                    "type": "Boolean",
                    "description": "Indicates whether a read receipt is requested for the message."
                },
                {
                    "name": "IsReadReceiptRequested",
                    "type": "Boolean",
                    "description": "Indicates whether a read receipt is requested for the message."
                },
                {
                    "name": "IsRead",
                    "type": "Boolean",
                    "description": "Indicates whether the message has been read."
                },
                {
                    "name": "IsDraft",
                    "type": "Boolean",
                    "description": "Indicates whether the message is a draft. A message is a draft if it hasn't been sent yet."
                },
                {
                    "name": "WebLink",
                    "type": "String",
                    "description": "The URL to open the message in Outlook Web App."
                },
                {
                    "name": "CreatedDateTime",
                    "type": "DateTime",
                    "description": "The date and time the message was created."
                },
                {
                    "name": "LastModifiedDateTime",
                    "type": "DateTime",
                    "description": "The date and time the message was last changed."
                },
                {
                    "name": "ChangeKey",
                    "type": "String",
                    "description": "The version of the message."
                },
                {
                    "name": "Categories",
                    "type": "String",
                    "description": "The categories associated with the message. Format: ARRAY<STRING>"
                },
                {
                    "name": "Id",
                    "type": "String",
                    "description": "The unique identifier of the message."
                },
                {
                    "name": "Attachments",
                    "type": "String",
                    "description": "The FileAttachment and ItemAttachment attachments for the message. Navigation property. Format: ARRAY<STRUCT<LastModifiedDateTime: STRING, Name: STRING, ContentType: STRING, Size: INT, IsInline: BOOLEAN, Id: STRING>>"
                }
            ],
            "typeProperties": {
                "tableName": "BasicDataSet_v0.Message_v0"
            }
        }
    }

    {
        "name": "AzureBlob1",
        "properties": {
            "linkedServiceName": {
                "referenceName": "AzureBlobStorage1",
                "type": "LinkedServiceReference"
            },
            "type": "AzureBlob",
            "typeProperties": {
                "format": {
                    "type": "JsonFormat",
                    "filePattern": "setOfObjects"
                },
                "fileName": "",
                "folderPath": "maildump"
            }
        }
    }


    Eric Legault (MVP: Outlook)
    Try MessageFiler for Outlook!

    Thursday, March 28, 2019 4:52 AM

All replies

  • Hi Eric,

    I’m working with the product team and get back to you when I have more information.

    Thursday, March 28, 2019 8:52 AM
    Moderator
  • Hi Eric,

    To help us investigate further, could you please provide the “RunId” of the copy activity that failed?

    Steps to find this information are located here.

    Thursday, March 28, 2019 4:47 PM
    Moderator
  • Sure: b8d89939-118e-4536-8502-f798a2b3be80

    Eric Legault (MVP: Outlook)
    Try MessageFiler for Outlook!

    Thursday, March 28, 2019 5:00 PM
  • FYI I tried creating another Data Factory and the same thing happened. RunId: 431c1414-0759-4cb6-9406-a971f9774877

    Eric Legault (MVP: Outlook)
    Try MessageFiler for Outlook!

    Thursday, March 28, 2019 9:48 PM
  • Hi Eric,

    Make sure you are attempting to access and process data using an organization is supported.

    For connecting to Office 365, the data movement region determined above should be the same region where the Office365Tenant users' mailbox is located. Users' mailbox that are not located in the computed data movement region are dropped during extraction. In the event that none of the users' mailbox matches the azure data movement region, the pipeline will fail. Below is a mapping between the Office Region and the Azure Regions.

    More details on our region support is located here.

    Hope this helps. Do let us know if you any further queries.

    Friday, March 29, 2019 5:31 AM
    Moderator
  • Understood, thanks! I created a new Storage Account in the proper region and changed the connection in the factory and the run proceeded - up to a point.  When I go to the O365 admin portal to grant consent to the request, it throws a "Object reference not set to an instance of an object" error.  Same if I try to run the consent approval via PowerShell. The request id is: 19b44517-4ee5-4172-b0f0-6c7d439ed4a9

    Eric Legault (MVP: Outlook)
    Try MessageFiler for Outlook!

    Saturday, March 30, 2019 8:20 PM
  • Hi Eric,

    Could you please share the below information:

    1. The request ID which you are trying to approve in the Office 365 Admin portal and PowerShell. I believe that this is 19b44517-4ee5-4172-b0f0-6c7d439ed4a9 but wanted to confirm that it is correct, and that it is the approval request ID and not the copy activity runId.
    2. A timestamp for when you have attempted to approve through PowerShell.
    3. The output from PowerShell during approval. There is occasionally a bit of additional information given in the error output in PowerShell that is helpful with our investigation.

    Please also feel free to reach out to us directly at dataconnect@microsoft.com.

    Tuesday, April 2, 2019 6:14 AM
    Moderator