none
Event Trigger - Error when getting container list RRS feed

  • Question

  • Hi

    I have a working Event Trigger against our test blob storage (regular blob storage v2 for our test environment), but when I try to create a new trigger against out Production blob storage (also v2) I can't list any containers. It just says "Unable to list containers", and when I check the request-log in Chrome Development Tools I see this error:

    1. Request URL:
      https://myblobstorage.blob.core.windows.net/?api-version=2018-03-28&comp=list
    1. 403 CORS not enabled or no matching rule found for this request.

    I continued and created it anyways but the trigger does not work. 

    Any ideas? Is there a setting in azure I've forgot to set?

    Thursday, December 5, 2019 11:18 AM

Answers

  • Additionally of the details already mentioned by Martin, another possible workaround would be to add the data factory domain directly to your blob storage CORS policy, so you will be able to list the containers.

    • Go to your storage account resource, in the left pane, under settings, look for CORS
    • Under blob service tab, in the Allowed origins column, add the following domain: https://adf.azure.com
    • select GET, POST as allowed methods 
    • Under allowed headers put this value: *
    • Under exposed headers put this value: *

    Could you please confirm the scenario that Martin described?

    Are your prod and dev storage configured in the same way and are both the same storage type?

    Is it possible that you have set CORS rules only for your prod storage?

    Wednesday, December 11, 2019 10:11 PM

All replies

  • You might need to add the CORS setting for storage blob service as it is disabled by default.

    More info here.

    • Edited by Zekaryah Thursday, December 5, 2019 11:38 AM
    Thursday, December 5, 2019 11:38 AM
  • Ok, but I don't remember having to do that for my other blob storage. Isn't ADF supposed to work with Storage without additional settings except enabling Microsoft.EventGrid on the subscription?
    Thursday, December 5, 2019 12:38 PM
  • Hello Stranded Software.  Does your storage have Heirarchical namespace enabled (Azure Data Lake gen2), or is it purely BLOB?
    Friday, December 6, 2019 11:40 PM
    Moderator
  • Hi Martin,

    We don't use Data Lake and I haven't done anything with Hierachial namespaces so I guess it's purely BLOB.

    Monday, December 9, 2019 11:08 AM
  • Stranded Software, I have some questions:

    • Are you still blocked, or are you no longer facing the issue?
    • Do you have any pipelines in that Factory using the same Linked Service, and functioning correctly?
    • Is the storage account behind a firewall or in a Vnet?
    • Have you migrated your factory or done CI/CD?

    Monday, December 9, 2019 10:13 PM
    Moderator
    • Are you still blocked, or are you no longer facing the issue?

    Still blocked

    • Do you have any pipelines in that Factory using the same Linked Service, and functioning correctly?

    No. I have these: 

    1. Azure SQL DB for test environment
    2. Azure Blob Storage for test environment
    3. Azure SQL DB for prod environment
    4. Azure Blob Storage for prod environment

    None is used twice

    • Is the storage account behind a firewall or in a Vnet?

    No firewall rules i.e. Allow access from all networks

    • Have you migrated your factory or done CI/CD?

    No, everything is from scratch and no Git or anything is being used at the moment

    When I test the connection to my Azure Blob Storage for prod it's successful. The problem only occurs when trying to create a Blob Created event trigger. Everything works flawlessly with my test enironment.

    And for some more info: I have 2 pipelines, one for test and one for prod. 

    Also I can trigger prod manually which works fine.

    Edit: The Blob storages are used with CSV 
    Tuesday, December 10, 2019 11:58 AM
  • Thank you for the reply.  So the blob trigger has no issues with test, only issues with prod.  To allow you to move forward with your development, I do have a work-around.  This will not solve the underlying issue, but will allow you to continue development.

    If you know the container name, you can manually input it via editing the JSON code.  In the UX go to the broken trigger you created and look for a button with "{ }".  This should bring up the trigger definition.  It should look something like:

    {
        "name": "trigger1",
        "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [],
            "type": "BlobEventsTrigger",
            "typeProperties": {
                "blobPathBeginsWith": "/My_Container_Name/blobs/My_Folder_Name",
                "ignoreEmptyBlobs": true,
                "scope": "/subscriptions/My_Subscription_Id/resourceGroups/My_Resource_Group_Name/providers/Microsoft.Storage/storageAccounts/My_Storage_Account_Name",
                "events": [
                    "Microsoft.Storage.BlobCreated"
                ]
            }
        }
    }

    Wednesday, December 11, 2019 1:57 AM
    Moderator
  • Is it weird that it's already there? 

    {
        "name": "ProdTrigger",
        "properties": {
            "annotations": [],
            "runtimeState": "Started",
            "pipelines": [
                {
                    "pipelineReference": {
                        "referenceName": "My_Pipline",
                        "type": "PipelineReference"
                    },
                    "parameters": {
                        "PipelineCID": "@triggerBody().folderPath",
                        "PipelineFilename": "@triggerBody().fileName"
                    }
                }
            ],
            "type": "BlobEventsTrigger",
            "typeProperties": {
                "blobPathBeginsWith": "/my_container/blobs/",
                "ignoreEmptyBlobs": true,
                "scope": "/subscriptions/My_Subscription_Id/resourceGroups/My_Resource_Group/providers/Microsoft.Storage/storageAccounts/My_Storage_Account_Name",
                "events": [
                    "Microsoft.Storage.BlobCreated"
                ]
            }
        }
    }

    I changed sensitive data to your example

    It looks exactly like my TestTrigger except for "My_Pipeline" and "My_Storage_Account_Name"

    Wednesday, December 11, 2019 1:29 PM
  • It is not strange that the code is already there.  What is strange, is that it already has populated "my_container" (I assume from your example, it is populated with a valid container name) when you were unable to input a container in the GUI.  If everything is present and you still cannot enable the trigger, that is even more strange.

    If that is the case, I think a support ticket may be able to help you better.  I could talk back and fourth with you and speculate about possible causes, but from here I am unable to give a definitive answer.  Do you assert that both storage accounts are identical (except for data)?  Same subscription, same region/datacenter, same permissions, same settings.

    For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter. 
    Thread URL:
    Subscription ID: 
    Please let me know once you have done the same

    Wednesday, December 11, 2019 9:52 PM
    Moderator
  • Additionally of the details already mentioned by Martin, another possible workaround would be to add the data factory domain directly to your blob storage CORS policy, so you will be able to list the containers.

    • Go to your storage account resource, in the left pane, under settings, look for CORS
    • Under blob service tab, in the Allowed origins column, add the following domain: https://adf.azure.com
    • select GET, POST as allowed methods 
    • Under allowed headers put this value: *
    • Under exposed headers put this value: *

    Could you please confirm the scenario that Martin described?

    Are your prod and dev storage configured in the same way and are both the same storage type?

    Is it possible that you have set CORS rules only for your prod storage?

    Wednesday, December 11, 2019 10:11 PM
  • Hi Cesar,

    Thank you that did the trick. I know Zekaryah said it at the beginning, but it felt weird doing an extra step for something that already worked for my test env but I didn't wanna spend anymore time on this. 

    Also thank you MartinJaffer-MSFT for all your help. I learned a few new things on the way.

    Thursday, December 12, 2019 7:50 AM
  • Thank you both.
    Thursday, December 12, 2019 6:38 PM
    Moderator
  • Hi Stranded Software,

    Glad to know that it worked! As you mentioned earlier, you should not be required to add this setting, could you please help us providing the following information to diagnose better the issue in our end?

    Are your prod and dev storage configured exactly in the same way and are both the same storage type?

    What is the storage type of your storage accounts? (storageV2 (general purpose), BlobStorage, standard or premium)

    Thanks for your help :D !

    Thursday, December 12, 2019 7:27 PM