locked
Azure Data Explorer Sink - NetDefaultDB permissions RRS feed

  • Question

  • I'm trying to use the Copy Data activity with an Azure Data Explorer (ADX) table as a sink.

    I'm getting this error:

    { "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorKustoNameResolutionFailure,

    'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,

    Message=Failed to resolve Kusto service name: No queues were returned from Kusto endpoint.

    Error: 'An exception was thrown while attempting to retrieve queues from endpoint: 'Data Source=https://ingest-XXXXXX.westus2.kusto.windows.net;Initial Catalog=NetDefaultDB;Application Client Id=XXXXXX-XXXX-XXXX-9b47-9d2cab0022bd;Application Key=****;AAD Federated Security=True;Authority Id=XXXX-XXX-XXX-847f-958f3d479f4a'.

    Error: 'Forbidden (403-Forbidden): {\r\n \"error\": {\r\n \"code\": \"Forbidden\",\r\n \"message\": \"Caller is not authorized to perform this action\",\r\n \"@type\": \"Kusto.Common.Svc.Exceptions.UnauthorizedOperationException\",\r\n \"@message\": \"Principal 'aadapp=x-xxxx-4174-9b47-9d2cab0022bd;xxxxx-a769-4654-847f-958f3d479f4a' is not authorized to perform operation 'Access the Kusto DM service for Ingest on any database' on '[unspecified target]'.\",\r\n \"@context\": {\r\n \"timestamp\": \"2019-11-21T17:16:14.7366942Z\",\r\n \"serviceAlias\": \"INGEST-NPPADXPOC\",\r\n \"machineName\": \"KDataMana000000\",\r\n \"processName\": \"Kusto.WinSvc.DM.Svc\",\r\n \"processId\": 4780,\r\n \"threadId\": 2052,\r\n \"appDomainName\": \"Kusto.WinSvc.DM.Svc.exe\",\r\n \"clientRequestId\": \"KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea\",\r\n \"activityId\": \"e6a813de-8690-4477-8e56-3fbc447d3f26\",\r\n \"subActivityId\": \"d278766a-9088-4032-98de-92466a89a347\",\r\n \"activityType\": \"P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract\",\r\n \"parentActivityId\": \"b9d41de5-7166-4b71-a469-f651c5f87720\",\r\n \"activityStack\": \"(Activity stack: CRID=KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea ARID=e6a813de-8690-4477-8e56-3fbc447d3f26 > DN.Admin.Client.ExecuteControlCommand/b9d41de5-7166-4b71-a469-f651c5f87720 > P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract/d278766a-9088-4032-98de-92466a89a347)\"\r\n },\r\n \"@permanent\": true\r\n }\r\n}. This normally represents a permanent error, and retrying is unlikely to help.\r\nPlease provide the following information when contacting the Kusto team @ https://aka.ms/kustosupport :\r\nDataSource='https://ingest-nppadxpoc.westus2.kusto.windows.net/v1/rest/mgmt',\r\nDatabaseName='NetDefaultDB',\r\nClientRequestId='KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea',\r\nActivityId='e6a813de-8690-4477-8e56-3fbc447d3f26,\r\nTimestamp='2019-11-21T17:16:14.7881569Z'.''.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.CloudQueuesNotFoundException,Message=No queues were returned from Kusto endpoint. Error: 'An exception was thrown while attempting to retrieve queues from endpoint: 'Data Source=https://ingest-nppadxpoc.westus2.kusto.windows.net;Initial Catalog=NetDefaultDB;Application Client Id=xxxxx-xxxx-xxxx-9b47-9d2cab0022bd;Application Key=****;AAD Federated Security=True;Authority Id=xxxxx-xxxx-4654-847f-958f3d479f4a'. Error: 'Forbidden (403-Forbidden): {\r\n \"error\": {\r\n \"code\": \"Forbidden\",\r\n \"message\": \"Caller is not authorized to perform this action\",\r\n \"@type\": \"Kusto.Common.Svc.Exceptions.UnauthorizedOperationException\",\r\n \"@message\": \"Principal 'aadapp=xxxxxx-xxxx-xxxx-9b47-9d2cab0022bd;12a3af23-a769-4654-847f-958f3d479f4a' is not authorized to perform operation 'Access the Kusto DM service for Ingest on any database' on '[unspecified target]'.\",\r\n \"@context\": {\r\n \"timestamp\": \"2019-11-21T17:16:14.7366942Z\",\r\n \"serviceAlias\": \"INGEST-NPPADXPOC\",\r\n \"machineName\": \"KDataMana000000\",\r\n \"processName\": \"Kusto.WinSvc.DM.Svc\",\r\n \"processId\": 4780,\r\n \"threadId\": 2052,\r\n \"appDomainName\": \"Kusto.WinSvc.DM.Svc.exe\",\r\n \"clientRequestId\": \"KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea\",\r\n \"activityId\": \"e6a813de-8690-4477-8e56-3fbc447d3f26\",\r\n \"subActivityId\": \"d278766a-9088-4032-98de-92466a89a347\",\r\n \"activityType\": \"P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract\",\r\n \"parentActivityId\": \"b9d41de5-7166-4b71-a469-f651c5f87720\",\r\n \"activityStack\": \"(Activity stack: CRID=KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea ARID=e6a813de-8690-4477-8e56-3fbc447d3f26 > DN.Admin.Client.ExecuteControlCommand/b9d41de5-7166-4b71-a469-f651c5f87720 > P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract/d278766a-9088-4032-98de-92466a89a347)\"\r\n },\r\n \"@permanent\": true\r\n }\r\n}. This normally represents a permanent error, and retrying is unlikely to help.\r\nPlease provide the following information when contacting the Kusto team @ https://aka.ms/kustosupport :\r\nDataSource='https://ingest-nppadxpoc.westus2.kusto.windows.net/v1/rest/mgmt',\r\nDatabaseName='NetDefaultDB',\r\nClientRequestId='KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea',\r\nActivityId='e6a813de-8690-4477-8e56-3fbc447d3f26,\r\nTimestamp='2019-11-21T17:16:14.7881569Z'.'',Source=Kusto.Ingest,''Type=Kusto.Data.Exceptions.KustoRequestDeniedException,Message=Forbidden (403-Forbidden): {\r\n \"error\": {\r\n \"code\": \"Forbidden\",\r\n \"message\": \"Caller is not authorized to perform this action\",\r\n \"@type\": \"Kusto.Common.Svc.Exceptions.UnauthorizedOperationException\",\r\n \"@message\": \"Principal 'aadapp=xxxxx-xxxx-xxxx-9b47-9d2cab0022bd;xxxxx-xxxx-xxxx-847f-958f3d479f4a' is not authorized to perform operation 'Access the Kusto DM service for Ingest on any database' on '[unspecified target]'.\",\r\n \"@context\": {\r\n \"timestamp\": \"2019-11-21T17:16:14.7366942Z\",\r\n \"serviceAlias\": \"INGEST-NPPADXPOC\",\r\n \"machineName\": \"KDataMana000000\",\r\n \"processName\": \"Kusto.WinSvc.DM.Svc\",\r\n \"processId\": 4780,\r\n \"threadId\": 2052,\r\n \"appDomainName\": \"Kusto.WinSvc.DM.Svc.exe\",\r\n \"clientRequestId\": \"KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea\",\r\n \"activityId\": \"e6a813de-8690-4477-8e56-3fbc447d3f26\",\r\n \"subActivityId\": \"d278766a-9088-4032-98de-92466a89a347\",\r\n \"activityType\": \"P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract\",\r\n \"parentActivityId\": \"b9d41de5-7166-4b71-a469-f651c5f87720\",\r\n \"activityStack\": \"(Activity stack: CRID=KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea ARID=e6a813de-8690-4477-8e56-3fbc447d3f26 > DN.Admin.Client.ExecuteControlCommand/b9d41de5-7166-4b71-a469-f651c5f87720 > P.WCF.Service.ExecuteControlCommandInternal..IAdminClientServiceCommunicationContract/d278766a-9088-4032-98de-92466a89a347)\"\r\n },\r\n \"@permanent\": true\r\n }\r\n}. This normally represents a permanent error, and retrying is unlikely to help.\r\nPlease provide the following information when contacting the Kusto team @ https://aka.ms/kustosupport :\r\nDataSource='https://ingest-nppadxpoc.westus2.kusto.windows.net/v1/rest/mgmt',\r\nDatabaseName='NetDefaultDB',\r\nClientRequestId='KI.KustoQueuedIngestClient.IngestFromDataReader.ef8cb540-1476-46a4-ba7a-23bfc6d0d00e;ec592e1a-6dee-409b-af35-942944db3aea',\r\nActivityId='e6a813de-8690-4477-8e56-3fbc447d3f26,\r\nTimestamp='2019-11-21T17:16:14.7881569Z'.,Source=Kusto.Data,'", "failureType": "UserError", "target": "Copy pets2 pipe to adx" }

    I have granted Database Ingestor permissions to the App ID for the database. I'm getting permission denied in the call and I think it's because it is not specifying the database, it's using NetDefaultDB in the data source connection string. That's bmy guess based on the messages in the error. 

    The sink dataset specifies the database and the table name. The ADX Connection also specifies the database. 

    How do I get the pipeline Copy activity to use the ADX database name for its call to the Ingestion endpoint? Or, how do I set the default database for an App ID so that NetDefaultDB uses the correct database context for permission evaluation?

    Mike

    Thursday, November 21, 2019 6:04 PM

All replies

  • Can you please share the runid ?

    Thanks Himanshu

    Monday, November 25, 2019 11:05 PM
  • Hi Mike

    I was able to repro the issue and I must tell you that I was expecting the database name in the drop down , but it never showed up . I did just typed in my database name and it worked  ( I had to do the same with the Table Name ) .

    For the column mapping i had to use the mapping something like .

    "translator": {
    "type": "TabularTranslator",
    "mappings": [{
    "source": {
    "name": "Prop_0",
    "type": "Int32"
    },
    "sink": {
    "name": "Someint"
    }
    }]

    Also in my case the SP in having the contributor role on the ADX . 

    Please do let me know how it goes , since I was able to repro the error and make the pipeline work , i am sure you are very close .


    Thanks Himanshu

    Tuesday, November 26, 2019 3:19 AM