locked
Data Factory - Copy from REST to Table Storage - PartitionKey null RRS feed

  • Question

  • Good day,

    We are importing the following JSON format via REST in Data Factory v2:

    [
        {
            "ERROR": false, 
            "USERNAME": "DUMMY"
        }, 
        [
            {
                "A": 1, 
                "B": 2, 
                "C": 3, 
            }, 
            {
                "A": 1, 
                "B": 0, 
                "C": 3, 
            }   

         ]
    ]

    Partition key value: USE SOURCE COLUMN

    Partition key column: C

    Mapping is properly se for A B and C. ERROR and USERNAME are -- Not Included --.

    Gives the following error in debug:

    "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorAzureTableKeyColumnWithNullValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'C' contains 'NULL' value thus cannot be used as 'PartitionKey' for Azure Table.

    I understand why the error happens, the COPY DATA tool tries to create three rows:

    First row is based on ERROR and USERNAME. This row has indeed value null for C

    Second row is based on A, B and C. This row is correct as there are no null values for C.

    Third row is based on A, B and C. This row is correct as there are no null values for C.

    We know above because it works with fixed Partition Key value (for example value: test)

    We can't change the format of the input. What can we do in Data Factory to solve this issue? How can we disregard first array with ERROR and USER, or how can we skip rows with Partition Key if null?



    • Edited by SinbadAZURE Saturday, January 12, 2019 4:51 PM
    Saturday, January 12, 2019 4:49 PM

All replies

  • Hi,

    Have you tried using a custom activity ? You can use it to parse your REST API response and format it before storing it to table storage. Here's the doc on custom activities :

    https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-use-custom-activities

    Let us know if this helps. Else we can gladly continue the dialogue.


    MSDN

    Monday, January 14, 2019 6:33 AM