none
Connecting Azure Data Factory to Influxdb

    Question

  • I'm working on some time series data that I want migrated to cloud. Working in Australia and PIE is stopping me from using Time Series Insights, so I've decided to use InfluxDb as my Time Series database. I've set up a Grafana VM on Azure and installed InfluxDB in it. The task where I'm stuck is.

    1. Import a csv file (with time series data) to blob storage using Azure Data Factory (Have done this) 

    2. Use ADF to transfer the files to InfluxDb. (Need help here) 

    3. Do cool stuff on the data (have nice people in the team who're experts in this task)

    Need help in point 2. Appreciate the help.

    Thanks

    Monday, June 18, 2018 2:35 AM

All replies

  • Hello,

    I understand you were able to get your data into the CSV form and onto blob storage.  If you could get that CSV file onto a file share which your VM has access to, you could use influx CLI -import to import the data:

    https://docs.influxdata.com/influxdb/v1.2/tools/shell/#import-data-from-a-file-with-import

    This document outlines how to move data into a file system:

    https://docs.microsoft.com/en-us/azure/data-factory/connector-file-system

    The file system you are moving the data to can be Azure Files in a Storage Account, or your own file server.  Assuming your VM is connected to the file server/file system, and the data has been copied to the file server, you could then run the influx -import command.  

    As far as the triggering of the influx -import command, you could set up a cron job to run it on an interval, but staying with ADF you might use a custom activity:

    https://docs.microsoft.com/en-us/azure/data-factory/transform-data-using-dotnet-custom-activity

    The custom activity runs in Azure Batch, which you might use to send an SSH command to your VM to execute the influx -import.  I am not aware of any limitations Azure Batch has in terms of networking that would block SSH.  Here is more information on the firewall configuration of Azure Batch:

    https://docs.microsoft.com/en-us/azure/batch/batch-api-basics#virtual-network-vnet-and-firewall-configuration


    Wednesday, June 20, 2018 5:43 PM
    Moderator