checking the correlation of data

Question

Hi,

For more than a year I am gathering different data on my health: habits, some food, medications, some other lifestyle data.

I would like input them all to some system and check the correlations between elements of gathered data. This data currently are in different applications, but I could probably get them into some reasonable format. The data is only text, probably few thousands of records altogether?

Therefore I have a question: what service should I use for such analytics? Data Lake Analytics? I didnt use any of those before.  Maybe some other service?

Thx.

Loonak

Answer 1

Hi Loonak,

Firstly, I would assess if the data you have needs to be enriched by adding additional property values to help with the correlation analysis. Or is the data set fairly complete and you just need a means to look through the data? Where does this data reside currently?

Since you posted this forum question in the Azure Data Explorer forum, I assume this data resides in an Azure Data Explorer cluster. If not, you can load the data to Azure Data Lake via Azure Storage and query the data using the Azure Data Explorer. Please see: Query data in Azure Data Lake using Azure Data Explorer (link).

This option would allow you the ability to groom and organize the data to further build the correlation for analysis.

Regards,

Mike

Answer 2

Hi Loonak, 

I worked with Data Lakes Analytics (USQL) and Data Explorer. For your purpose, I would choose Data Explorer and the Kusto time series offering. 

With Kusto, you can calculate co-variances across all of your properties. If it is not enough, you can also use Python and pull data from Kusto. 

However, as you have very few lines, you could load the data in a CSV and use Python on your PC at no cost. Data Explorer really shines when you have million/billion rows.

Have a great day, 

Benoit