none
Consuming result stored in Azure Data Lake Analytics tables

    Question

  • I tried googling and searching the forum but couldn't find anything related. To evaluate ADLA, I'm putting together a PoC of our workflow. The plan is to have ADLA process .csv stored in ADLS, process it with ADLA and store the result in ADLA tables, which our API will query against. From the fact that ADLA tables have partitions and index, it seems to be a viable big data store. My questions are:

    1. Is there any size limit on ADLA tables? They seems to fall under the analytics section not the data lake store. If the tables are actually stored in ADLS then I presume there's no limit to the size of table? As long as we have a good partition scheme the retrieval should be fast.

    2. Assuming #1 is true, is the cost just storage cost in ADLS?

    3. Last but perhaps most important, is this use of ADLA table recommended? Specifically, have the table service REST API requests? (I know this depends on our partition scheme and query). If not, what would be the best practice on saving computed results?

    Thanks,

    Richard


    Wednesday, October 12, 2016 10:56 PM

Answers

  • Dear Richard

    To answer your questions:

    1. There is no physical size limit to the size of the table. There are row size limits (4MB) and there are limits on data types. These are 128kB for string and 4MB (or whatever is available in the row) for byte[] and the SqlMap and SqlArray types.

    And yes the data is stored in ADLS files (inside the /catalog directory).

    2. Yes.

    3. There is currently no ODBC-like or REST access to the table content. They are currently mainly suggested to speed-up your U-SQL queries. If you want to return data, you will have to extract the result of the query into a file that you then download. Interactive access of the form "SELECT * FROM table" from APIs is on the roadmap and being worked on. 


    Michael Rys

    Wednesday, October 19, 2016 11:02 PM
    Moderator

All replies

  • Dear Richard

    To answer your questions:

    1. There is no physical size limit to the size of the table. There are row size limits (4MB) and there are limits on data types. These are 128kB for string and 4MB (or whatever is available in the row) for byte[] and the SqlMap and SqlArray types.

    And yes the data is stored in ADLS files (inside the /catalog directory).

    2. Yes.

    3. There is currently no ODBC-like or REST access to the table content. They are currently mainly suggested to speed-up your U-SQL queries. If you want to return data, you will have to extract the result of the query into a file that you then download. Interactive access of the form "SELECT * FROM table" from APIs is on the roadmap and being worked on. 


    Michael Rys

    Wednesday, October 19, 2016 11:02 PM
    Moderator
  • Has there been any progress on this? 

    "Interactive access of the form "SELECT * FROM table" from APIs is on the roadmap and being worked on. "


    Kyle Clubb

    Thursday, February 1, 2018 6:12 PM