none
Fixed length mutli record type file ingestion into ADL

    Question

  • Hi all,

    New member, I am a business analyst that wants to start using Azure for analytics.

    Want to start simple first, basically creating files based on filtering raw source files.

    Unfortunately, I have to work not with CSVs nor XML/JSON but with mainframe files, fixed record length, multiple record type files representing sales transactions.

    I initially was told by a Microsoft rep (helping us with a POC) that the above was not possible with Azure. Without knowing any better, what I did was load the file into a SQL Server table and break out each record type into its own table. Then, performing left outer joins on each of the tables using the business key, created denormalized records for each transaction with the specific granularity required that I then extracted to a file and put into the Azure Data Lake. At that point, I was fine using U-SQL to get the filtered results I needed.

    It should not be that difficult to get the data from the raw state into something I could write U-SQL on. My question is: What Azure tool/functionality is available to make short work for "schematizing" these kinds of non-tagged/non-delimited files?

    Thanks and regards,

    Paul

    Friday, August 10, 2018 6:33 PM

All replies