Fixed length mutli record type file ingestion into ADL


  • Hi all,

    New member, I am a business analyst that wants to start using Azure for analytics.

    Want to start simple first, basically creating files based on filtering raw source files.

    Unfortunately, I have to work not with CSVs nor XML/JSON but with mainframe files, fixed record length, multiple record type files representing sales transactions.

    I initially was told by a Microsoft rep (helping us with a POC) that the above was not possible with Azure. Without knowing any better, what I did was load the file into a SQL Server table and break out each record type into its own table. Then, performing left outer joins on each of the tables using the business key, created denormalized records for each transaction with the specific granularity required that I then extracted to a file and put into the Azure Data Lake. At that point, I was fine using U-SQL to get the filtered results I needed.

    It should not be that difficult to get the data from the raw state into something I could write U-SQL on. My question is: What Azure tool/functionality is available to make short work for "schematizing" these kinds of non-tagged/non-delimited files?

    Thanks and regards,


    Friday, August 10, 2018 6:33 PM

All replies