Requirement:
We have large number (around 500000) of small zip files (file size 200MB) stored in Azure Blob Storage. We need to extract xml file and process it, ignoring metadata files using HDinsight Spark Cluster. From HDinsight Spark Cluster we need to store the results
into Azure SQL Database.
Problem:
- To extract zip files which is better approach - Extract xml file and store it into some storage using Data Factory e.g. xml file or SQL database or flat file - Use Spark to extract xml from zip file do further processing in one go
- We are using using a blob storage to store dependency jar files and refer via classpath. Is it better to create a fat Jar?
-----------------------------------------------------------------------------------------------
Do click on "Mark as Answer" and “Vote as Helpful” on the post that helps you, this can be beneficial to other community members.
