locked
Working with CSV File for Web Application RRS feed

  • Question

  • User1122355199 posted

    Hello everyone and thanks for your help in advance.  I am developing a web application that extracts a CSV file from an IOT device.  The device has no built-in function to push data when a change to the CSV file occurs.  So my thinking is to use some type of timer function to poll the device periodically and pull the CSV file from the device.  Not the most efficient, but the only game available.  However, I need to develop a strategy to parse the CSV file to add the new data to a SQL database and I'm relly not sure of the most efficient way to handle this.  Is there some way to compare the previous and current CSV or do I have to pull everything into the database and go from there?  Any insight would be appreciated.

    Wednesday, February 20, 2019 10:57 PM

Answers

  • User303363814 posted

    I find that https://www.codeproject.com/Articles/25133/LINQ-to-CSV-library is an easy to use library that handles all my csv reading requirements.

    CSVHelper on Nuget also seems to be very popular.  https://www.nuget.org/packages/CsvHelper/  Get Started is at https://joshclose.github.io/CsvHelper/getting-started

    There are literally dozens of packages written by people that you can choose from.  Don't write your own, it is very unlikely that you will handle delimiters correctly.  Make use of the hard work that someonelse has done.

    As far as comparing between versions of the CSV file - I find it easiest to just keep the previous file on disk and compare the new one to it.  Simple and effective.  Read the new file into a string, read the saved file into a string.  Use == to see if they are the same...

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, February 21, 2019 9:55 PM

All replies

  • User475983607 posted

    It's difficult to answer this question as we have no idea how to determine new records or what fields make a record.  I generally use a work table to load bulk data then use SQL to move the data to the final table.

    Wednesday, February 20, 2019 11:30 PM
  • User303363814 posted

    I find that https://www.codeproject.com/Articles/25133/LINQ-to-CSV-library is an easy to use library that handles all my csv reading requirements.

    CSVHelper on Nuget also seems to be very popular.  https://www.nuget.org/packages/CsvHelper/  Get Started is at https://joshclose.github.io/CsvHelper/getting-started

    There are literally dozens of packages written by people that you can choose from.  Don't write your own, it is very unlikely that you will handle delimiters correctly.  Make use of the hard work that someonelse has done.

    As far as comparing between versions of the CSV file - I find it easiest to just keep the previous file on disk and compare the new one to it.  Simple and effective.  Read the new file into a string, read the saved file into a string.  Use == to see if they are the same...

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, February 21, 2019 9:55 PM
  • User1122355199 posted

    Boy, is this a great answer!  Thanks so much for the help.  I didn't know about these libraries, but they look like a great solution.  Love the solution of comparing the two files.  Great idea.  Thanks again.

    Thursday, February 21, 2019 11:32 PM