locked
What's the best way to get data imports into production tables? RRS feed

  • Question

  • User-1335903275 posted

    If you were tasked with creating (or rather, I should say, fixing) a solution to importing customer data from a 3rd party api, into a sql server database, how would you go about it?  Just looking for a general overview.  Here's why I ask...

    I'm doing some side work for a small company that's trying to develop a dashboard website for their customers.  They have a part time "db admin" (really just a .net developer but he's doing the best he can) who's setup an automated process to import the customer data from a 3rd party api (Mindfire) and imports it into their database.  They have only one database.  No separation of dev, qa & prod.  

    So the problem is, yesterday, something went wrong with their data import and instead of updating customer data, it somehow removed ALL customer data from the database.  Which led me to realize they have a big problem on their hands if data is going to get zapped anytime their data import goes down.  So I want to take this opportunity to suggest some fixes,& curious on your thoughts of the best way of going about this.  

    A few quick things:

    • I don't want to remove or completely replace the other developers existing automated import process.  I can make some recommendations for him to fix things but I don't want to go in and reinvent the wheel here, when it comes to his process.
    • Again, the data currently goes from Mindfire > data import process > database tables (production).  There are no dev or staging tables setup yet

    Any recommendations?

    Thanks

    Friday, September 15, 2017 1:10 PM

All replies

  • User77042963 posted

    If they are using ssis package now, it is time to review the code to make sure everything is working correctly.

    Friday, September 15, 2017 1:38 PM
  • User347430248 posted

    Hi Goalie35,

    As you said,

    • I don't want to remove or completely replace the other developers existing automated import process.  I can make some recommendations for him to fix things but I don't want to go in and reinvent the wheel here, when it comes to his process.
    • Again, the data currently goes from Mindfire > data import process > database tables (production).  There are no dev or staging tables setup yet

    here, we are not aware of how Mindfire Api work and imports the data.

    so you can try to refer the documentation provided by them to come to know about the exact process to import data correctly.

    I try to find the documentation on their website but unfortunately did not get anything useful in that direction.

    if we talk generally and want to import data without using Mindfire then you can try to refer link below.

    Import and Export Data with the SQL Server Import and Export Wizard

    Regards

    Deepak

    Monday, September 18, 2017 6:26 AM