How to refresh data in a Data Warehouse? RRS feed

  • Question

  • I have some data in DW but I need to upload some new generated data. How can I upload only new generated data and also the data that has changed? Can I do row comparisons, or do I have to load duplicate data each time I do ETL? Thanks
    Thursday, July 11, 2019 1:20 PM

All replies

  • You can use the MERGE statement to do that. The DW tables are the target and the generated data are the source.

    A Fan of SSIS, SSRS and SSAS

    Thursday, July 11, 2019 1:57 PM
  • You can either:

    A) Truncate the table, load the entire table again

    B) Make a comparison and load only the new data

    C) Load the new data and eliminate the duplicated

    The A is very good for tiny table (max  5 millions rows), the B works with aggregate table or complex scenarios, the C can help with huge table but rather than this solution we would adopt some cool SSIS schema.

    Please mark as answer if this post helped you

    Thursday, July 11, 2019 1:59 PM