Incremental fact update concept

Question

Hi,

My source table is Transactions, which has ID, Value and Date, It reflects history of transactions.
My fact holds transactions time frame, each row has ID, Value, StartDate and EndDate.

The incremental process takes the delta transactions.

In first run Transactions table is:
Transactions: ID: 1, Value: 500, Date: 2012-08-01
The incremental process takes this single row to staging and copy to fact:
Fact: ID: 1, Value: 500, StartDate: 2012-08-01, EndDate: NULL

In second run Transactions table is:
Transactions: ID: 1, Value: 500, Date: 2012-08-01
Transactions: ID: 1, Value: 600, Date: 2012-08-02

The incremental process takes the second row to staging and copy this row to the fact:
Fact: ID: 1, Value: 500, StartDate: 2012-08-01, EndDate: NULL
Fact: ID: 1, Value: 600, StartDate: 2012-08-02, EndDate: NULL

BUT the EndDate of the first row should be update as well:
Fact: ID: 1, Value: 500, StartDate: 2012-08-01, EndDate: 2012-08-02
Fact: ID: 1, Value: 600, StartDate: 2012-08-02, EndDate: NULL

Now, the question is:
Should all FACT retro change rows (those records that exist in FACT but not in the incremental delta records but still needs to get update in FACT) get into STAGING table first (extract), get update (transform), and then to return to FACT (load)? If yes - should it get from source table or from FACT table?
Should this update should be done directly on the FACT (update previous row EndDate with next row StartDate)?

Thanks in advance!

Answer 1

Follow these steps,

• Maintain one entry called LoadDate in configuration table, this will store last modified date (max) of your record
• Read data from source based on the modified date, like Date>LoadDate
• Read records will come to your staging
• Perform transformation to the delta records
• Load it to the destination
• If the same record comes once again update the same with new date

Hope I have answered your question.

Regards,

YB