none
SCD using U-SQL Tables

Answers

  • In theory yes you can. But ask yourself the following questions:

    1. How are you going to use the U-SQL tables. Right now the benefit of the U-SQL tables lays in the ability to have better performance in batch processing since the data in the tables are guaranteed to be organized according to the chosen distribution, partitioning and clustering schemes. If you need interactive access, the blog post referenced above may be better.
    2. How frequent and big are the updates? Are they additive? Are you expiring data? This will have an impact on your need to manage the table rebuild.
    3. How are you planning on querying the data. This will impact how you handle versioning of the slow changing dimensions (e.g., are you using a join on the validity range etc). 

    Note that U-SQL gives you Windowing functions and the ability to write your own custom reducers and combiners. So that gives you some additional capabilities to express some of the temporal semantics.


    Michael Rys

    Monday, July 2, 2018 10:54 PM
    Moderator

All replies

  • What type of SCD are you looking for? There is an example of implementing SCD type 2 on this link.
    Monday, July 2, 2018 6:21 PM
    Moderator
  • Hi,

    My question is can I use U-SQL table for SCD(SCD 2). Please let me know. The link you have provided is not having the answer I am looking for, it is using SQL DB.

    Thanks,

    Soumitra. 

    Monday, July 2, 2018 9:07 PM
  • In theory yes you can. But ask yourself the following questions:

    1. How are you going to use the U-SQL tables. Right now the benefit of the U-SQL tables lays in the ability to have better performance in batch processing since the data in the tables are guaranteed to be organized according to the chosen distribution, partitioning and clustering schemes. If you need interactive access, the blog post referenced above may be better.
    2. How frequent and big are the updates? Are they additive? Are you expiring data? This will have an impact on your need to manage the table rebuild.
    3. How are you planning on querying the data. This will impact how you handle versioning of the slow changing dimensions (e.g., are you using a join on the validity range etc). 

    Note that U-SQL gives you Windowing functions and the ability to write your own custom reducers and combiners. So that gives you some additional capabilities to express some of the temporal semantics.


    Michael Rys

    Monday, July 2, 2018 10:54 PM
    Moderator
  • Thank you !
    Tuesday, July 3, 2018 1:21 AM