locked
Generate time dataframe with pyspark Azure RRS feed

  • Question

  • Hello,

      I am new in Pyspark, and i need hlep please. There are my subject : I need to create a dataframe with six columns :

    Column 1 = date

    Column 2 = day

    Column 3 = month

    Column 4 = quarter

    Column 5 = Year

    And i need this dataframe starting the current date.

    So how i can do this please?

    Best Regards,

    Adghar


    Reponse

    Wednesday, July 12, 2017 10:02 AM

Answers

  • The below code should get you started with deriving those columns from today's date:

    from datetime import *
    from pyspark.sql.functions import *

    df0 =spark.createDataFrame([(datetime.today(),)], ['t'])
    df1 = select(to_date('t').alias('date'), dayofmonth('t').alias('day'), month('t').alias('month'), quarter('t').alias('quarter'), year('t').alias('year'))

    +----------+---+-----+-------+----+
    |      date|day|month|quarter|year|
    +----------+---+-----+-------+----+
    |2017-07-13| 13|    7|      3|2017|
    +----------+---+-----+-------+----+

    Thursday, July 13, 2017 5:51 PM

All replies

  • Could you elaborate your query bit more to understand better?

    Are you trying to select multiple columns given in a sequence of column names?

    -----------------------------------------------------------------------------------------------
    Do click on "Mark as Answer" on the post that helps you, this can be beneficial to other community members.

    Wednesday, July 12, 2017 4:19 PM
  • hello,

      I' ma trying to generate a new time dataframe. 

    regards,


    Reponse

    Thursday, July 13, 2017 12:03 PM
  • The below code should get you started with deriving those columns from today's date:

    from datetime import *
    from pyspark.sql.functions import *

    df0 =spark.createDataFrame([(datetime.today(),)], ['t'])
    df1 = select(to_date('t').alias('date'), dayofmonth('t').alias('day'), month('t').alias('month'), quarter('t').alias('quarter'), year('t').alias('year'))

    +----------+---+-----+-------+----+
    |      date|day|month|quarter|year|
    +----------+---+-----+-------+----+
    |2017-07-13| 13|    7|      3|2017|
    +----------+---+-----+-------+----+

    Thursday, July 13, 2017 5:51 PM