none
Using pandas.concat() in python extension for U-SQL RRS feed

  • Question

  • Hello all,

    I'm trying to learn the use of Python Extension for U-SQL as shown in this blog post: https://blogs.msdn.microsoft.com/azuredatalake/2016/11/22/u-sql-advanced-analytics-introducing-python-extensions-for-u-sql/

    I thought about starting with something simple and as a test, whether I can implement the following python code in u-sql:

    import pandas as pd from pandas import DataFrame def append_data(df_a, df_b): frames = [df_a, df_b] df = pd.concat(frames, axis = 0, ignore_index = True) return df x = DataFrame({"greeks": ["alpha", "beta","gamma"], "random_values":[1234,5678,9012]}) y = x.copy()

    append_data(x,y)

    So basically, here I plan to take two DataFrames and then join them from below. However, this is as far as I could come:

    REFERENCE ASSEMBLY [ExtPython];
    
    DECLARE @pandasDF = @"
    import pandas as pd
    from pandas import DataFrame
    
    def append_data(df_a, df_b):
        frames = [df_a, df_b]
        df = pd.concat(frames, axis = 0, ignore_index = True)
        return df
    ";
    
    @x =  
        SELECT * FROM
                     (
                        VALUES
                                ("delta", 1234),
                                ("gamma", 4567),
                                ("vega", 8901),
                                ("theta", 2345)
                     )  AS
                           D(greeks, random_values);
    
    @y = 
        SELECT * FROM
                     (
                        VALUES
                                ("delta", 1234),
                                ("gamma", 4567),
                                ("vega", 8901),
                                ("theta", 2345)
                     )  AS
                           D(greeks, random_values); 
    

    How do I proceed from this point? I assume for this case the whole thing can be done easily using UNION. But how can I accomplish it using the python extension. Any advice and suggestions will be greatly appreciated. 

    Thanks 

    Wednesday, February 1, 2017 12:48 PM