Utility Code which can help in conversion of Pandas code to Pyspark

[TecSachinGupta]

Hi,

I have created a sample code for one of the issue and while doing so I created some functions which can help in converting the pandas dataframe to pyspark dataframe etc.

Attaching the code and sample notebook.

def get_pyspark_compatible_dict(data):
    new_data = []
    keys = list(data.keys())
    for i in range(len(data[keys[0]])):
        ndict = {}
        for key in keys:
            ndict[key] = data[key][i]
        new_data.append(ndict)
    return new_data

# below function is created with the help of stackoverflow
def groupby_apply_describe(df, groupby_col, stat_col):
    """From a grouby df object provide the stats
    of describe for each key in the groupby object.

    Parameters
    ----------
    df : spark dataframe groupby object
    col : column to compute statistics on
    
    """
    output = df.groupby(groupby_col).agg(
        F.count(stat_col).alias("count"),
        F.mean(stat_col).alias("mean"),
        F.stddev(stat_col).alias("std"),
        F.min(stat_col).alias("min"),
        F.expr(f"percentile({stat_col}, array(0.25))")[0].alias("%25"),
        F.expr(f"percentile({stat_col}, array(0.5))")[0].alias("%50"),
        F.expr(f"percentile({stat_col}, array(0.75))")[0].alias("%75"),
        F.max(stat_col).alias("max"),
    )
    return output.orderBy(groupby_col)

RegimentSol.zip

[TecSachinGupta]

Hi @areibman,
FYI

[areibman]

Very cool. I'm not actively working on this project anymore, but if you raise a PR I'll merge it!