pontem

Treat PySpark DataFrames like pandas.

This is currently just a hobby project, not suitable for use.

Turn somethinig like this:

# Pure PySpark API; df is type pyspark.sql.DataFramedefmultiply(n):
returnudf(lambdacol: col*n, FloatType())
df=df.withColumn('new_col', df.select(multiply(2)(df['other_col'])))

...into this:

# Using pontem.core.DataFrame object.df['new_col'] =df['other_col'] *2