Skip to content

functional-python.md

Latest commit

History

History
507 lines (353 loc) · 8.95 KB

functional-python.md

File metadata and controls

507 lines (353 loc) · 8.95 KB
Raw

Functional Programming in Python

—

About me

Data Scientist @ idalab (mainly Python) Used Ruby, JS, Python, Haskell, Swift for nontrivial projects Played with Clojure, Scala, Erlang, Elixir

http://kirelabs.org/fun-js

—

About this talk

  • Not a motivation of functional programming
  • How can FP by used in Python

—

Disclaimer

There should be one — and preferably only one — obvious way to do it. -- PEP 20 — The Zen of Python

^ Python has a philosophy and many of the things you will see go against this philosopy This goes so far that

—

Disclaimer (cont)

The fate of reduce() in Python 3000

not having the choice streamlines the thought process -- Guido van Rossum

^ 2005 Disagree: language should support me Don’t recommend you to use what I’m presenting here Show you the entrance to the rabbit hole Jupyter Notebook! Don’t be afraid to interrupt me!

—

Functional Programming (in Python)

  • first class functions
  • higher order functions
  • purity
  • immutability
  • composition
  • partial application & currying
  • recursion

^ Vocabulary of fuctional concepts

—

Functional Programming (in Python)

  • first class functions
  • higher order functions
  • purity
  • immutability (not today)
  • composition
  • partial application & currying
  • recursion (neither)

—

Purity

functions without side-effects

defadd(a, b):
returna+badditions_made=0defadd(a, b):
globaladditions_madeadditions_made+=1returna+b

—

First class functions

defadd(a, b):
returna+badd_function=addadd=lambdaa,b: a+b

—

higher order functions

deftimer(fn):
deftimed(*args, **kwargs):
t=time()
fn(*args, *kwargs)
print"took {time}".format(time=time()-t)
returntimeddefcompute():
#…timed_compute=timer(compute)
timed_compute()

—

Decorators

@timerdefcompute():
sleep(1)
compute()

—

Partial function application

defadd1(num):
returnadd(1, num)
add1(1)
# simplerfromfunctoolsimportpartialadd1=partial(add, 1)
add1(1)

^ Toy example - just building up a toolbox for the fun stuff

—

Currying

[…] transforming a function that takes multiple arguments in such a way that it can be called as a chain of functions, each with a single argument (partial application) — Wikipedia

—

Currying

defcurried_add(a):
definner(b):
returnadd(a,b)
returninneradd(1) # => <function …>add(1)(1) # => 2

—

Currying

fromtoolzimportcurryadd=curry(add)
add(1) # => <function …>add(1, 1) # => 2

—

Interlude: Closures

defcurried_add(a):
definner(b):
returnadd(a,b)
returninneradd(1) # => <function …>add(1)(1) # => 2

—

Currying example from the stdlib

from operator import itemgetter, attrgetter, methodcaller

obj.method()
fromoperatorimportmethodcallermethodcaller("method")(obj)

^ We will see how this is useful

—

[fit] Functional collection transformations

—

map

map(f, iter)
[f(el) forelinseq]

—

filter

filter(p, seq)
[elforelinseqifp(el)]

—

reduce

fromfunctoolsimportreducereduce(f, seq, initial)
result=initialforelinseq:
result=f(result, el)

—

^ almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do

—

function composition

[f(x) forxinseqifp(x)]
map(f, filter(p, seq))
fromtoolz.curriedimportcompose, map, filtercompute=compose(map(f), filter(p))
compute(seq)

^ Illustrational purposes

—

Example: A bad CSV parser (1/3)

csv="""firstName;lastNameJim;DrakeBen;JamesTim;Banes"""target= [{'firstName': 'Jim', 'lastName': 'Drake'},
{'firstName': 'Ben', 'lastName': 'James'},
{'firstName': 'Tim', 'lastName': 'Banes'}]

—

Example: Imperative Python (2/3)

lines=csv.split("\n")
matrix= [line.split(';') forlineinlines]
header=matrix.pop(0)
records= []
forrowinmatrix:
record= {}
forindex, keyinenumerate(header):
record[key] =row[index]
records.append(record)

—

Example: Functional Python (3/3)

fromtoolz.curriedimportcompose, mapfromfunctoolsimportpartialfromoperatorimportmethodcallersplit=partial(methodcaller, 'split')
split_lines=split("\n")
split_fields=split(';')
dict_from_keys_vals=compose(dict, zip)
csv_to_matrix=compose(map(split_fields), split_lines)
matrix=csv_to_matrix(csv)
keys=next(matrix)
records=map(partial(dict_from_keys_vals, keys), matrix)

—

Example: PySpark

docker run --rm -v ${PWD}:/home/jovyan/work -p 8888:8888 jupyter/pyspark-notebook
defsample(p):
x, y=random(), random()
return1ifx*x+y*y<1else0count=sc.parallelize(range(0, NUM_SAMPLES)) \
.map(sample) \
.reduce(lambdaa, b: a+b)
print("Pi is roughly %f"% (4.0*count/NUM_SAMPLES))

^ http://spark.apache.org/docs/latest/programming-guide.html#transformations

—

Example: K-Means

(Stolen and modified from Joel Grus)

defkmeans(points, k):
returnuntil_convergence(
iterate(
find_new_means(points),
random.sample(points, k)))

—

defuntil_convergence(it):
returnlast(accumulate(no_repeat, it))
defno_repeat(prev, curr):
ifprev==curr: raiseStopIterationelse: returncurr

—

importrandomfromtoolz.curriedimportiterate, accumulate, curry, groupby, last, composedefkmeans(k, points):
returnuntil_convergence(iterate(find_new_means(points), random.sample(points, k)))
@currydeffind_new_means(points, old_means):
k=len(old_means)
clusters=groupby(compose(str, closest_mean(old_means)), points).values()
returnlist(map(cluster_mean, clusters))

—

@currydefclosest_mean(means, point):
returnmin(means, key=squared_distance(point))
@currydefsquared_distance(p, q):
returnsum((p_i-q_i)**2forp_i, q_iinzip(p, q))

—

defcluster_mean(points):
num_points=len(points)
dim=len(points[0]) ifpointselse0sum_points= [sum(point[j] forpointinpoints)
forjinrange(dim)]
return [s/num_pointsforsinsum_points]

—

Main takeaways

  • FP is possible in Python (to a degree)
  • small composable functions are good
  • FP == build general tools and compose them

^ Functional programming enables writing small composable functions Decide for yourself and with your team if this is a good idea

—

[fit] Whats missing in Python (or what I am missing)

  • More list functions
  • Nicer lambda syntax
  • Automatic currying, composition syntax
  • ADTs (sum types)1
  • Pattern Matching

—

Functional libraries

(More list functions)

^ Matthew Rocklin last year’s keynote ALEXEY KACHAYEV

—

Nicer lambda syntax

map(lambdax: x**2, range(5)) # => [0, 1, 4, 9, 16]fromfnimport_map(_**2, range(5)) # => [0, 1, 4, 9, 16]

^ Might not work as expected in i.e. PySpark

—

Other interesting stuff

—

Other talks (where I have stolen material)

^ Matthew Rocklin PyData NYC 2013 - pytoolz ALEXEY KACHAYEV PyCon UA 2012 - fn.py Joel Grus: PyData Seattle 2015

—

More FP?

—

original

Thank you

Daniel Kirsch daniel.kirsch@idalab.de@kirelhttps://github.com/kirel/functional-python

Footnotes

  1. Possible but ugly http://stupidpythonideas.blogspot.de/2014/08/adts-for-python.html↩