Python optimization cheetsheet

Implementation of python optimization cheetsheet (yield, generators, coroutines and asyncio). The source code is located here.

Tools

• python 3.6, 3.7, 3.8
• asyncio
• black
• pylint

Content

• generators
  • functions
  • expressions
  • new iteration patterns with generators
  • advanced operations with generators
  • memory efficient objects
• coroutines
• asyncio

Generators

Basic functions calculate values and returns them, otherwise generators return a lazy iterator that returns a stream of values.

A common use case of generators is to work with data streams or large files like .csv files

functions

Basic generator sample

# materials/generator_sample.pydefgenerator_sample():
yield100generator=generator_sample()
print(generator)
print(type(generator))
print(dir(generator))
print(hasattr(generator, '__next__'))
print(next(generator))
print(next(generator))
generator_list=list(generator_sample())
print(generator_list)
print(len(generator_list))
print(sum(generator_sample()))

Generator with multiple yield statements

# materials/multiple_yields.pydefmultiple_yields():
yield'This'yield'is'yield'my'yield'generator'yield'function'yield'!'values=multiple_yields()
print(next(values))
print(next(values))
print(next(values))
print(next(values))
print(next(values))
other_values=multiple_yields()
forvalueinother_values:
print(value)

Yielding iterable with generator

Any function that has yield operator is a generator.

Generation an infinite sequence, however, will require the use of a generator, since your computer memory is finite. Yield is an expression rather than statement.

# materials/yielding.pydefis_palindrome_number(number):
returnnumber==int(str(number)[::-1])
definfinite_sequence():
num=0whileTrue:
yieldnumnum+=1fornumberininfinite_sequence():
ifis_palindrome_number(number):
print(number)
defcountdown_from(number):
print(f'Starting to count from {number}!')
whilenumber>0:
yieldnumbernumber-=1print('Done!')
defincrement(start, stop):
yieldfromrange(start, stop)
countdown=countdown_from(number=10)
forcountincountdown:
print(count)
incremental=increment(start=1, stop=10)
forincinincremental:
print(inc)

expressions

# materials/generator_expressions.pyeven_numbers= (numfornuminrange(15) ifnum%2==0)
print(even_numbers)
fornumineven_numbers:
print(num)
defmultiply_each_by(multiplier):
return (element*multiplierforelementinrange(5))
multiplied_container=multiply_each_by(multiplier=3)
print(multiplied_container)
forobjinmultiplied_container:
print(obj)

new iteration patterns with generators

# materials/float_range.pydeffloat_range(start, stop, increment):
initial_point=startwhileinitial_point<stop:
yieldinitial_pointinitial_point+=incrementfornumberinfloat_range(0, 4, 0.5):
print(number)

# materials/countdown.pyclassCountdown:
def__init__(self, start):
self._start=startdef__iter__(self):
number=self._startwhilenumber>0:
yieldnumbernumber-=1def__reversed__(self):
number=1whilenumber<=self._start:
yieldnumbernumber+=1forward_countdown=Countdown(10)
forf_countinforward_countdown:
print(f_count)
reversed_countdown=reversed(Countdown(10))
forr_countinreversed_countdown:
print(r_count)

advanced operations with generators

Slice generator elements

# materials/slice_generators.pyimportitertoolsdefdoubles_of(number):
fornuminrange(number):
yield2*numprint(help(itertools.islice))
forelementinitertools.islice(doubles_of(50), 10, 15):
print(element)

Concatenate generators sequence

# materials/concatenate_generators.pyimportitertoolsdeffruits():
forfruitin ('apple', 'orange', 'banana'):
yieldfruitdefvegetables():
forvegetablein ('potato', 'tomato', 'cucumber'):
yieldvegetableprint(help(itertools.chain))
bucket=itertools.chain(fruits(), vegetables())
foriteminbucket:
print(item)

Zip generators elements

# materials/zip_generators.pyimportitertoolsdefascending():
yieldfrom (1, 2, 3, 4, 5)
defdescending():
yieldfrom (5, 4, 3, 2, 1)
forpairinitertools.zip_longest(ascending(), descending()):
print(pair)

memory efficient objects

# materials/memory_efficacy.pyimportsysimportcProfilegenerator_container= (num*3fornuminrange(10000000) ifnum%6==0ornum%7==0)
print(sys.getsizeof(generator_container))
list_container= [num*3fornuminrange(10000000) ifnum%6==0ornum%7==0]
print(sys.getsizeof(list_container))
print(cProfile.run('sum(generator_container)'))
print(cProfile.run('sum(list_container)'))

Coroutines

Coroutines can consume and produce data. They can pause stream execution till next message is sent.

Generators produce data for iteration while coroutines can also consume data.

Sending values with .send method

# materials/send_coroutines.pydefcoroutine():
whileTrue:
value=yield# allows to manipulate yielded valueprint(value)
i=coroutine()
i.send(None) # initial value should be 'None'i.send(1)
i.send(10)
defcounter(maximum):
initial=0whileinitial<maximum:
value= (yieldinitial) # equals to None till .send(number) is called# If value is given (remember default is None) then change the counterifvalueisnotNone:
initial=valueelse:
initial+=1c=counter(10)
print(next(c)) # 0print(next(c)) # 1print(c.send(5)) # 5print(next(c)) # 6defis_palindrome_number(number):
returnnumber==int(str(number)[::-1])
definfinite_palindromes():
number=0whileTrue:
ifis_palindrome_number(number):
i= (yieldnumber)
ifiisnotNone:
number=inumber+=1c=infinite_palindromes()
print(next(c)) # 0print(next(c)) # 1print(c.send(100)) # 101print(next(c)) # 111defprint_name(prefix):
print("Search for ", prefix, " prefix")
whileTrue:
name=yieldifprefixinname:
print(name)
pn=print_name("Dear")
next(pn) # calls first yield expressionpn.send("Alex")
pn.send("Dear Alex") # matches with prefixdefgrep(pattern):
print(f"Search for '{pattern}' pattern")
whileTrue:
value=yieldifpatterninvalue:
print(f"Matched: '{value}'")
g=grep("hey")
next(g) # to start coroutineg.send("hello")
g.send("hey")
g.send("hey Mike")

Raise an exception with .throw method

.throw() allows you to throw exceptions through the generator.

# materials/throw_coroutines.pydefcounter(maximum):
initial=0whileinitial<maximum:
value= (yieldinitial) # equals to None till .send(number) is called# If value is given (remember default is None) then change the counterifvalueisnotNone:
initial=valueelse:
initial+=1c=counter(10)
foriinc:
print(i)
ifi==5:
c.throw(ValueError("It is too large"))

Stop generator with .close method

.close() allows you to stop a generator. Instead of calling .throw(), you use .close() (it calls StopIteration error).

# materials/close_coroutines.pydefcounter(maximum):
initial=0whileinitial<maximum:
value= (yieldinitial) # equals to None till .send(number) is called# If value is given (remember default is None) then change the counterifvalueisnotNone:
initial=valueelse:
initial+=1c=counter(10)
foriinc:
print(i)
ifi==5:
c.close() # stops as here is raises 'StopIteration' exceptiondefprint_name(prefix):
print("Search for", prefix, "prefix")
try:
whileTrue:
name=yieldifprefixinname:
print(name)
exceptGeneratorExit:
print("Closing generator!")
pn=print_name("Dear")
next(pn) # calls first yield expressionpn.send("Alex")
pn.send("Dear Alex") # matches with prefix

Create pipelines

Coroutines can be used to set pipes

# materials/coroutine_chaining.pydefproducer(sentence: str, next_coroutine):
"""Split strings and feed it to pattern_filter coroutine."""tokens=sentence.split(" ")
fortokenintokens:
next_coroutine.send(token)
next_coroutine.close()
defpattern_filter(pattern="ing", next_coroutine=None):
"""Search for pattern and if pattern got matched, send it to print_token coroutine."""print(f"Search for {pattern} pattern")
try:
whileTrue:
token=yieldifpatternintoken:
next_coroutine.send(token)
exceptGeneratorExit:
print("Done with filtering")
defprint_token():
"""Act as a sink, simply print the token."""print("I'm sink, I'll print tokens")
try:
whileTrue:
token=yieldprint(token)
exceptGeneratorExit:
print("Done with printing")
pt=print_token()
next(pt)
pf=pattern_filter(next_coroutine=pt)
next(pf)
sentence="Bob is running behind a fast moving car"producer(sentence, pf)

Tricks

# materials/decorator.pydefcoroutine(func):
"""A decorator function that eliminates the need to call .next() when starting a coroutine."""defstart(*args, **kwargs):
cr=func(*args, **kwargs)
next(cr)
returncrreturnstartif__name__=="__main__":
@coroutinedefgrep(pattern):
print(f"Search for '{pattern}' pattern")
whileTrue:
value=yieldifpatterninvalue:
print(value)
g=grep("python")
# Notice now you don't need a next() call hereg.send("Yeah, but no, but yeah, but no")
g.send("A series of tubes")
g.send("python generators rock!")

# materials/benchmark.pyfromtimeitimporttimeitfrommaterials.decoratorimportcoroutine# An objectclassGrepHandler:
def__init__(self, pattern, target):
self._pattern=patternself._target=targetdefsend(self, line):
ifself._patterninline:
self._target.send(line)
# a coroutine@coroutinedefgrep(pattern, target):
whileTrue:
line=yieldifpatterninline:
target.send(line)
# A null-sink to send data@coroutinedefnull():
whileTrue:
item=yieldif__name__=="__main__":
# A benchmarkline="python is nice"p1=grep("python", null()) # coroutinep2=GrepHandler("python", null()) # an objectprint("Coroutine: ", timeit("p1.send(line)", "from __main__ import line, p1"))
print("Object: ", timeit("p2.send(line)", "from __main__ import line, p2"))

# materials/broadcast.py"""An example of broadcasting a data stream onto multiple coroutine targets."""importtimefrommaterials.decoratorimportcoroutine# A data source. This is not a coroutine, but it sends data into one targetdeffollow(thefile, target):
thefile.seek(0, 2) # Go to end of a filewhileTrue:
line=thefile.readline()
ifnotline:
time.sleep(0.1)
continuetarget.send(line)
# A filter@coroutinedefgrep(pattern, target):
whileTrue:
line=yield# Receive a lineifpatterninline:
target.send(line) # Send to next stage# A sink. A coroutine that receives data@coroutinedefprinter():
whileTrue:
line=yieldprint(line)
# Broadcast a stream onto multiple targets@coroutinedefbroadcast(targets):
whileTrue:
item=yieldfortargetintargets:
target.send(item)
if__name__=="__main__":
f=open("access.log", "+a")
follow(f, broadcast((grep("python", printer()), grep("ply", printer()), grep("swig", printer()))))

AsyncIO

Asynchronous IO is a concurrent programming design (paradigm). Coroutines (specialized generator functions) are the heart of async IO in Python.

Parallelism consists of performing multiple operations at the same time. Multiprocessing is a means to effect parallelism, and it entails spreading tasks over a computer’s central processing units (CPUs, or cores).

Concurrency is a slightly broader term than parallelism. Multiple tasks have the ability to run in an overlapping manner. Concurrency (concurrent.futures package) include both multiprocessing and threading.

Threading is a concurrent execution model whereby multiple threads take turns executing tasks. One process can contain multiple threads.

asyncio is a library to write concurrent code. It is not threading, nor is it multiprocessing. In fact, async IO is a single-threaded, single-process design: it uses cooperative multitasking. Coroutines (a central feature of async IO) can be scheduled concurrently, but they are not inherently concurrent.

async IO is a style of concurrent programming, but it is not parallelism. It’s more closely aligned with threading than with multiprocessing but is very much distinct from both of these and is a standalone member in concurrency’s bag of tricks

What is asynchronous ?

• Asynchronous routines are able to “pause” while waiting on their ultimate result and let other routines run in the meantime
• Asynchronous code, facilitates concurrent execution

Async IO takes long waiting periods in which functions would otherwise be blocking and allows other functions to run during that downtime

async built on non-blocking sockets, callbacks and event loops. async def syntax stand for native coroutine or asynchronous generator. await keyword passes function control back to event loop. It suspends the execution of coroutine.

# materials/async_.pyimportasyncioasyncdefcount(): # single event loopprint("One")
awaitasyncio.sleep(1) # when task reaches here it will sleep to 1 seconds ands says to do other job meantimeprint("Two")
asyncdefmain():
awaitasyncio.gather(count(), count(), count())
if__name__=="__main__":
importtimes=time.perf_counter()
asyncio.run(main())
elapsed=time.perf_counter() -sprint(f"{__file__} executed in {elapsed:0.2f} seconds.")

# materials/sync.pyimporttimedefcount():
print("One")
time.sleep(1)
print("Two")
defmain():
for_inrange(3):
count()
if__name__=="__main__":
s=time.perf_counter()
main()
elapsed=time.perf_counter() -sprint(f"{__file__} executed in {elapsed:0.2f} seconds.") # 3.01 seconds

If Python encounters an await f() expression in the scope of g(), this is how await tells the event loop, “Suspend execution of g() until whatever I’m waiting on—the result of f() — is returned. In the meantime, go let something else run.” async def is a coroutine. It may use await, return, or yield, but all of these are optional.

asyncdefg():
# Pause here and come back to g() when f() is readyr=awaitf()
returnr

Using await and/or return creates a coroutine function. To call a coroutine function, you must await it to get its results.

Using yield in an async def block creates an asynchronous generator, which you iterate over with async for. yield from in an async def will raise SyntaxError.

# materials/async_gen.pyasyncdefgenfunc():
yield1yield2gen=genfunc()
assertgen.__aiter__() isgenassertawaitgen.__anext__() ==1assertawaitgen.__anext__() ==2awaitgen.__anext__() # This line will raise StopAsyncIteration.

asyncdeff(x):
y=awaitz(x) # OK - `await` and `return` allowed in coroutinesreturnyasyncdefg(x):
yieldx# OK - this is an async generatorasyncdefm(x):
yieldfromgen(x) # No - SyntaxErrordefm(x):
y=awaitz(x) # Still no - SyntaxError (no `async def` here)returny

Materials

• https://docs.python.org/3/howto/functional.html#generator-expressions-and-list-comprehensions
• https://www.python.org/dev/peps/pep-0289
• https://www.python.org/dev/peps/pep-0342
• https://www.python.org/dev/peps/pep-0525
• https://docs.python.org/3/library/asyncio.html
• https://docs.python.org/3.6/glossary.html#term-generator
• https://realpython.com/introduction-to-python-generators
• https://www.geeksforgeeks.org/coroutine-in-python
• http://www.dabeaz.com/coroutines
• https://realpython.com/async-io-python

Meta

Author – Volodymyr Yahello vyahello@gmail.com

Distributed under the Apache (2.0) license. See LICENSE for more information.

You can reach out me at:

• https://github.com/vyahello
• https://www.linkedin.com/in/volodymyr-yahello-821746127

Contributing

1. clone the repository
2. configure git for the first time after cloning with your name and email
3. pip install -r requirements.txt to install all project dependencies