Persistent results

Persistent results is a Python class that ensures the results will be available even if interruptions occur.

Installation:

pip install pers

Usage:

Example 1

If the code below breaks for some reason (let's say the computer shuts down because of a power outage or some exception), you can restart it to continue from the step it finished.

frompersimportPersistentResultsimportpandasaspd# pandas is not requiredresults=PersistentResults(
'test.pickle', # filename for result cachinginterval=1, # how often dump the resultstmpfilename='~test.pickle'# tmp cache file (optional)
)
fun=lambdax, y, a, b: x**2+yforxinrange(10):
foryinrange(11):
results.append(fun, x, y, a=x, b=y)
results.save()
print(pd.DataFrame(results.data))

Output:

{'result': 66, 'x': 8, 'y': 2, 'a': 8, 'b': 2}
110
| | result | x | y | a | b |
|----:|---------:|----:|----:|----:|----:|
| 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 1 | 0 | 1 | 0 | 1 |
| 2 | 2 | 0 | 2 | 0 | 2 |
[..]
| 107 | 89 | 9 | 8 | 9 | 8 |
| 108 | 90 | 9 | 9 | 9 | 9 |
| 109 | 91 | 9 | 10 | 9 | 10 |

Example 2

frompersimportPersistentResultsimportpandasaspdresults=PersistentResults(
'test2.pickle',
interval=1,
result_prefix=''# we do not want a prefix _result_
)
deffun(x, y, a, b):
print(f'x: {x}, y:{y}')
return { # yes, we can return dictionary'out': x**2+y,
'x': x, # yes, we can return input in dict'a': a,
}
try:
forxinrange(10):
foryinrange(11):
results.append(fun, x=x, y=y, a=x, b=y)
ifx==5andy==5: # simulate rebootraiseException('Unexpected reboot...')
except:
pass# RERUN the tests# will skip already processed elementsforxinrange(10):
foryinrange(11):
results.append(fun, x=x, y=y, a=x, b=y)
results.save()
print(pd.DataFrame(results.data).to_markdown())

x: 0, y:0
x: 0, y:1
[..]
x: 5, y:4
x: 5, y:5
Unexpected reboot...
x: 5, y:6
x: 5, y:7
[..]
x: 9, y:9
x: 9, y:10
| | out | x | a | y | b |
|----:|------:|----:|----:|----:|----:|
| 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 1 | 0 | 0 | 1 | 1 |
| 2 | 2 | 0 | 0 | 2 | 2 |
[..]
| 107 | 89 | 9 | 9 | 8 | 8 |
| 108 | 90 | 9 | 9 | 9 | 9 |
| 109 | 91 | 9 | 9 | 10 | 10 |