Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 3
Persistence
Benoit Daloze edited this page Mar 25, 2014
·
5 revisions
A set of YAML files, saved by absolute path of the benchmark file, which contains multiple documents, each of them being a "Result":
{metadata: {# mandatoryfile: "/full/path/to/benchmark/file.rb",session: "Array sorting",# session namejob: "#sort!",# job titleruby: RUBY_DESCRIPTION,run_time: Time.now,# time at which the benchmark tool started# Benchmarks ran in the same context/session should have identical :run_time# So :run_time should be initialized once in the application# optional# configurationminimal_time: 1.0,measurements: 10,verbose: true,# git, git data of the current project, if availablegit_branch: "master",git_commit: "3433deb...",# benchmark type dependentiterations: 1234567,# numbers of iterationsn: 1024,# input size# otherscommand_line: "perfer run ...",# full command linebench_file_checksum: "78e6530...",# benchmark file SHA-1 checksum}data: [{# mandatoryreal: 3.14,# real time# optionalutime: 0.01,# user timestime: 0.01,# system time},
...
]}Given the benchmark file /path/to/benchmark/file.rb,
results are saved in ~/.perfer/results/path/to/benchmark/file.rb.
(On Windows, the drive letter is added: ~/.perfer/results/C/path/...)
- necessary for history comparisons
- allows to reuse data a nice way (by keeping at least last run results):
$ perfer run bench_sort
$ perfer report bench_sort
$ perfer graph bench_sortDimensions:
- benchmark:
- project (detected by folder containing .git?)
- file (should be "relocatable" if absolute path)
- job
- version (md5, git hash?)
- time: which code/implementation it ran on (commit/blob hash or "working directory version")
- ruby implementation (RUBY_DESCRIPTION)
- hardware
on_change:
- benchmark code: either merge or remove old data OR manage bench versions
- implementation code: should manage with impl code commit hash change (overwrite last if working dir version? (=> ctime?) )
- hardware change: should still allow comparison, but not mix results by default
Where to save?
- ~/.perfer: avoid generated files in project dir (persist if repo deleted, better for multi-repos)
- in project bench{,mark}/results (has to be git-ignored, but easy to find and keep around, always relative paths to bench{,mark} dir?)
Format:
- plain ruby Hashes, one per line (but lot of space wasted, no explicit results grouping)
- group them, and add a header before each set of results
- or inline results in Arrays (but grouping by result less clear, and never-ending-line)
Typical use cases:
- new user: a few benches, mostly last results, but then maybe compare a few versions (does not want too much setup like git and config files)
- CLI guru: do everything in "-e" (does not want any setup at all)
- ruby implementation dev: perf history across versions, commits
- library developer: keep benchmarks in bench/, run regularly, perf history