Added results to benchmark suffix by MatteoWohlrapp · Pull Request #821 · marrlab/DomainLab

MatteoWohlrapp · 2024-05-07T11:50:57Z

Added time when the benchmark script was executed as a suffix. For that, I am getting the folder name from the configuration file and extending this name with the timestamp, then adding it to the snakemake command to overwrite the existing output folder path. See format below:

smilesun · 2024-05-08T12:42:48Z

# only plot
# snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml" --keep-going --allowed-rules agg_partial_results
# snakemake --cores 1 -s "domainlab/exp_protocol/benchmark.smk" --configfile "examples/yaml/demo_benchmark.yaml" --keep-going --allowed-rules gen_plots
,method,model,params
0,method_diva,diva,"{'gamma_d': 10000.0, 'zy_dim': 32}"
1,method_diva,diva,"{'gamma_d': 10000.0, 'zy_dim': 64}"
2,method_diva,diva,"{'gamma_d': 310000.0, 'zy_dim': 32}"
3,method_diva,diva,"{'gamma_d': 310000.0, 'zy_dim': 64}"
4,method_hduva,hduva,{'zy_dim': 32}
5,method_hduva,hduva,{'zy_dim': 64}
6,erm,erm,{'lr': 0.001}
7,erm,erm,{'lr': 0.0001}
cat: zoutput/benchmarks/mnist_benchmark_grid/results.csv: No such file or directory
Error: Process completed with exit code 1.

MatteoWohlrapp · 2024-05-08T13:19:37Z

When did you get this output?

smilesun · 2024-05-08T13:23:53Z

When did you get this output?

i just copied the CI log, maybe it gives you a hint what went wrong.

codecov-commenter · 2024-05-10T15:07:23Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.17%. Comparing base (44271c8) to head (e67e212).

Additional details and impacted files

@@ Coverage Diff @@## master #821 +/- ##
=======================================
Coverage 95.17% 95.17% =======================================
Files 128 128 Lines 5080 5080 =======================================
Hits 4835 4835 Misses 245 245

Flag	Coverage Δ
unittests	`95.17% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

smilesun · 2024-05-13T11:53:49Z

When did you get this output?
i just copied the CI log, maybe it gives you a hint what went wrong.

I just added a wildcard to the benchmark results folder in the CI script, hopefuly it can fix the issue

smilesun · 2024-05-13T12:18:47Z

 export DOMAINLAB_CUDA_START_SEED=$2
 fi

+# Extract output_dir from the YAML configuration file


could you put the common bash operations into a function in a separate script in scripts folder?

smilesun · 2024-05-13T13:38:55Z

When did you get this output?
i just copied the CI log, maybe it gives you a hint what went wrong.
I just added a wildcard to the benchmark results folder in the CI script, hopefuly it can fix the issue

CI passed now

…/DomainLab into benchmark_results_suffix

smilesun · 2024-06-25T10:04:08Z

I clicked two times rerun the CI, but it looks like it did not re-run

smilesun · 2024-06-26T16:32:42Z

still the same error with poetry

NameError: name 'extend' is not defined
Error: Process completed with exit code 1.

@agisga , do you have any idea what is going on?

agisga · 2024-06-27T02:31:12Z

@smilesun I just looked through the commit history of DomId, and I had solved an issue that seems to be the same error with "name 'extend' is not defined"! What worked for me was adding the following to pyproject.toml:

setuptools = "^69.5.1"

There seems to be a compatibility bug between backpack and newer versions of setuptools (70.x). Hope that helps!

smilesun · 2024-06-28T13:06:51Z

=========================== short test summary info ============================
FAILED tests/test_benchmark_plots.py::test_benchm_plots - AttributeError: 'Le...
FAILED tests/test_exp_protocol.py::test_agg_main - AttributeError: 'Legend' o...
=========== 2 failed, 96 passed, [581](https://github.com/marrlab/DomainLab/actions/runs/9697814097/job/26762854758?pr=821#step:7:582)7 warnings in 2332.02s (0:38:52) ===========

smilesun · 2024-06-28T13:18:05Z

2024-06-27T14:37:42.3965653Z tests/test_utils.py .. [100%]
2024-06-27T14:37:42.3966347Z 2024-06-27T14:37:42.3966597Z =================================== FAILURES ===================================
2024-06-27T14:37:42.3967304Z ______________________________ test_benchm_plots _______________________________
2024-06-27T14:37:42.3967882Z 2024-06-27T14:37:42.3968070Z def test_benchm_plots():
2024-06-27T14:37:42.3968605Z """
2024-06-27T14:37:42.3969007Z test benchmark plots
2024-06-27T14:37:42.3969481Z """
2024-06-27T14:37:42.3969870Z > gen_benchmark_plots(
2024-06-27T14:37:42.3970499Z "domainlab/zdata/ztest_files/aggret_res_test1",
2024-06-27T14:37:42.3971238Z "zoutput/benchmark_plots_test/outp1",
2024-06-27T14:37:42.3971858Z use_param_index=False,
2024-06-27T14:37:42.3972340Z )
2024-06-27T14:37:42.3972535Z 2024-06-27T14:37:42.3972729Z tests/test_benchmark_plots.py:11: 2024-06-27T14:37:42.3973401Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-06-27T14:37:42.3974363Z domainlab/utils/generate_benchmark_plots.py:57: in gen_benchmark_plots
2024-06-27T14:37:42.3975233Z gen_plots(raw_df, output_dir, use_param_index)
2024-06-27T14:37:42.3976013Z domainlab/utils/generate_benchmark_plots.py:102: in gen_plots
2024-06-27T14:37:42.3976732Z scatterplot_matrix(
2024-06-27T14:37:42.3977820Z domainlab/utils/generate_benchmark_plots.py:303: in scatterplot_matrix
2024-06-27T14:37:42.3978870Z sns.move_legend(g_p, loc="upper right", bbox_to_anchor=(1.0, 1.0), ncol=1)
2024-06-27T14:37:42.3979839Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 2024-06-27T14:37:42.3980416Z 2024-06-27T14:37:42.3981270Z obj = <seaborn.axisgrid.PairGrid object at 0x7f29cc9a57e0>, loc = 'upper right'
2024-06-27T14:37:42.3982302Z kwargs = {'bbox_to_anchor': (1.0, 1.0), 'ncol': 1}
2024-06-27T14:37:42.3983080Z Grid = <class 'seaborn.axisgrid.Grid'>
2024-06-27T14:37:42.3983887Z old_legend = <matplotlib.legend.Legend object at 0x7f29cc24fa00>
2024-06-27T14:37:42.3985032Z legend_func = <bound method FigureBase.legend of <Figure size 1250x1200 with 14 Axes>>
2024-06-27T14:37:42.3985812Z 2024-06-27T14:37:42.3986016Z def move_legend(obj, loc, **kwargs):
2024-06-27T14:37:42.3986573Z """
2024-06-27T14:37:42.3987511Z Recreate a plot's legend at a new location.
2024-06-27T14:37:42.3988155Z 2024-06-27T14:37:42.3988806Z The name is a slight misnomer. Matplotlib legends do not expose public
2024-06-27T14:37:42.3990011Z control over their position parameters. So this function creates a new legend,
2024-06-27T14:37:42.3991262Z copying over the data from the original object, which is then removed.
2024-06-27T14:37:42.3992500Z 2024-06-27T14:37:42.3992894Z Parameters
2024-06-27T14:37:42.3993353Z ----------
2024-06-27T14:37:42.3993807Z obj : the object with the plot
2024-06-27T14:37:42.3994582Z This argument can be either a seaborn or matplotlib object:
2024-06-27T14:37:42.3995309Z 2024-06-27T14:37:42.3995942Z - :class:`seaborn.FacetGrid` or :class:`seaborn.PairGrid`
2024-06-27T14:37:42.3997056Z - :class:`matplotlib.axes.Axes` or :class:`matplotlib.figure.Figure`
2024-06-27T14:37:42.3997854Z 2024-06-27T14:37:42.3998223Z loc : str or int
2024-06-27T14:37:42.3998934Z Location argument, as in :meth:`matplotlib.axes.Axes.legend`.
2024-06-27T14:37:42.3999672Z 2024-06-27T14:37:42.4000014Z kwargs
2024-06-27T14:37:42.4000726Z Other keyword arguments are passed to :meth:`matplotlib.axes.Axes.legend`.
2024-06-27T14:37:42.4001526Z 2024-06-27T14:37:42.4001834Z Examples
2024-06-27T14:37:42.4002236Z --------
2024-06-27T14:37:42.4002573Z 2024-06-27T14:37:42.4003001Z .. include:: ../docstrings/move_legend.rst
2024-06-27T14:37:42.4003550Z 2024-06-27T14:37:42.4003844Z """
2024-06-27T14:37:42.4004463Z # This is a somewhat hackish solution that will hopefully be obviated by
2024-06-27T14:37:42.4005495Z # upstream improvements to matplotlib legends that make them easier to
2024-06-27T14:37:42.4006290Z # modify after creation.
2024-06-27T14:37:42.4006738Z 2024-06-27T14:37:42.4007254Z from seaborn.axisgrid import Grid # Avoid circular import
2024-06-27T14:37:42.4007959Z 2024-06-27T14:37:42.4008483Z # Locate the legend object and a method to recreate the legend
2024-06-27T14:37:42.4009231Z if isinstance(obj, Grid):
2024-06-27T14:37:42.4009750Z old_legend = obj.legend
2024-06-27T14:37:42.4010311Z legend_func = obj.figure.legend
2024-06-27T14:37:42.4010932Z elif isinstance(obj, mpl.axes.Axes):
2024-06-27T14:37:42.4011552Z old_legend = obj.legend_
2024-06-27T14:37:42.4012111Z legend_func = obj.legend
2024-06-27T14:37:42.4012731Z elif isinstance(obj, mpl.figure.Figure):
2024-06-27T14:37:42.4013361Z if obj.legends:
2024-06-27T14:37:42.4013961Z old_legend = obj.legends[-1]
2024-06-27T14:37:42.4014532Z else:
2024-06-27T14:37:42.4014948Z old_legend = None
2024-06-27T14:37:42.4015470Z legend_func = obj.legend
2024-06-27T14:37:42.4015998Z else:
2024-06-27T14:37:42.4016689Z err = "`obj` must be a seaborn Grid or matplotlib Axes or Figure instance."
2024-06-27T14:37:42.4017843Z raise TypeError(err)
2024-06-27T14:37:42.4018334Z 2024-06-27T14:37:42.4018703Z if old_legend is None:
2024-06-27T14:37:42.4019268Z err = f"{obj} has no legend attached."
2024-06-27T14:37:42.4019906Z raise ValueError(err)
2024-06-27T14:37:42.4020406Z 2024-06-27T14:37:42.4020909Z # Extract the components of the legend we need to reuse
2024-06-27T14:37:42.4021678Z > handles = old_legend.legendHandles
2024-06-27T14:37:42.4022630Z E AttributeError: 'Legend' object has no attribute 'legendHandles'

smilesun · 2024-07-01T13:20:23Z

I tried sometime to fix the seborn problem above: currently, matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0" and seaborn==0.12.2 ; python_version >= "3.9" and python_version < "4.0" in the requirements.txt, but i could not reproduce the error on my laptop after chaning the version of matplotlib and seaborn, any idea @MatteoWohlrapp @agisga

smilesun · 2024-07-01T13:27:49Z

the error occurs at rule:


- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1

smilesun · 2024-07-02T08:12:51Z

2024-07-01T14:18:47.4395753Z old_legend = <matplotlib.legend.Legend object at 0x7f71b8923a00>
2024-07-01T14:18:47.4396366Z legend_func = <bound method FigureBase.legend of <Figure size 1250x1200 with 14 Axes>>
2024-07-01T14:18:47.4396787Z 2024-07-01T14:18:47.4396904Z def move_legend(obj, loc, **kwargs):
2024-07-01T14:18:47.4397217Z """
2024-07-01T14:18:47.4397812Z Recreate a plot's legend at a new location.
2024-07-01T14:18:47.4398176Z 2024-07-01T14:18:47.4398540Z The name is a slight misnomer. Matplotlib legends do not expose public
2024-07-01T14:18:47.4399189Z control over their position parameters. So this function creates a new legend,
2024-07-01T14:18:47.4399834Z copying over the data from the original object, which is then removed.
2024-07-01T14:18:47.4400273Z 2024-07-01T14:18:47.4400475Z Parameters
2024-07-01T14:18:47.4400726Z ----------
2024-07-01T14:18:47.4400979Z obj : the object with the plot
2024-07-01T14:18:47.4401409Z This argument can be either a seaborn or matplotlib object:
2024-07-01T14:18:47.4401804Z 2024-07-01T14:18:47.4402163Z - :class:`seaborn.FacetGrid` or :class:`seaborn.PairGrid`
2024-07-01T14:18:47.4402771Z - :class:`matplotlib.axes.Axes` or :class:`matplotlib.figure.Figure`
2024-07-01T14:18:47.4403205Z 2024-07-01T14:18:47.4403412Z loc : str or int
2024-07-01T14:18:47.4403802Z Location argument, as in :meth:`matplotlib.axes.Axes.legend`.
2024-07-01T14:18:47.4404210Z 2024-07-01T14:18:47.4404401Z kwargs
2024-07-01T14:18:47.4404801Z Other keyword arguments are passed to :meth:`matplotlib.axes.Axes.legend`.
2024-07-01T14:18:47.4405267Z 2024-07-01T14:18:47.4405452Z Examples
2024-07-01T14:18:47.4405695Z --------
2024-07-01T14:18:47.4406093Z 2024-07-01T14:18:47.4406361Z .. include:: ../docstrings/move_legend.rst
2024-07-01T14:18:47.4406689Z 2024-07-01T14:18:47.4406866Z """
2024-07-01T14:18:47.4407237Z # This is a somewhat hackish solution that will hopefully be obviated by
2024-07-01T14:18:47.4407847Z # upstream improvements to matplotlib legends that make them easier to
2024-07-01T14:18:47.4408304Z # modify after creation.
2024-07-01T14:18:47.4408576Z 2024-07-01T14:18:47.4408883Z from seaborn.axisgrid import Grid # Avoid circular import
2024-07-01T14:18:47.4409271Z 2024-07-01T14:18:47.4409580Z # Locate the legend object and a method to recreate the legend
2024-07-01T14:18:47.4410006Z if isinstance(obj, Grid):
2024-07-01T14:18:47.4410308Z old_legend = obj.legend
2024-07-01T14:18:47.4410635Z legend_func = obj.figure.legend
2024-07-01T14:18:47.4411000Z elif isinstance(obj, mpl.axes.Axes):
2024-07-01T14:18:47.4411344Z old_legend = obj.legend_
2024-07-01T14:18:47.4411649Z legend_func = obj.legend
2024-07-01T14:18:47.4411992Z elif isinstance(obj, mpl.figure.Figure):
2024-07-01T14:18:47.4412333Z if obj.legends:
2024-07-01T14:18:47.4412673Z old_legend = obj.legends[-1]
2024-07-01T14:18:47.4412986Z else:
2024-07-01T14:18:47.4413214Z old_legend = None
2024-07-01T14:18:47.4413507Z legend_func = obj.legend
2024-07-01T14:18:47.4413898Z else:
2024-07-01T14:18:47.4414548Z err = "`obj` must be a seaborn Grid or matplotlib Axes or Figure instance."
2024-07-01T14:18:47.4415393Z raise TypeError(err)
2024-07-01T14:18:47.4415851Z 2024-07-01T14:18:47.4416109Z if old_legend is None:
2024-07-01T14:18:47.4416712Z err = f"{obj} has no legend attached."
2024-07-01T14:18:47.4417350Z raise ValueError(err)
2024-07-01T14:18:47.4417811Z 2024-07-01T14:18:47.4418286Z # Extract the components of the legend we need to reuse
2024-07-01T14:18:47.4419012Z > handles = old_legend.legendHandles
2024-07-01T14:18:47.4419922Z E AttributeError: 'Legend' object has no attribute 'legendHandles'
2024-07-01T14:18:47.4420541Z 2024-07-01T14:18:47.4421168Z ../../../.cache/pypoetry/virtualenvs/domainlab-okngg5wM-py3.10/lib/python3.10/site-packages/seaborn/utils.py:456: AttributeError
2024-07-01T14:18:47.4421977Z ________________________________ test_agg_main _________________________________
2024-07-01T14:18:47.4422291Z 2024-07-01T14:18:47.4422775Z bm_config = 'zoutput/test', agg_output_file = 'zoutput/test/results.csv'
2024-07-01T14:18:47.4424194Z agg_expected_output = 'param_index, method, algo, epos, te_d, seed, params, acc, precision, recall, specificity, f1, aurocy\n0, diva, diva, ..., caltech, 1, "{\'gamma_y\': 70037, \'zy_dim\': 48}", 0.7307692, 0.557971, 0.5333333, 0.5333333, 0.5297158, 0.73333335'
2024-07-01T14:18:47.4425202Z 2024-07-01T14:18:47.4425427Z def test_agg_main(bm_config, agg_output_file, agg_expected_output):
2024-07-01T14:18:47.4425958Z """Testing the csv aggregation from a full directory."""
2024-07-01T14:18:47.4426394Z > agg_main(bm_config, skip_plotting=True)
2024-07-01T14:18:47.4426643Z 2024-07-01T14:18:47.4426757Z tests/test_exp_protocol.py:148:

… seed and added comments to the utils function for the benchmark script

smilesun · 2024-07-02T09:23:54Z

I tried sometime to fix the seborn problem above: currently, matplotlib==3.8.2 ; python_version >= "3.9" and python_version < "4.0" and seaborn==0.12.2 ; python_version >= "3.9" and python_version < "4.0" in the requirements.txt, but i could not reproduce the error on my laptop after chaning the version of matplotlib and seaborn, any idea @MatteoWohlrapp @agisga

@MatteoWohlrapp @agisga , I FIXED it, in fact, the unit test is run with the poetry dependencies, instead of requirements.txt, i have updated pyproj.yaml and now the unit test passed for seaborn and matplotlib

smilesun

Thanks for the nice work @MatteoWohlrapp , i will now test it on the cluster if the benchmark works as expected.

smilesun · 2024-07-02T09:52:32Z

Thanks for the nice work @MatteoWohlrapp , i will now test it on the cluster if the benchmark works as expected.

It works on the cluster!

MatteoWohlrapp added 2 commits May 7, 2024 12:19

Added changes to benchmark script
7173bd0

Added changes to benchmark script to add timestamp to output directory
064471b

MatteoWohlrapp linked an issue May 7, 2024 that may be closed by this pull request
benchmark results folder should be named with suffix indicating the time when the benchmark started #783
Closed

MatteoWohlrappand others added 2 commits May 7, 2024 14:16

Added quotes around variables
2d869fa

Merge branch 'master' into benchmark_results_suffix
e66ed7a

smilesun added 2 commits May 10, 2024 15:59

Merge branch 'master' into benchmark_results_suffix
367bb15

Merge branch 'master' into benchmark_results_suffix
46b9d4b

smilesun added 3 commits May 10, 2024 20:05

Merge branch 'master' into benchmark_results_suffix
a5b5a95

Merge branch 'master' into benchmark_results_suffix
1e788bd

add wildcard * to benchmarks results to represent date
547c3a3

smilesun requested changes May 13, 2024
View reviewed changes

smilesun added the priority Further information is requested label May 13, 2024

smilesun assigned MatteoWohlrapp May 13, 2024

smilesunand others added 8 commits May 13, 2024 22:54

Merge branch 'master' into benchmark_results_suffix
d573b92

Abstracted common functions from the script
abebfd8

Merge branch 'benchmark_results_suffix' ofhttps://github.com/marrlab…
472a0e4
…/DomainLab into benchmark_results_suffix

Merge branch 'master' into benchmark_results_suffix
2a347e7

Remove set -e to see if it extends the logs
7d792c3

Adding debug statements to standalone script, exiting without errors
ee84bcb

ci running, exiting with 0, removing prints
bc3c808

Renamed common benchmark file
5a99aa6

Matteo Wohlrapp added 2 commits June 25, 2024 18:30

Specified pip version
88e383d

Specified pip version
c672a04

smilesun added 2 commits June 27, 2024 15:36

Update pyproject.toml for version lower than 70 of setup tools
13af9f0

delete poetry lock
1220cc1

update requirments.txt using poetry
c546ae9

Update pyproject.toml
3b91f70

smilesun reviewed Jul 2, 2024
View reviewed changes

Comment thread.github/workflows/ci.yml Outdated

smilesun requested changes Jul 2, 2024
View reviewed changes

Comment thread.github/workflows/ci.yml
Comment threadrun_benchmark_slurm.sh
Comment threadrun_benchmark_slurm.sh
Comment threadscripts/sh_benchmark_utils.sh

Removed explicit pip version in ci, added comment about user starting…
e67e212
… seed and added comments to the utils function for the benchmark script

smilesun approved these changes Jul 2, 2024
View reviewed changes

smilesun merged commit faf7589 into masterJul 2, 2024

smilesun deleted the benchmark_results_suffix branch July 2, 2024 09:58

Uh oh!

Conversation

MatteoWohlrapp commented May 7, 2024

Uh oh!

smilesun commented May 8, 2024

Uh oh!

MatteoWohlrapp commented May 8, 2024

Uh oh!

smilesun commented May 8, 2024

Uh oh!

codecov-commenter commented May 10, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

smilesun commented May 13, 2024

Uh oh!

smilesunMay 13, 2024

Choose a reason for hiding this comment

Uh oh!

smilesun commented May 13, 2024

Uh oh!

smilesun commented Jun 25, 2024

Uh oh!

smilesun commented Jun 26, 2024

Uh oh!

agisga commented Jun 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smilesun commented Jun 28, 2024

Uh oh!

smilesun commented Jun 28, 2024

Uh oh!

smilesun commented Jul 1, 2024

Uh oh!

smilesun commented Jul 1, 2024

Uh oh!

smilesun commented Jul 2, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smilesun commented Jul 2, 2024

Uh oh!

smilesun left a comment

Choose a reason for hiding this comment

Uh oh!

smilesun commented Jul 2, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented May 10, 2024•
edited
Loading

agisga commented Jun 27, 2024•
edited
Loading