yaml, grid search mnist

[lisab00]

No description provided.

[smilesun]

does this configuration work with master already?

[lisab00]

Yes, this already works on the master branch.
I'm still working on adding more models, e.g. hduva.

[lisab00]

• matchdg creates an error because the hyperparameters which must be sampled as an int (e.g. epochs_ctr) are converted to a float in the sampling process
• hduva also does not work (might be caused by the same error?)

[smilesun]

@lisab00 ,I have merged the shared parameter branch into master, and master into this branch, when you get time, maybe you can test if your yaml file works. I would say it is better to test this yaml file than the pacs. the pacs can be done on Friday so the week after we get the results.

[lisab00]

there seems to be a problem with jigen

2023-09-14T163129.952669.snakemake.log

[lisab00]

[smilesun]

So the conclusion is 3 epochs are not enough to get a good result ?

…

On Fri, 15 Sep 2023 at 09:24, Lisa ***@***.***> wrote: [image: bench_mnist_grid_graphic] <https://user-images.githubusercontent.com/73993822/268207167-bc31c13c-3b3c-4830-bc25-44e33a15580d.png> — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELKRSN4W5RSYNUIOVAAY3X2P7BJANCNFSM6AAAAAA4KPBLXQ> . You are receiving this because you commented.Message ID: ***@***.***>

[lisab00]

Should I run this file with 100 epochs on master to compare with fbopt (like you told me on slack)
or should I create a new file contaning all algorithms?

[smilesun]

All algorithm please

…

On Fri, 15 Sep 2023 at 09:51, Lisa ***@***.***> wrote: Should I run this file with 100 epochs on master to compare with fbopt (like you told me on slack) or should I create a new file contaning all algorithms? — Reply to this email directly, view it on GitHub <#348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELKRWZSXCKQFPPYQFOYKLX2QCJBANCNFSM6AAAAAA4KPBLXQ> . You are receiving this because you commented.Message ID: ***@***.***>

[lisab00]

I added another file with more algos, but it's not working yet

[smilesun]

I took a look at the file, shouldn't the file be based on the yaml file Carla has generated yesterday? Since that already include a lot of algorihtms and one only need to change the task and neural network?

[lisab00]

Which file did Carla create yesterday?

[smilesun]

Which file did Carla create yesterday?

maybe she did not push to any branch. But i think we could start using your new yaml file to make it work first, if a small one does not work, a big one won't work neither.

[smilesun]

I added another file with more algos, but it's not working yet

would be great to have the error message so i can look into it as well.

[lisab00]

[lisab00]

but the hyperparameter sampling itself should work, I tested it on master

[lisab00]

Also, I based my file on this one: https://github.com/marrlab/DomainLab/blob/master/examples/benchmark/benchmark_pacs_resnet.yaml

I included algos up until line 150, changed dataset and sampling mode

[smilesun]

why grid search would have a distribution?

[smilesun]

Which file did Carla create yesterday?

maybe she did not push to any branch. But i think we could start using your new yaml file to make it work first, if a small one does not work, a big one won't work neither.

@lisab00 , I was wrong, actually she pushed the changes to the master already

https://github.com/marrlab/DomainLab/blob/master/examples/benchmark/benchmark_pacs_resnet.yaml

[smilesun]

Which file did Carla create yesterday?

maybe she did not push to any branch. But i think we could start using your new yaml file to make it work first, if a small one does not work, a big one won't work neither.

@lisab00 , I was wrong, actually she pushed the changes to the master already

https://github.com/marrlab/DomainLab/blob/master/examples/benchmark/benchmark_pacs_resnet.yaml

but the file is not grid search, so there is distribution

[lisab00]

I thought you need distribution for grid search as well
https://github.com/marrlab/DomainLab/blob/master/examples/benchmark/benchmark_mnist_shared_hyper_grid.yaml

[smilesun]

I thought you need distribution for grid search as well https://github.com/marrlab/DomainLab/blob/master/examples/benchmark/benchmark_mnist_shared_hyper_grid.yaml

I agree, will take some deeper look into it

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.14%. Comparing base (0f5c7e4) to head (8e546d3).
⚠️ Report is 519 commits behind head on master.

Additional details and impacted files

@@ Coverage Diff @@## master #348 +/- ##
=======================================
Coverage 95.14% 95.14% =======================================
Files 124 124 Lines 4862 4862 =======================================
Hits 4626 4626 Misses 236 236 

Flag	Coverage Δ
unittests	95.14% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.