Report each start's final objective value at the end of a multi-start fit (#658) by wshlavacek · Pull Request #674 · lanl/PyBNF

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek · 2026-08-30T03:34:22Z

Closes#658.

A fit that runs several searches from different starting points reports the best of them. That one number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else.

A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run.

Here is the file from a Powell fit with six starts on a two-mode test objective.

job_type	powell
starts	6
best_objective	0.6930237441
worst_objective	0.6931471806
median_objective	0.6930854654
starts_near_best	6
# rank	start	objective	iterations	evaluations	reason
1	1	0.6930237441	1	66	converged after 2 cycles (objective 0.693024)
2	3	0.6930237441	1	83	converged after 2 cycles (objective 0.693024)
3	4	0.6930237502	1	43	converged after 2 cycles (objective 0.693024)
4	2	0.6931471806	1	85	converged after 2 cycles (objective 0.693147)
5	5	0.6931471806	1	86	converged after 2 cycles (objective 0.693147)
6	6	0.6931471806	1	45	converged after 2 cycles (objective 0.693147)

The file also carries a short comment block above that explaining how to read it. On screen the run prints:

Multi-start summary: 6 starts. Best objective 0.693024, median 0.693085, worst 0.693147.
6 of the 6 came within 0.1% of the best. The more starts that reach the same low value, the more likely the fit has found the best answer available.
Per-start table, best first: output/Results/multistart_summary.txt

A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled. Otherwise a run cut short by wall_time_fit would print a short table that reads as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start, since there would be nothing to compare it against. No fitting method searches any differently.

What is covered

trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso.

The issue left the metaheuristics optional. They are included here because de is the default fit type, so leaving them out would have left the most common multi-start fit with no report. Those methods keep no per-start iteration count that means the same thing across all of them, so their iterations column reads n/a and the evaluations column carries the work each start cost instead.

A bug fixed along the way

A bootstrap replicate of a multiple shooting fit reuses the algorithm object, and the list of per-start ladder results was never cleared between replicates. Everything that reports on the ladder reads that list, so the second replicate could report a start belonging to the first, fitted to different resampled data, as the one behind Results/continuity_defects.txt and the best stage trace. The list is now cleared on reset. The new test fails without the fix.

Where the code went

The row type and the formatting are in a new module, pybnf/algorithms/multistart_report.py. Three families of fit type produce these rows and each keeps the numbers somewhere different, so the shared part sits apart from all three. Each family fills in Algorithm.multistart_records, and Algorithm._emit_multistart_summary writes whatever it is handed. A failure writing the file is logged and swallowed, as it is for the other end of run reports.

Testing

New tests in tests/test_multistart_summary.py cover the formatting on its own and all three families. The concurrent local optimizers and the metaheuristics are driven end to end through the real run loop against an analytical objective. Multiple shooting is checked white box, since that fit type needs a sensitivity capable simulation backend. One new test in tests/test_shooting_sbml.py covers the bootstrap bug.

Local runs: the default suite is 4745 passed and 14 skipped, the slow tier is 49 passed, and the gradient, profile likelihood and shooting modules pass with their recovery tests included. The docs build with no new warnings.

Several fit types run more than one search from different starting points and report the best result. That single number cannot be checked. Twenty starts that all reached about the same objective value mean the fit has very likely found the best answer available and that more starts would not help. Twenty starts that all landed somewhere different mean the reported answer is only the least bad of twenty poor ones. Both cases used to print one number and nothing else. A fit with more than one start now writes Results/multistart_summary.txt, one row per start sorted by final objective value from best to worst, and prints a short version of it at the end of the run. Each row gives the objective that start reached, the steps it took, the simulations it cost, and why it stopped. Reading the objective column downward is the check the parameter fitting literature calls a waterfall plot. This covers trf, lbfgs, gntr, powell, sim, ms, the polishing phase of profile_likelihood, and the metaheuristics de, ade, ss and pso. A start that was still running when the fit ended, and a start the fit never reached, are both listed and labelled, so a run stopped early by wall_time_fit cannot be misread as a complete set of starts that agreed with each other. Nothing is written for a fit with a single start. No fitting method searches any differently. This also fixes a bug found while doing the work. A bootstrap replicate of a multiple shooting fit kept the previous replicate's ladder results, so it could report a start belonging to the replicate before it in continuity_defects.txt and in the best stage trace.

wshlavacek merged commit aaffe9e into mainAug 30, 2026
9 checks passed

wshlavacek deleted the multistart-summary branch August 30, 2026 04:27