benchmarks: create the .net cache gen_networks.py exists to fill (#492) by wshlavacek · Pull Request #494 · lanl/bngsim

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek · 2026-08-27T17:01:53Z

Closes#492.

Taken as asked — this one is on a recovery path, so the mkdir and the errno
check land together.

The fix

ensure_cache_dir(NETS) runs before the header, so the cache: ... line the
pass prints names a directory that exists rather than one every write is about
to fail against.

$ rm -rf benchmarks/suites/ode_fullnet/nets
$ python gen_networks.py --models Lang2024 --workers 1
cache: .../ode_fullnet/nets # now true when printed
[1/1] ok 0.41s Lang_2024.bngl N=73 rxn=332

The errno check, paired as you asked

Once the directory is created first, the old failure is unreachable on a normal
run. What is left is the case mkdir cannot cover — the directory removed
while a multi-hour sweep is in flight — and that is what
describe_write_failure() is for:

.../ode_fullnet/nets (the .net cache directory) no longer exists -- it is
created at the start of the pass, so it was removed mid-run; the network
itself generated fine

Three details, matching the three things you said conspired to make it hard to
read:

It names the directory, not the file. And the row's status is now
cache_write_failed rather than a bare error, so the console tag alone
answers netgen-or-write without needing the detail at all. Nothing downstream
enumerates statuses — run_timing.py only compares against "ok".
The path comes first in the message, because the console truncates a
row's detail at 70 characters. That truncation is why the path was cut off in
your reproduction.
One correction here, and it may save you time later: the manifest did
carry the message, under detail rather than error. There is no error
key in a manifest row, so .get("error") is None for every row including
the healthy ones — the 585 ok rows have no error either. detail holds
up to 400 characters and had the full path all along. Checking a
netgen_timeout row in the current manifest:
```
{"status": "netgen_timeout", "detail": "BNG2.pl netgen timed out after 180.0s"}
```
So the report could have answered it; the key name was the obstacle, not the
record. I have not renamed or aliased the key — that would rewrite the shape
of a tracked 592-row manifest for a naming preference, which is your call to
make, not a fix to slip into this PR.

The test

python/tests/test_gen_networks_cache_dir.py — 6 cases, no BNG2.pl needed,
which is the point: it drives main() with a model filter matching nothing, so
the run reaches its "nothing to do" exit without generating anything. That is
the ordering under test — #492 was not that the cache was never created, it was
that it was not created first.

Negative-verified precisely: removing only the ensure_cache_dir call (leaving
the helper and everything else in place) fails that one test on its assertion,
with the run still printing cache: <path> for a path that does not exist. The
other five cover idempotence on the 585-network resume, and the three
classification branches — vanished directory, a real ENOENT with the directory
present, and a non-ENOENT PermissionError (a full disk or read-only mount is
not a missing directory).

For the run machine

This is safe to take before the measurement phase, on your reasoning: it changes
what happens when the cache is missing, and nothing about what is generated
once it exists. The generated .net bytes are untouched — the only new code
paths are a mkdir and an error-message branch that a successful run never
reaches.

The mkdir -p workaround stays valid, so nothing forces the timing of this.

Phase 1 of the full-network ODE benchmark writes every generated network into NETS = HERE / "nets" and never created that directory. On a machine where it did not exist -- a fresh checkout, or one where the cache had been cleaned to reclaim disk -- every model failed. Expensively, and that is the part worth naming: the generation work happens first and only the write fails, so six models spent 28 s to produce nothing, and the full corpus would spend hours the same way. A missing cache is also exactly the state in which this script is the documented recovery path, so the one script whose job is to *build* the cache was the one script that could not create it. The pass now calls ensure_cache_dir(NETS) before the header, so the "cache: ..." line it prints names a directory that exists rather than one every write is about to fail against. That makes the old failure unreachable on a normal run, which leaves the case it cannot cover: the directory removed *while* a multi-hour sweep is in flight. describe_write_failure() handles that one and says so in those words. It matters because the old message named the wrong thing -- FileNotFoundError on a .net path reads as "this model could not be generated", which is what a genuine BNG2.pl netgen failure looks like too -- and because the console truncates a row's detail, so the path the reader needed was cut off. The new detail leads with the directory, and the row's status is cache_write_failed rather than a bare error, so the console tag alone answers netgen-or-write without needing the detail at all. Nothing downstream enumerates statuses; run_timing.py only compares against "ok". test_gen_networks_cache_dir.py drives main() with a model filter that matches nothing, so it reaches the "nothing to do" exit without BNG2.pl and without generating anything -- which is the ordering under test. #492 was not that the cache was never created, it was that it was not created first. Removing only the ensure_cache_dir call fails that test on its assertion, with the run still printing "cache: <path>" for a path that does not exist. One correction to the issue's third point: the manifest did carry the message, under "detail" rather than "error" -- `.get("error")` is None because there is no such key. The 400-char detail held the full path all along; it was the 70-char console truncation that hid it.

wshlavacek mentioned this pull request Aug 27, 2026
benchmarks: let a scoped run write somewhere other than the committed report (#493) #495
Merged

wshlavacek merged commit f8be826 into mainAug 27, 2026
4 checks passed