Follow-up to #175 / PR #186. Deliberately out of scope there (the orchestrator brief named three deliverables; this was recommendation 3).
Summary
PR #186 makes the ECOHERENCE contract honest — a client can now tell applied-but-unacknowledged from not-applied, and a non-mutating turn no longer inherits a write's coherence failure. What it does not do is recover.
If the Redis version key for a sandbox holds a non-integer (ERR value is not an integer) or the wrong type (WRONGTYPE), every subsequent write to that sandbox returns 503 ECOHERENCE forever. There is no path that resets the key or tears the session down, so the sandbox is permanently write-wedged until someone intervenes in Redis by hand.
Reproduction
From the load-test harness, against the shared Redis (db 9):
redis-cli -n 9 SET "vfs:default:ver:<sandbox>" not-an-integer
Every write exec on that sandbox then returns 503 ECOHERENCE with retryable: false. Correct per the new contract — the write is applied — but it never clears.
Why it matters
retryable: false is the right answer for one occurrence and the wrong answer as a steady state: it tells the client "do not retry", and nothing else ever fixes it. A single corrupt key takes a sandbox out of service indefinitely.
Suggested fix
On an INCR failure whose cause is a malformed key (ERR value is not an integer, WRONGTYPE) — as distinct from a transport failure — either reset the key to a safe value derived from the database, or invalidate the session so the next request rebuilds it. A transport failure should keep the current deferral behaviour; only a structurally invalid key should trigger recovery.
Provenance
Found while verifying #175 against the load-test harness.
Follow-up to #175 / PR #186. Deliberately out of scope there (the orchestrator brief named three deliverables; this was recommendation 3).
Summary
PR #186 makes the ECOHERENCE contract honest — a client can now tell applied-but-unacknowledged from not-applied, and a non-mutating turn no longer inherits a write's coherence failure. What it does not do is recover.
If the Redis version key for a sandbox holds a non-integer (
ERR value is not an integer) or the wrong type (WRONGTYPE), every subsequent write to that sandbox returns503 ECOHERENCEforever. There is no path that resets the key or tears the session down, so the sandbox is permanently write-wedged until someone intervenes in Redis by hand.Reproduction
From the load-test harness, against the shared Redis (db 9):
redis-cli -n 9 SET "vfs:default:ver:<sandbox>" not-an-integerEvery write exec on that sandbox then returns
503 ECOHERENCEwithretryable: false. Correct per the new contract — the write is applied — but it never clears.Why it matters
retryable: falseis the right answer for one occurrence and the wrong answer as a steady state: it tells the client "do not retry", and nothing else ever fixes it. A single corrupt key takes a sandbox out of service indefinitely.Suggested fix
On an INCR failure whose cause is a malformed key (
ERR value is not an integer,WRONGTYPE) — as distinct from a transport failure — either reset the key to a safe value derived from the database, or invalidate the session so the next request rebuilds it. A transport failure should keep the current deferral behaviour; only a structurally invalid key should trigger recovery.Provenance
Found while verifying #175 against the load-test harness.