Appends and reads permanently fail on any stream where a single WAL-callback latency metric throws

[JayJamieson]

Was checking out your repo and do some comparisons against something i built and noticed that it breaks in some ways even without concurrency.

Appends and reads permanently fail on any stream where a single WAL-callback latency metric throws. Reproducible from a cold start with zero concurrency.

Root cause

1. S3Storage.append() (stream/src/main/java/io/streamstack/s3/S3Storage.java:328-337) times itself with System.nanoTime() and records the delta in a whenComplete callback that runs on a different thread (AUTOMQ_S3STREAM_APPEND_CALLBACK-N):

 final long startTime = System.nanoTime();
 ...
 return cf.whenComplete((nil, ex) -> {
 streamRecord.release();
 APPEND_STORAGE_LATENCY.record(TimerUtil.timeElapsedSince(startTime, TimeUnit.NANOSECONDS));
 });

Cross-thread nanoTime() subtraction occasionally goes negative on this host. DeltaHistogram.record() -> HdrHistogram Recorder.recordValue() throws:

java.lang.ArrayIndexOutOfBoundsException: Histogram recorded value cannot be negative.
 at org.HdrHistogram.AbstractHistogram.recordSingleValue(AbstractHistogram.java:559)
 at org.HdrHistogram.Recorder.recordValue(Recorder.java:136)
 at io.streamstack.s3.metrics.wrapper.DeltaHistogram.record(DeltaHistogram.java:108)
 at io.streamstack.s3.S3Storage.lambda$19(S3Storage.java:337)

2. CompletableFuture.whenComplete semantics turn this into a false failure. whenComplete returns a new future; if the action throws, the new future completes exceptionally even though the antecedent cf succeeded. The WAL write is durable, the caller is told it failed anyway, solely because the metrics callback threw.
3. S3Stream.append0() (stream/src/main/java/io/streamstack/s3/S3Stream.java:267-279) treats any exception, regardless of type, as grounds for permanent fencing:

return cf.whenComplete((rst, ex) -> {
 if (ex == null) return;
 status.markFenced(); // unconditional
 ...
});

3. Status.isWritable() / isReadable() both just check status == 0. There is no unfencing path. Once one bad sample fences a stream, every subsequent append and read on it fails permanently with:

io.streamstack.api.exceptions.StreamClientException: code: 1, [streamId=N epoch=1] stream is already closed

Impact

• Single writer, zero concurrency: 23.5% of appends failed (6,508/8,513 succeeded).
• 200 concurrent writers, 8 streams: 47.5% succeeded (
• All bounded catch-up reads failed (0/5,619, 0/9,350 across two runs).
• Reproducible on a fresh, empty in-memory (mem://) in

Suggested fix

• Don't let a whenComplete side effect (metrics) deterthe returned future, record latency without re-throwing into the chain (e.g. wrap record() in try/catch, or move it off the completion path).
• In S3Stream.append0, only fence on exceptions that amClientException with EXPIRED_STREAM_EPOCH); let otherfailures fail that single request without poisoning the stream.

Environment

• Commit: 88c9910 (2026-08-12)
• JDK 17, in-memory (mem://) storage/WAL backend, sing.mem.yaml

[addu390]

Hi @JayJamieson, thanks for reporting this. I fixed the metrics exception bubbling into append futures. I didn’t however change fencing to only handle expired epochs yet, because offsets advance before storage completes, continuing after a genuine failure might create gaps or ambiguous durability. I'd say that requires a separate review/discussion of retry and offset semantics.

[addu390]

PR: #2

[JayJamieson]

Nice, I'll have another try with latest changes in master. I'm not familiar with Raft or Java but I'll take a look anyway and see how hard it would be to fix.

FWIW I am comparing against yasdb and objwal. The way I handle append failures and offset/seq numbers is by waiting for a writes to become durable before returning to the caller. This way even if offsets advance and a failure happens, a caller can't see the gaps nor will anything stored durably reflect that.

[addu390]

Thanks for sharing! I'll checkout yasdb.

StreamStack does wait for durability before acking the caller, no client ever sees a successful append that isn't durable. What advances early is only the internal sequencing, so multiple batches can be pipelined per stream. Otherwise the per-stream throughput drops significantly.

Saw the notifier mode in yasdb, same idea, although StreamStack pipelines from the client side so a single writer can saturate a stream. The real question then is what happens to the sequenced-but-not-durable range after a failure. On my side everything queued behind the failed write fails too and the stream gets fenced, so curious how you handle that range?

[JayJamieson]

I don't fence, because the failure model only accounts for the whole store closing/failing as a while. SlateDB is a single-writer, so a commit failure is treated as storage failure, and that path already drains and fails everything cleanly. There was never a reason to build a separate per-stream fence as it was never in the durable stream spec.

Does it matter? probably not in practice, but I have not proven it. Sequence assignment for the next burst runs ahead of the actual commit call, and isn't rolled back on failure, an isolated single-stream failure (store otherwise healthy) would silently skip that offset range rather than fence it. I on SlateDB's single-writer property to make that scenario not really occur independently of a whole store failure, but it's untested and i have tried to create a scenario of "one stream's commit fails, store stays up."

Only thing I've really tested and verified is linearizability with porcupine, though I'm not 100% sure how sound the test is in terms of correctness.

[addu390]

Ahh! Makes sense given SlateDB's single-writer model, if a commit failure means the whole store is failing, then draining and failing everything gives you the same protection as fencing, just at store level instead of per stream.

The difference on my side is that the failure domain is smaller. StreamStack writes through a shared S3-backed WAL but appends can fail for a single stream while the node and store stay healthy, so I can't fold that into store failure. That's why the per-stream fence exists: the sequenced-but-not-durable range can't be re-sequenced into the gap, so fencing forces a clean re-open at the durable tail instead of skipping the range.

The multi-node cluster setup adds to that too. Stream metadata is raft-replicated across nodes, so a stream can move to another node after a failure, and the fence guarantees the new owner starts from a tail that's actually durable.

[JayJamieson]

Makes sense. FWIW This can probably be closed now since the fix landed on main. Re ran my tests and couldn't reproduce.

[addu390]

Sounds good, closing it out. Don’t hesitate to open another issue if you hit something else, or drop suggestions/feedback anytime.