Worker process exits when Netty frame decoding throws a non-IOException

[L1nq0]

While testing Storm 3.0.0 locally (official binary distribution, local Docker cluster, default configuration), we found that a worker process terminates whenever Netty frame decoding throws anything other than an IOException.

StormServerHandler.exceptionCaught (storm-client, messaging/netty/StormServerHandler.java) only tolerates IOException. Any other throwable raised while decoding a frame - for example IllegalArgumentException from Kryo for an unregistered class name, or KryoException on a truncated buffer - escalates to Utils.handleUncaughtException and terminates the worker process (exit code 20, then supervisor restart).

On a stock 3.0.0 cluster with default settings, a single malformed 27-byte frame sent to a worker port deterministically kills the worker before the message is routed to any task. We verified two independent frames, two kills, each within about three seconds.

Since this failure happens at the decode stage and involves no topology logic, widening the tolerated exception set for decode-stage errors (or skipping the offending frame with a log line) and reserving process termination for executor-stage failures would make workers resilient to arbitrary malformed input without changing behavior for valid traffic.

Happy to share our reproduction steps or a prototype patch if that would be useful.

[rzo1]

Can you try latest SNAPSHOT from nightlies.apache.org?

[L1nq0]

I tested the latest nightly build; the behavior is unchanged.

Build tested: apache-storm-3.0.1-SNAPSHOT.tar.gz from nightlies.apache.org (Last-Modified 2026-09-03 01:00 GMT), on Temurin 25.

Result: still reproduces. Two independent single 27-byte frames (a tuple referencing an unregistered class), two worker kills.
Each time:

• IllegalArgumentException: Class is not registered from Kryo.getRegistration via KryoTupleDeserializer, on the netty server
  thread (StormServerHandler.channelRead)
• Received error in netty thread.. terminating server... → the worker exits (code 20) after ~3s via the Halting after 3 seconds
  shutdown hook, and the supervisor relaunches it
• No Swallowing lines in the logs and the process actually dies, so it is neither the tolerated-skip nor the reset-but-alive path.

I also disassembled the shipped storm-client-3.0.1-SNAPSHOT.jar: StormServerHandler.ALLOWED_EXCEPTIONS still contains only
IOException, matching the observed behavior.

I can share the payloads, worker logs, and environment details if useful.

[rzo1]

Feel free to come up with a PR+tests if possible :)

[L1nq0]

Thanks — a PR with tests is doable. Before I start, the design question is which exceptions should follow the existing "log + close connection, worker survives" path instead of terminating the worker. Today exceptionCaught tolerates only IOException, and notably it already closes the connection after a tolerated exception, so "close the connection, not the worker" isn't a new mechanism — it's extending an existing one. The client-side handler (StormClientHandler.exceptionCaught) also never terminates the process, so this would bring the two sides closer.

Two directions:

1. Extend the tolerated set at StormServerHandler with the decode exception types. Smallest diff, but the observed failure arrives as IllegalArgumentException / IllegalStateException from Kryo's registration lookup, and those types are broad enough that extending the set with them could also downgrade genuine bugs elsewhere in the handler into connection-closes. KryoException alone wouldn't cover the observed case.

2. Catch at the deserialization boundary (inside the server's received/decode path, not in exceptionCaught): any throwable from tuple deserialization becomes the tolerated path — WARN log + ctx.close(), worker survives. Everything not from the decode stage keeps today's fail-fast semantics. Slightly larger diff, but precise: it can't mask bugs outside the decode path, and it covers every decode failure mode (KryoException, IllegalArgumentException, truncated buffers, future Kryo changes) without maintaining an exception-type list.

My lean is (2). Happy to hear your take — or if you see a better direction, I'll build that instead. Tests would cover: a decode failure closes the connection and the worker keeps serving, plus valid-traffic regression.

[rzo1]

ping @GGraziadei

[GGraziadei]

Hi, thanks for reporting!
Happy to receive these issues to improve the quality of the release