Add integration tests for memory safety and concurrency

[jeeyo]

Groundwork for the memory-safety work: make the server's behaviour observable before changing any of it.

Unblocking the build

3rdparty/quickjs pointed at github.com/jeeyo/quickjs, which returns 404, so the default build couldn't be checked out. Switched to quickjs-ng v0.16.2 and adapted the port: JSMallocFunctions gained js_calloc and passes void *opaque instead of JSMallocState *, libbf.c/cutils.c are gone in favour of dtoa.c, JS_IsError() is single-argument, and JS_BOOL is now bool.

quickjs-ng also keeps its internal assertions on, which is how the first bug below became visible as a clean abort rather than silent corruption.

The tests

Black-box: they run the real isere binary and speak HTTP to it.

File	Covers
test_smoke.py	Startup, routing, malformed input, abrupt disconnects
test_concurrency.py	Latency distribution, concurrent clients, fd leaks, event-loop starvation
test_memory.py	RSS stability across requests, connect/disconnect, partial requests, timers, large bodies
test_risky_js.py	Hostile handler code — throwing, embedded NULs, non-string headers, oversized headers, pipelining
test_leaks.py	valgrind (marked slow)

Two things worth calling out about the design:

• The harness compiles each test's .js into handler.so with the same xxd/sed recipe as the root CMakeLists.txt, so what's tested matches a normal build.
• Leak detection compares memory retained at two different request counts rather than reading one run. Most of what isère leaks stays reachable — a leaked FreeRTOS timer block is still linked into a kernel list — so it never shows as "definitely lost", and RSS is too coarse to see it. Growth against request count is the signal that actually works.

Failing tests — these are findings, not broken tests

1. Pipelined requests abort the process. Two requests in one TCP segment:

isere: quickjs.c:2704: JS_FreeRuntime: Assertion `list_empty(&rt->gc_obj_list)' failed.

__on_message_complete (src/httpd.c:196) calls __httpd_cleanup_conn, which vPortFrees the connection — from inside an llhttp callback, while llhttp_execute still holds &conn->llhttp. On the run before this one the same input left the process alive but no longer accepting connections. Timing-dependent behaviour from one input is the usual signature of memory corruption.

2. Response bodies can contain adjacent heap. A handler returning a body with an embedded NUL gets body_len from the JS string (which counts the NUL) but a buffer from strdup() (which stops at it). Writeback sends body_len bytes from the shorter allocation. The test captured what look like live 64-bit heap pointers on the wire — remotely observable, and enough to defeat ASLR.

3. One slow handler blocks every other client. A handler awaiting a timer stalls an unrelated request for ~6s. ISERE_HTTPD_HANDLER_TIMEOUT_MS is declared at include/httpd.h:26 and never read.

4. The drain loop busy-waits without yielding. Under valgrind's thread serialisation, a handler waiting on a FreeRTOS timer starves the timer task and the server stops accepting entirely. It passes without valgrind, so this is about the loop never yielding rather than valgrind itself.

What passes

33/35 fast tests, 6/8 slow. RSS is flat across 1000 requests, connect/disconnect cycles, partial requests, 64 KiB bodies and allocation-heavy handlers; no fd leaks. Latency: p50 0.4 ms serial, 2.4 ms at 8 concurrent clients; at 40 clients (past the 12-connection cap) p95 goes to 1.0 s and throughput drops to ~96 req/s.

CI

Split into build / integration / leaks. The previous workflow installed CppUTest and ran ./unittests — a target that no longer exists in CMakeLists.txt, so that step could never pass. Those steps are removed here; tests/loader_test.cpp is still stale and unreferenced, and is left for a follow-up.

Fixes are deliberately not in this PR.

🤖 Generated with Claude Code

[jeeyo]

Reporting added

The pass/fail list didn't explain what any test was for, so every case now carries structured metadata and there's an HTML report built on top.

Metadata. Each test has a case marker — stable id (RISK-11, MEM-04…), title, area, severity, what a failure would mean, and the source lines it exercises — with the docstring as the description.

JUnit XML stays the standard artifact, and each <testcase> now carries that metadata as <properties>, so it's self-describing for anything reading it directly. This needs junit_family=xunit1; xunit2 silently drops properties.

HTML report (tests/integration/report.py) merges JUnit results with a catalogue written at collection time — before-m/-k deselection. So it documents all 44 tests whether or not a given run executed them; skipped ones show as "not run" with their documentation intact. Failures are listed up front ordered by severity, then everything is grouped by area.

Captured output is included, because for these tests the numbers are the result, not the green tick:

CONC-01 Serial latency baseline PASSED 0.10s
One client, no concurrency -- the floor for per-request latency.
If this fails: Per-request latency regressed badly, or the server is
doing something blocking on the happy path.
Exercises: src/httpd.c:423
[serial] ok=50 fail=0 wall=0.02s throughput=2622.1 req/s
p50=0.4ms p95=0.5ms max=0.8ms

--junit is repeatable, so the fast and slow runs merge into a single report. A new report CI job runs on always() and uploads it as the test-report-html artifact.

Full run

44 tests — 40 passed, 4 failed. Failures by severity:

	Case	What it means
critical	RISK-02	Response body with an embedded NUL discloses adjacent heap — live pointers observed on the wire
critical	RISK-11	Two pipelined requests abort the process in JS_FreeRuntime
high	CONC-06	One slow handler blocks unrelated clients for ~6s
medium	LEAK-04	Fails, but not for the leak it targets — under valgrind's thread serialisation the drain loop busy-waits without yielding and starves the timer task. Passes without valgrind, so the finding is the missing yield.

Areas: Smoke 11 · Concurrency 6 · Memory 8 · Hostile handler code 12 · Leaks 7.

Also dropped a stray __pycache__ that got committed with the first push; the net branch diff is clean.

🤖 Generated with Claude Code