core: status page exceeds the API rate limit, leaving cards permanently loading

[andre8244]

Component and version

ns8-core main (d4435d58) — affects api-server, the cluster-admin UI and the Python agent.

Introduced by the rate limiter added in dcc0a3fc (2026-07-16).

Steps to reproduce

1. Install a single-node NS8 cluster from main (latest version).
2. Log in to https://<leader>/cluster-admin/ and open the browser DevTools Network panel.
3. Clear the browser cache, then navigate to Cluster status and hard-reload the page.
4. Watch the Network panel for requests failing with HTTP 429.

Expected behavior

No request fails with 429 and every Status card finishes loading.

Actual behavior

Some requests fail with 429, and the cards whose task-status call was throttled might stay in the loading state indefinitely.

Analysis

A cold load of #/status fires more same-origin requests than the limiter's burst allows, and the failures begin while the initial burst is still in flight — so the burst ceiling is the binding constraint, not the sustained rate. Burst 100 was mirrored from nethsecurity-controller PR #282 rather than derived from cluster-admin's real request profile.

Three things inflate the request count:

• Webpack prefetch — Vue CLI's default prefetch fires all prefetch-only chunks immediately instead of deferring them to browser idle time.
• Duplicate /context requests — notification.js checks the cache, awaits the HTTP call and only writes the cache afterwards, while websocket.js handleTaskMessage calls the async handler without awaiting it. Every progress frame of the same task therefore enters the cache-miss branch concurrently and refetches the same context.
• Uncacheable static assets — assets are served with no Cache-Control and no ETag (only Last-Modified), so most of them are re-downloaded in full even on a warm reload.

Two defects turn a transient 429 into a permanently broken page:

• core/ui/src/mixins/notification.js — the getTaskStatus error branch creates a notification but has no return, then dereferences statusResponse.data.data on an undefined response → TypeError → the *-completed event never fires and the card spins forever.
• core/imageroot/usr/local/agent/pypkg/agent/tasks/apiclient.py — http_temporary_errors = [500, 502, 503, 504] omits 429, so _retry_request re-raises instead of using its existing exponential backoff. Nested tasks call the same rate-limited HTTP API (agent/tasks/run.py defaults to http://cluster-leader:9311), so throttling fails whole tasks.

The 429 response also carries no Retry-After header, so clients cannot back off intelligently.

Suggested fix or workaround

Workaround: set GLOBAL_RATE_LIMIT_AVERAGE=0 in /etc/nethserver/api-server.env to disable the limiter.

Efficacy below is judged against the arithmetic above: a fix either removes requests from the first ~2 s, or raises the ceiling, or it does nothing for this bug. "first" / "repeat" = cold vs. warm browser cache.

#	Fix	Feasibility	Complexity	Efficacy on the 429	Verdict
A	Differentiated limits: strict bucket on /api/login, raise global burst	High	Low	Decisive — first + repeat	Do first (needs author review)
B	Add missing return in the getTaskStatus error branch (notification.js)	High	1 line	None on 429; removes the permanent hang	Do
C	Add 429 to http_temporary_errors (apiclient.py)	High	1 line	None on browser 429; stops task failures	Do
D	Coalesce in-flight /context requests	High	Low	High — first + repeat	Do
E	Send Retry-After on the 429	High	Low	None alone; enables H	Do
F	Don't re-arm the poll timer on 429 (notification.js)	High	Low	Medium — stops amplification	Do
G	Cache-Control on static assets	High	Medium	High on repeat, zero on first	Do (after A–F)
H	429 retry with backoff in the axios interceptor	Medium	Medium	High on user-visible outcome	Do, carefully
I	Disable or trim webpack prefetch	High	1 line	High — first + repeat	Judgement call
J	De-duplicate redundant task creation	Medium	Medium	Medium	Optional
K	Serve precompressed assets	Medium	High	Zero — CPU only, not request count	Defer

Notes on the non-obvious rows:

• A — needs review by the author of dcc0a3fc before anyone implements it. Raising the burst is close to free in flood-resistance terms, because sustained throughput is what bounds a flood: over 60 s an attacker lands 100 + 25×60 = 1600 requests at burst 100 vs. 300 + 25×60 = 1800 at burst 300 (+12%). What a bigger burst does increase is instantaneous concurrency on the one genuinely expensive pre-auth path, /api/login (password hashing) — so the proposal is not a blanket bump but a stricter per-route bucket on login (e.g. 5 rps / burst 20, stronger than today) plus a global burst of 300. RateLimiter is already a self-contained closure with its own visitors map, so it works as per-route middleware with no refactor.
• G does nothing on a first-ever load — it only removes repeat-load requests. Requires a carve-out: index.html and config/config.production.js must stay no-cache, because install-coreimage unlinks content-hashed chunks that disappear on a core update, so a stale index.html would reference files that no longer exist.
• H must retry GET only. POST /cluster/tasks is not idempotent and blind retry would create duplicate tasks.
• I is stock Vue CLI behaviour, not a misconfiguration — removing it trades slower first navigation to lazy routes for a large drop in request count. With A in place it is no longer needed to fix the bug, so it is a product decision.
• K has zero efficacy here because the limiter counts requests, not bytes or CPU. It is a genuine CPU improvement (gzip.Gzip wraps static.Serve, recompressing every asset on every request) but gin-contrib/gzip is pinned at v0.0.6, which has no precompressed support — and neither does v1.2.6. Better as a standalone performance PR; G largely obviates it.

Relevant logs or output

Throttled response (no Retry-After):

HTTP/2 429
content-type: application/json; charset=utf-8

{"code":429,"data":null,"message":"too many requests"}

Limiter parameters confirmed empirically — 800 parallel GETs completing over 25.2 s yielded 729 successes, matching burst + rate × elapsed = 100 + 25 × 25.2 = 730.

Duplicate /context fetches for a single task, from the Network panel:

GET /api/node/1/task/33c2edbb-.../context 200
GET /api/node/1/task/33c2edbb-.../context 200
GET /api/node/1/task/33c2edbb-.../context 200
GET /api/node/1/task/33c2edbb-.../context 200

[andre8244]

QA

• Testing version: core 3.21.1-dev.3
• Make sure the bug is no longer reproducible