My other machine-learning projects call a model. This one is the layer underneath them: the thing that decides which version of a model answers a request, groups concurrent requests into one forward pass, and notices when the inputs stop looking like the data the model was trained on.
I wrote the scorer too, rather than shelling out to a Python process. It loads
models saved by XGBoost's own save_model(*.json) and reproduces XGBoost's
predictions in pure Go — and the fixtures that prove it are generated by
XGBoost itself, which is how it caught a bug I would never have found by
reading the format.
XGBoost compares a feature against a split threshold in float32, and writes
split_conditions to JSON at float32 precision. So a threshold reads back as
-0.3775961, while the feature value that produced it — a float32 widened to
float64 — is -0.37759611010551453.
Compared as float64 those differ and the row goes left. Compared as float32
they are the same number, < is false, and it goes right, which is what
XGBoost does.
It only matters for a row sitting exactly on a threshold. But thresholds are chosen from training values, so rows land on them constantly: one row in 64 in the binary fixture, seven in the regression one. The symptom is the worst kind there is — predictions correct for ~99% of traffic and quietly wrong for the rest, with nothing downstream able to tell. No amount of reading the format finds that. Only running both implementations against the same rows does.
The second finding was that base_score lives in two different spaces:
single-output models store it in prediction space, so the margin intercept is
the objective's output transform run backwards — logit(0.25) for binary
logistic, log(1.16) for Poisson — while multi-class stores a vector already in
margin space. The binary fixture deliberately uses a base_score other than
0.5, because logit(0.5) is 0 and would make an intercept applied in the wrong
space look correct.
Seven fixtures cover binary logistic, missing values, squared-error regression, Poisson, multi-class softprob and softmax, and a hand-checkable stump. A CI job reinstalls XGBoost, regenerates them from scratch and re-runs the parity tests, so a change in how XGBoost scores a tree surfaces as a failure rather than as slow divergence. It compares predictions rather than files: retraining produces different trees, and the claim under test is that the scorer agrees with XGBoost, not that training is reproducible.
A registry where a version is immutable. Artifacts are named by the SHA-256 of their contents, so a name cannot be reused for different bytes and uploading the same model twice is a no-op rather than a second copy. That is what makes "roll back to version 3" a meaningful instruction: version 3 today is the same bytes and the same input contract as version 3 last week. Version numbers are assigned by the database under a row lock on the model, so concurrent registrations of one model serialise while different models never contend — a test fires twelve simultaneous registrations and asserts the numbers come back as exactly 1..12, no duplicates and no gaps.
Dynamic batching. Scoring one row is about a microsecond; the HTTP handling, decoding, routing and metrics around it are considerably more, so under concurrent load the per-call overhead sets throughput. Measured against a scorer with 200µs of fixed per-call cost and 320 concurrent callers: 59.5µs/op at batch size 1 against 1.38µs/op at batch size 64, a 43x difference.
It groups on real traffic, not only in a benchmark. 3000 requests from 48 concurrent clients through the HTTP API:
batching 3000 rows in 136 batches, mean 22.1, largest 32
Send those same 3000 requests sequentially with curl and the mean is exactly
1.0 — there is nothing to batch when nothing overlaps, which is the honest
behaviour and the reason stats reports the mean at all.
The window opens when the first request of a batch arrives, not on a fixed tick. A ticker looks equivalent and is not — a request arriving just after a tick waits a full interval, the tick fires on an empty queue when traffic is light, and the added latency belongs to the clock rather than to the request. Opening it per batch makes the delay a genuine ceiling: an idle system answers a lone request after exactly one window, a busy one dispatches full batches without waiting at all.
Canary splits that are stable per entity. Version assignment is a deterministic function of a caller-supplied key. If a user flips between control and canary from request to request, every per-user metric mixes both models and the experiment measures nothing. The model name is mixed into the hash as well, so an entity in the canary bucket for one model is not thereby in the canary bucket for every model — otherwise the experiments correlate and one bad canary appears to hit the same unlucky users everywhere.
Shadow deployments. A candidate version receives a copy of every request; its prediction is compared against the served one and discarded. It runs on its own goroutine with its own context — inheriting the request's context would kill it the moment the handler returned, which is to say on nearly every request, and running it inline would put the candidate's latency into production's. A shadow that fails is recorded, never propagated.
A rollout guard. It removes a version whose error rate crosses a threshold, but only after a minimum number of requests: the first request to fail puts a version at a 100% error rate, and the start of a rollout is the smallest sample there is. It sets the weight to zero rather than deleting the route, so an operator can see a version was pulled rather than guessing why it is absent. And it will never remove the last version receiving traffic — turning "this version is failing" into "this model serves nothing" is strictly worse.
Drift detection. PSI over quantile bins, on a sliding window of bin counts rather than retained samples, so memory depends on the window rather than on how busy the service is. Missing values are counted separately instead of binned: a feature that becomes 40% NaN is a serious problem, and folding those rows into a bucket would both hide it and shift the value distribution as the missing rate changed — reporting a broken pipeline as a change in the values.
Needs Go 1.24+, Docker (for Postgres), and Python with XGBoost only if you want to regenerate fixtures.
make db # Postgres on :5432, plus the test database
make run # mints development tokens, then serves on :8080Every host port the compose stack publishes is overridable, which matters because 5432 and 3000 are usually already taken by something else. The variables are exported by the Makefile, so setting one moves the published port and the connection strings that point at it:
POSTGRES_HOST_PORT=5433 GRAFANA_HOST_PORT=3001 make db run| Service | Default | Override |
|---|---|---|
| Postgres | localhost:5432 | POSTGRES_HOST_PORT |
| Prometheus | http://localhost:9090 | PROMETHEUS_HOST_PORT |
| Grafana | http://localhost:3000 | GRAFANA_HOST_PORT |
Or copy .env.example to .env and edit it, which the Makefile reads. A
variable set in the environment still beats the file, so the inline form above
overrides a .env rather than fighting it.
Only the host side moves; inside the compose network the services still reach each other on the standard ports.
make run writes deploy/tokens on first use — the server's reloadable token
file — plus deploy/dev-tokens.env holding the matching admin token for the
CLI. Source it before using the client:
source deploy/dev-tokens.envIf port 5432 is already spoken for, or you would rather not run Postgres in
Docker at all, skip make db and point everything at your own instance. The
server reads MODELFORGE_DATABASE_URL and the tests read
MODELFORGE_TEST_DATABASE_URL; both need to be separate databases, because the
suite truncates its own between cases.
export MODELFORGE_DATABASE_URL="postgres://you@localhost:5433/modelforge?sslmode=disable"
export MODELFORGE_TEST_DATABASE_URL="postgres://you@localhost:5433/modelforge_test?sslmode=disable"
make run # and make testThe server creates its own schema on startup, so an empty database is all it
needs. Every flag has a matching MODELFORGE_-prefixed environment variable —
./bin/modelforge -h lists them.
Train something and deploy it:
python3 - <<'PY'
import numpy as np, xgboost as xgb
X = np.random.randn(2000, 4); y = (X[:,0] + X[:,1] > 0).astype(int)
xgb.XGBClassifier(n_estimators=30, max_depth=4).fit(X, y).save_model("fraud.json")
np.savetxt("train.csv", X, delimiter=",")
PY
./bin/modelforgectl create fraud-score "scores a transaction"
./bin/modelforgectl push fraud-score fraud.json amount age tenure disputes
./bin/modelforgectl deploy fraud-score 1
curl -s localhost:8080/v1/models/fraud-score/predict \
-H "Authorization: Bearer $MODELFORGE_TOKEN" \
-d '{"features":{"amount":1.2,"age":-0.3,"tenure":0.8,"disputes":-1.1},"key":"user-42"}'
# {"model":"fraud-score","version":1,"prediction":[0.9976470962194207]}Roll a second version out gradually, watch it, then commit or revert:
./bin/modelforgectl push fraud-score fraud-v2.json amount age tenure disputes
./bin/modelforgectl shadow fraud-score 2 # mirror traffic, serve nothing from it
./bin/modelforgectl canary fraud-score 1 2 10 # 10% of entities to v2
./bin/modelforgectl stats fraud-score 2
./bin/modelforgectl deploy fraud-score 2 # promote
./bin/modelforgectl rollback fraud-score 1 # or go backdeploy and canary leave an existing shadow in place, since shadow already
leaves the serving split alone — otherwise promoting a candidate would silently
stop whatever else was being watched, and you would find out by noticing an
empty graph. The exception is promoting the shadow itself, which clears it and
says so: the server refuses a policy where a version is both shadow and
serving, because shadowing a version against itself produces a divergence rate
that mixes two different comparisons and means nothing.
rollback is deliberately the same operation as deploy under a name that says
what it is for. A rollback needing different mechanics from a deploy is a
rollback nobody trusts at 3am, and the point of immutable versions is that going
back is just going forward to an older number.
Turn on drift monitoring by giving a version the distributions it was trained on:
./bin/modelforgectl baseline fraud-score 1 baseline.json
./bin/modelforgectl drift fraud-score 1Real output from the smoke run in this repo, against traffic where amount was
shifted by about 1.2 sigma and tenure by 0.35, with the other two left alone:
realistic:1 over 15m0s (3000 samples)
FEATURE PSI SEVERITY MISSING
amount 1.4064 significant 0.0%
tenure 0.1306 moderate 0.0%
disputes 0.0058 stable 0.0%
age 0.0038 stable 0.0%
The two features that did not move read as stable rather than as mild noise, which is what makes the significant one worth paging on.
baseline.json is {"samples": {"amount": [...], ...}, "bins": 10} — raw
training samples per feature, not pre-computed bins. The binning rules
(quantile edges, tie collapsing, the empty-bin correction) are the server's, and
a client computing its own would silently disagree with the next server version.
Every route except the health probes needs a bearer token, and the server holds only SHA-256 digests of those tokens — never the tokens themselves — so a leaked config file or a leaked environment does not hand anybody a working credential.
modelforgectl token ci admin
# token (shown once, store it now):
#
# <43 characters of base64url>
#
# add this to the server's MODELFORGE_TOKENS:
#
# ci:admin:<64 hex characters>The values above are elided on purpose. A README that printed a real token
beside its real digest would be publishing a working admin credential, and
anybody who pasted that digest into their own MODELFORGE_TOKENS would be
trusting a secret that is on GitHub.
Plain SHA-256, not bcrypt or argon2, and that is deliberate rather than an oversight. Password hashes are slow on purpose, to make brute force expensive over the small guessable space human-chosen passwords occupy. These tokens are 256 bits of cryptographic randomness: there is no space to brute force, so a slow hash would buy nothing and would put its cost on every single request. The threat a password hash defends against does not exist here.
The comparison is timing-safe because the caller's guess is hashed before the lookup. The attack a constant-time compare exists to stop is an early-exit byte-by-byte comparison against a secret, which leaks how many leading bytes were right. Hashing first means flipping one bit of a guess changes every bit of the key, so a near miss and a wild miss are indistinguishable — there is no "how close was I" signal to measure. Adding a constant-time compare after that lookup would compare a value against itself and prove nothing.
| Scope | Can do |
|---|---|
predict |
call the serving endpoint, nothing else |
read |
inspect models, versions, policies, drift, stats, /metrics |
admin |
everything, including changing what serves traffic |
admin implies the other two; read and predict imply nothing. The split is
what lets the credential shipped to a high-volume caller score and only score,
so leaking it does not also expose which models exist and how they are deployed
— and lets a dashboard read everything without being able to change what serves.
A whole-table test asserts, for every route, which of the three tokens gets in and which gets a 403. Writing it as one table rather than scattered assertions is what makes a route added later without a scope show up as a missing row.
A missing or unrecognised token is 401 with a WWW-Authenticate challenge:
the caller has not proved who they are, and retrying with a credential is the
fix. A valid token without the scope is 403: they have proved who they are
and the answer is still no, so retrying is pointless and somebody has to grant
the scope. Collapsing both into one status sends operators hunting a permissions
problem when the real one is an unset environment variable. The CLI says which
is which rather than making you read the server's logs.
Starting with no tokens and no explicit opt-out is refused, not defaulted to open:
modelforge: no API tokens configured. Mint one with `modelforgectl token <name> <scope>`
and set MODELFORGE_TOKENS, or pass -auth-disabled to run without authentication deliberately
Getting this wrong is silent in the worst way — the server comes up, serves, and
looks entirely healthy while its control plane is open to anyone who can reach
the port. No log line anybody actually reads beats simply not starting.
-auth-disabled exists because local development and the test suite genuinely
do not want tokens, and a flag somebody had to type is very different from a
default nobody chose. Setting both is rejected rather than resolved, because
guessing which half of a contradictory security configuration was meant is
exactly the wrong instinct.
Tokens come from a file that is re-read on SIGHUP and on a timer, so replacing
a credential never involves restarting the process:
modelforgectl token ci-next admin # mint the replacement
echo 'ci-next:admin:<digest>' >> deploy/tokens # both valid now
kill -HUP $(pgrep -f bin/modelforge) # overlap begins
# ... move clients onto the new token at their own pace ...
sed -i '/^ci:admin:/d' deploy/tokens # withdraw the old one
kill -HUP $(pgrep -f bin/modelforge)make rotate does exactly this for the development token.
The overlap is the point. A single-step swap breaks every client that has not been updated yet, which is why rotation gets deferred until a credential is years old. Two steps with both valid in between makes it something you can do on a Tuesday.
Measured on a running server: 400 requests during 40 live reloads, all 200, same PID throughout. The token set is swapped through an atomic pointer, so a request sees exactly one generation of it — never a state where the new token is present and the old one is already gone. The read path takes no lock, so rotation costs serving nothing.
A bad rotation is not an outage. The file is read, parsed and validated
before anything is swapped, so a malformed entry, an empty file, a file that is
briefly missing mid-rewrite, or one that got entirely commented out leaves the
running set in force. It logs, counts modelforge_auth_reloads_total{outcome="failed"},
and keeps serving. There is an alert for that counter, because a failed reload is
invisible from the outside — the server looks healthy and the rotation simply did
not happen.
Why a file and not an admin endpoint. An API that mints tokens turns any admin compromise into permanent access: the attacker issues a credential of their own and keeps it after the stolen one is revoked. Keeping issuance outside the serving process means this server only ever learns digests. It is also the only mechanism that works with a Kubernetes Secret or a Vault agent, both of which rewrite a mounted file in place and neither of which can call an API — which is why there is a poll as well as a signal.
An entry may carry an RFC 3339 deadline as a fourth field:
ci-temp:read:<digest>:2026-12-01T00:00:00Z
It is checked on every request rather than at load time, so the credential stops working at its deadline whether or not anybody reloads. That is what makes "we will clean up the old token later" safe — later happens on its own. An expired token gets a distinct error from an unrecognised one, which leaks nothing the holder does not already know and is the difference between a minute of rotation and an hour hunting a phantom typo.
Static tokens answer "which service is this". They cannot answer "which person changed the model that scores production traffic", because everybody with the credential looks identical. Pointing the server at an identity provider fixes that:
modelforge \
-oidc-issuer https://idp.example.com \
-oidc-audience modelforge \
-oidc-scope-map 'platform-oncall=admin,ml-eng=read,scorers=predict'A user presents their ID token as the bearer credential. The signature is verified against the provider's published keys, the claims decide what they may do, and the audit log names them:
{"msg":"authorised change","actor":"sahil@example.com","method":"POST",
"path":"/v1/models","subject":"sub-sahil@example.com",
"issuer":"https://idp.example.com","kind":"user"}
Both the readable name and the stable sub are recorded. An email is what makes
that line answerable without a directory lookup; a subject is what still joins
correctly after somebody's surname changes.
Static tokens and user logins coexist. A service scoring a thousand requests a second wants a credential it holds, not an interactive login. A person changing what serves traffic should be named rather than sharing a token with everyone else who has one. Both work at once, and the two are told apart by shape — a JWS has three dot-separated segments and a minted token has none — so a mistyped static token never has an RSA verification attempted against it.
Authenticated is not authorised. A user in no mapped group is refused. The alternative is that every employee at the company gets whatever the fallback is, on the service that decides which model answers production traffic.
That refusal is a 403, not a 401, and getting this wrong is a genuinely confusing outage: the holder's signature checked out and their provider vouched for them, so telling them to re-authenticate sends a real employee round a loop against a provider that is working perfectly, when the fix is somebody adding them to a group. It does not spend rate-limit budget either — it is a permission problem, not somebody guessing. A forged token does spend budget, because that is somebody guessing.
The audience is mandatory and there is no "any audience" setting. Skipping it is the classic confused-deputy bug: an identity provider mints tokens for many services, and a token issued for the expense tool would otherwise be a valid credential here. The user is real, the token is real, and they never intended to authenticate against this.
modelforgectl login
# to sign in, visit:
#
# https://idp.example.com/authorize?...
#
# opening your browser at https://idp.example.com...
# signed in as sahil@example.com (admin)
# this session expires at 2026-08-31T20:34:32Z
modelforgectl whoami
# sahil@example.com (user)
# scopes admin
# subject sub-sahil
# issuer https://idp.example.com
# expires 2026-08-31T20:34:32ZThe credential is stored at ~/.config/modelforge/credential, mode 0600 in a
0700 directory — both set explicitly rather than left to the umask, since a
umask of 022 leaves a readable credential in a home directory other accounts on
the machine can list. An existing directory is tightened rather than trusted,
because MkdirAll leaves the mode of one that already exists alone.
MODELFORGE_TOKEN still wins if set, so a script can override whoever happens
to be signed in on the machine it runs on.
--no-browser prints the URL and waits, for a headless box, a remote shell, or
anyone who would rather paste a URL into a browser they trust than have a
process launch one.
This is the RFC 8252 native-application flow, and its shape is dictated by what a command-line tool can safely do:
- No client secret. One shipped inside a binary is a string every user can read out of it. The client is public and PKCE takes the secret's place — the test provider recomputes the challenge from the verifier and refuses a mismatch, so the CLI cannot ship without it and still pass.
stateon every request, checked before the code is touched. Without it anybody who can make your browser load a URL can complete a login of their choosing, and you end up holding their session.noncebound to the login, so a token captured from somewhere else cannot be replayed into it.- Redirect to
127.0.0.1, notlocalhost.localhostresolves through DNS, and a hostile resolver points it elsewhere — at which point the authorization code is delivered to somebody else's machine. The literal address cannot be redirected. The port is ephemeral, which RFC 8252 requires providers to accept for exactly this reason. - The callback page interpolates nothing. A provider's
error_descriptionreflected into it would be cross-site scripting on a page that has just handled an authorization code.
The server advertises where to sign in at GET /v1/auth/config, which is
deliberately unauthenticated: a client cannot present a credential before it
knows where to obtain one, and every value there is visible to anybody watching
a browser perform a login.
login requests offline_access, and the stored credential carries a refresh
token. An expiring session is renewed before the command that needed it runs,
so nothing fails first:
$ modelforgectl whoami # ID token has 55s left
sahil@example.com (user) # renewed transparently; no sign-in
Renewal is proactive, not on a 401. Reacting to a rejection means the first call after expiry always fails once, and "it works on the second try" is a bad habit for a tool to teach. The margin is 60 seconds, covering clock skew against the provider plus the time the request itself spends in flight — a token with two seconds left is not usable even though it has not technically expired. A credential that is still fresh never touches the provider at all.
Rotation is honoured. A provider that issues a new refresh token on every use — which is what OAuth 2.1 tells them to do — retires the old one. Storing only the original works perfectly until it doesn't, so whatever comes back is kept. Three chained renewals are tested precisely because each depends on the previous one having been stored.
Concurrent commands do not break the session. Two invocations at once would both see a stale token and both refresh; against a rotating provider the second invalidates the first, and whichever writes last leaves a credential the other already broke. The symptom is a login that mysteriously dies when somebody runs two commands in parallel. Refresh happens under a file lock, and re-reads the credential inside it so a process that waited uses what the winner wrote. Tested with eight concurrent workers against a rotating provider: exactly one refresh, and the session still works.
Writes are atomic — temp file plus rename, the same as the artifact store — because a crash partway through would otherwise leave a truncated token where a valid one was, which fails in a way that looks like a server problem.
I added a longer-lived secret, so I added a way to withdraw it. logout calls
the provider's RFC 7009 revocation endpoint before deleting the local file:
$ modelforgectl logout
revoked this session at the identity provider
removed the stored login from this machine
Verified by putting the deleted credential back afterwards — the provider rejects it, so the session is genuinely dead rather than merely forgotten locally.
A provider with no revocation endpoint is reported plainly rather than glossed, because "logged out" quietly meaning two different things depending on the provider is how somebody ends up believing a credential is dead when it is not.
A browser can sign in and stay signed in:
GET /login → the identity provider → back with a session cookie
GET /logout → the session is destroyed server-side
The cookie carries an opaque random id and nothing else. Putting a signed identity in it instead would mean a session cannot be revoked before it expires, because the server has nothing to delete — and revocation is most of what makes sessions safe to hand out.
Every cookie attribute closes a specific hole, and there is a test asserting each one:
| Attribute | What it stops |
|---|---|
HttpOnly |
script reading the session — an XSS bug elsewhere cannot steal it |
Secure |
the session travelling over plaintext HTTP |
SameSite=Lax |
cross-site form posts carrying the cookie |
no Domain |
one compromised subdomain reading the session |
The CSRF cookie is the deliberate exception to HttpOnly: the double-submit
pattern depends on same-origin script being able to read it.
Reads need only the cookie; writes need a CSRF token. SameSite=Lax already
stops a cross-site page sending the cookie on a POST, and the token is the second
lock on the same door — worth having because SameSite is a browser behaviour,
and an old browser or a proxy that strips the attribute defeats it, while a token
the attacker cannot read is not defeated by any of that. The comparison is
constant-time, because unlike the token digests elsewhere here there is no hash
in front of it to scramble a near miss.
Verified against a running server:
POST /v1/models (cookie, no token) → 403 auth: CSRF check failed
POST /v1/models (cookie + token) → 201
A session is never a weaker front door. The ID token is verified through the same path an API request takes, so a cookie cannot carry an identity a bearer token could not — a user in no mapped group gets 403 and no session at all. Two verification routines would be two places for the rules to drift, and the weaker one becomes the way in.
A bearer token beats a cookie when both are present. A client that sent a credential meant it, and an ambient session silently overriding it is how a script ends up running as whoever last used the browser.
Session ids are minted only after authentication, which is what makes session fixation impossible: there is no way to hand somebody an id beforehand and have it become their session. The session also never outlives the token behind it, so access withdrawn at the provider is not extended by a cookie.
The redirect target is validated, not trusted. An open redirect on a login
endpoint is worth more than most: the victim authenticates for real at their real
provider and is then bounced somewhere the attacker controls, which is what a
convincing phish looks like. //evil.example is rejected explicitly — it is
protocol-relative, so a browser reads it as another origin while a naive
"starts with /" check reads it as a local path. That one character is the
whole bug.
The PKCE verifier, state and nonce are held server-side, keyed by a short-lived cookie. They could be put in a signed cookie, but there is no reason to hand a browser the secret that proves the token exchange belongs to this login.
Enable with -external-url, which must be configured rather than derived from
the request — Host is caller-controlled, and building the OAuth redirect URI
from it would let somebody point the provider's callback wherever they liked.
-insecure-cookies drops Secure for local HTTP and logs a warning saying
exactly what that costs.
/ lists the models; /models/{name} shows one in detail. Real output from a
running server with a 90/10 canary and a drifting feature:
Model Serving Versions Requests Failures Drift
fraud-score v1=90% v2=10% 2 1500 0 significant
churn-risk not deployed 0 0 0 —
Version Traffic Digest Features Requests Failures Mean batch
v1 90% 3a0468741e67 4 1079 0 26.3
v2 10% 3a0468741e67 4 121 0 3.1
Drift · v1 over 15m0s, 1079 samples
Feature PSI Severity Missing
amount 2.1556 significant 0.0%
tenure 0.0957 stable 0.0%
The split lands where the policy says, and the mean batch sizes differ for a reason worth noticing: the 10% canary fills batches more slowly than the incumbent, so the same batching window gives it 3.1 rows a call against 26.3.
Server-rendered HTML, no JavaScript at all. That is a decision, not a
shortcut. html/template does contextual auto-escaping — it knows whether a
value is landing in element text, an attribute or a URL — and that is the XSS
defence. It matters more here than it would have a week ago: this server hands
out session cookies now, so a script injected into one of these pages would run
with somebody's session.
Having no JavaScript is what makes the Content-Security-Policy meaningful:
default-src 'none'; style-src 'unsafe-inline'; form-action 'self';
base-uri 'none'; frame-ancestors 'none'
default-src 'none' with no script-src means no script can run at all, which
turns "everything is escaped correctly" from a claim into something the browser
enforces independently. A test asserts the pages contain no <script> and no
inline handlers, so a future addition fails loudly rather than being silently
blocked — which is a confusing way to find out.
It also means no bundler and no node in CI for a Go serving platform, and the
binary stays the single self-contained artifact everything else here assumes.
The templates are embedded with go:embed and parsed at startup, so a broken
one fails the build rather than the first request.
A browser gets a redirect where an API client gets a 401. That split is the one place the HTML surface should behave differently: a browser handed a bare 401 has nowhere to go and no way to know a sign-in exists, while an API client handed a redirect follows it and tries to parse a login page as JSON. The redirect preserves where you were heading, and the destination is re-validated on the way back through the same open-redirect check.
The dashboard is behind the read scope like any other read, so it is not a way
around authorisation — a predict-only credential gets 403.
Deploy, canary, shadow and rollback are available as forms. Every one is two steps — a plan that says exactly what will change, then an apply that carries it out:
Confirm this change to fraud-score
Send 10% of traffic to version 2, keeping 90% on version 1.
Now After
v1=100% v1=90% v2=10%
This changes which model answers live requests, immediately.
A single button that immediately moved traffic would be the wrong shape. These operations decide which model answers every request, they are being triggered by a mouse rather than a reviewed script, and the cost of a mis-click is wrong predictions rather than an error message. The confirmation is where somebody sees "90/10 becomes 100% v2" before it is true.
Two operators cannot silently overwrite each other. The policy as it stood when the confirmation was rendered is carried in the form, and apply refuses if it no longer matches:
This model changed while you were looking at it. It was "v1=100%" when the
confirmation was shown and is "v1=90% v2=10%" now, so nothing was applied.
Without that, the later click wins by arriving later and the person whose change was discarded has no way to know. The label doubles as the comparison token, and is order-independent so two policies that mean the same thing compare equal.
This is why the CSRF check accepts a form field. A page with no JavaScript
cannot set a header, so csrf_token in the body is the standard alternative —
equally safe, because what makes double-submit work is that a cross-site
attacker cannot read the cookie, and that holds wherever the value comes back.
Only form-encoded bodies are parsed for it: calling ParseForm on a JSON request
would consume the body the handler is about to decode, so a convenience for HTML
forms would have silently broken every API POST.
Verified against a running server: a form post with no token gets
403 auth: CSRF check failed: no X-CSRF-Token header or csrf_token field
Both surfaces share one code path. The dashboard calls the same
applyPolicy the JSON API does, so a version that cannot load is refused
identically and a shadow is carried across the same way. Two paths that both
change what serves production traffic would be two places for the rules to
drift, and the one with fewer checks becomes the way in.
The forms are behind the admin scope and are not rendered at all for a
read-only credential — showing buttons that always refuse is worse than not
showing them. Apply redirects afterwards, so a refresh does not re-apply. And
the audit line records the effect, not just the route:
{"msg":"policy changed from the dashboard","actor":"sahil@example.com",
"model":"fraud-score","summary":"Send 10% of traffic to version 2...",
"before":"v1=100%","after":"v1=90% v2=10%"}
The first version left Endpoint.AuthStyle unset. x/oauth2 then probes for
the server's preferred client-authentication style: it tries one, and if that
is rejected, retries with the other.
An authorization code is single-use. So the probe burns the code, and the
error you see is the second attempt's invalid_grant — which points at the
code, when the real problem was whatever made the first request fail. I spent a
while chasing a phantom double-exchange because of it.
Setting AuthStyleInParams explicitly is both correct — RFC 6749 says a public
client sends its id in the body, having no secret for a Basic header — and
removes the failure mode. Verified: one token request per login, where there
had been two.
I wrote the XGBoost scorer rather than binding to libxgboost, and did not write
the JWT verification. The difference is what a bug costs. A mistake in the
scorer produces a wrong number, which a differential test against XGBoost
catches. A mistake in JWT verification is a total authentication bypass —
alg:none, RSA-versus-HMAC confusion, an unchecked kid — and the failure mode
is that everything works perfectly while anybody can forge an admin identity.
That is not a place to demonstrate that I can write code.
So this uses go-oidc, and the tests verify it is wired up correctly rather
than that RSA works: a real key, a real JWKS endpoint over HTTP, real signed
tokens, and assertions that a token signed with the wrong key, a token with
swapped claims, an alg:none token, an expired token and a token for the wrong
audience are each refused. A stub returning "valid: true" would pass a
feature-shaped test suite while leaving the server forgeable.
Provider outages degrade logins, they do not stop the server. Discovery runs in the background with backoff. If the provider is unreachable at startup the process still boots, static credentials still work, the serving path still serves, and JWTs get a clean refusal until discovery succeeds.
/healthz and /readyz need no credential. A liveness probe that needs one
starts failing the moment a token is rotated or misconfigured, and would then
restart the very process that is serving correctly — and neither endpoint
reveals more than whether the process is up and how many models it holds.
/metrics is protected, by read. It is not neutral: it carries every
model name, version, request volume and drift reading, which together describe
what is being scored and how much of it. Prometheus sends a bearer token from
its scrape config, so the cost of protecting it is two lines of configuration.
Control-plane changes are logged with the name of the credential that made them:
{"msg":"authorised change","token":"dev","method":"PUT","path":"/v1/models/fraud-score/policy"}
Never the token itself — only its name, which is why the Token struct carries
no secret to leak in the first place. The condition is the admin scope rather
than simply a writing HTTP method, and that distinction was a bug I shipped and
caught in the smoke run: scoring is a POST, so keying on the method logged a
line per prediction. At serving volume that is millions of audit entries a day
burying the handful that record an actual change. Twenty predictions now produce
zero audit lines; one policy change produces one.
A client that keeps failing authentication gets a burst of attempts, then 429
with Retry-After until its budget refills.
What this is for, precisely. It is not brute-force protection, and claiming otherwise would be the security theatre I said I wanted to avoid: the tokens are 256 bits of randomness, so an attacker at a million guesses a second is not finding one, and a rate limit does not change that arithmetic at all.
What it bounds is the cost of failures. Every rejection runs the HTTP stack, a hash, and — the expensive part — writes a log line containing attacker-controlled request data. Unbounded, that is a free way to fill a disk, run up a log-ingest bill, and bury the entries an operator needs during an incident. Measured on a live server, 200 requests with a bad token:
195 × 429, 5 × 401 # burst of 5, then throttled
6 log lines total # 5 rejections + 1 throttle notice, not 200
A valid credential is never throttled. Authentication runs before the limit check, deliberately. Checking the limit first is marginally cheaper — it skips a SHA-256 over a short string — but an IP is a shared resource, and behind a NAT or a shared egress that would refuse a correct credential because a neighbour is failing. That is an outage for somebody who did nothing wrong, bought with a saving too small to measure. Verified live: a valid token from a fully throttled address still returns 200.
A 403 does not count against the budget. A valid token refused for lacking a scope is a misconfigured client, not somebody guessing, and throttling it would take a deploy script offline for holding the wrong role.
The tracking table is bounded, because a map keyed on client address is otherwise its own denial of service. It is capped, and sweeps buckets that have fully refilled — lossless, since a refilled bucket is indistinguishable from an absent one. If the table saturates it fails open: refusing to track a new client and denying it instead would let an attacker fill the table and lock everybody out, which is the outage the limiter exists to prevent.
modelforge_auth_throttled_total counts clients newly throttled. The address is
not a label — it is attacker-controlled and unbounded, so labelling by it would
let anybody mint arbitrary time series and take the monitoring down, which is
worse than the failed logins it describes. The address goes in the log line,
where a high-cardinality value belongs.
Tune with -auth-max-failures and -auth-failure-window; -auth-max-failures=-1
turns it off for deployments that do this upstream. Behind a proxy, set
-trust-forwarded-for — without it every request carries the proxy's address and
shares one bucket, and with it the header is trusted, so it is only safe when a
proxy overwrites it.
make up # adds Prometheus on :9090 and Grafana on :3000(Both ports are overridable — see the table in the quickstart.)
Grafana comes provisioned with a dashboard and Prometheus with four alert rules. Latency histogram buckets start at 250µs rather than Prometheus' default 5ms, because the batching window is measured in milliseconds and the default buckets would put every request in the first one — a histogram unable to show the thing it exists to show.
Latency is graphed per version, not per model: a canary that is slower than the incumbent is invisible in an aggregate that is still 90% incumbent traffic.
make db && make testThe suite requires a real Postgres and fails rather than skips when it is
missing. A suite that skips its own dependency keeps CI green while testing
nothing, and the gap is invisible precisely because everything looks like it
passed — CI greps its own output for --- SKIP and fails the job if it finds
any. The store is also the one component an in-memory double says nothing useful
about: uniqueness constraints, the row lock that serialises version assignment,
and what timestamptz does to a nanosecond timestamp all live in the database,
and a fake agrees with whatever the code happens to do.
go test runs with -p 1. Several suites drive the whole stack against the
same database and truncate it between cases, so running packages concurrently
has them resetting the database underneath each other.
Coverage is 89.9% of internal/, measured with -coverpkg so that code is
attributed to the package that owns it rather than the package running the test
— without it the end-to-end suites, which exercise serving and routing through
HTTP, would report those packages as untested.
The tests that matter most are the ones checking a thing that would otherwise fail silently:
| Test | What it stops |
|---|---|
TestAgainstXGBoost |
predictions drifting from the library they claim to match |
TestSplitComparisonUsesFloat32 |
the float32 threshold bug returning |
TestEveryCallerGetsItsOwnPrediction |
a batch scatter handing a caller someone else's answer |
TestVersionNumbersAreUniqueUnderConcurrency |
two versions sharing an identity |
TestSelectIsStableForTheSameKey |
a canary that measures noise |
TestGuardWillNotEmptyTheSplit |
an automatic rollback becoming an outage |
TestPoliciesSurviveARestart |
a restart quietly serving 404s |
TestOneCallerCancellingDoesNotCancelTheBatch |
one client's timeout failing its batch mates |
TestEveryRouteEnforcesItsScope |
a route shipped without a scope |
TestStartupFailsClosedWithoutTokens |
a server serving its control plane to anyone |
TestOnlyControlPlaneChangesAreAudited |
an audit log drowned in one line per prediction |
TestAValidCredentialIsNeverThrottled |
a shared IP turning a rate limit into an outage |
TestSaturatedTableFailsOpen |
an attacker filling the limiter's table to lock others out |
TestReloadIsAtomic |
a request mid-rotation seeing neither credential |
TestABadRotationDoesNotTakeTheServerDown |
a malformed token file locking everyone out |
TestForgedSignatureIsRejected |
a forged JWT authenticating as anybody |
TestUnsignedTokenIsRejected |
the alg:none bypass |
TestWrongAudienceIsRejected |
a token minted for another service working here |
TestAuthenticatedButNotAuthorised |
a valid login becoming access by default |
TestPKCEIsActuallySent |
a login flow shipping without PKCE |
TestCallbackRejectsAForgedState |
a forged callback completing somebody else's login |
TestStoredCredentialIsPrivate |
a world-readable credential in a home directory |
TestRotatedRefreshTokenIsStored |
a session that works until the old token is retired |
TestConcurrentRefreshDoesNotBreakTheSession |
two parallel commands invalidating each other's token |
TestLogoutRevokesAtTheProvider |
"logged out" leaving a working credential alive |
TestCrossSiteWriteIsRefused |
another origin acting with your session cookie |
TestLoginRejectsAnOpenRedirect |
a real sign-in bouncing to an attacker's page |
TestBearerTokenBeatsCookie |
an ambient session overriding an explicit credential |
TestDashboardEscapesHostileContent |
a script in a model description running with your session |
TestDashboardShipsNoJavaScript |
a future script tag silently blocked by the CSP |
TestConcurrentEditIsRefused |
one operator's deploy silently discarding another's |
TestDeployActionWithoutCSRFIsRefused |
a cross-site page moving production traffic |
I would rather write these down than let someone discover them.
- Only gbtree XGBoost models. dart applies per-tree weights the loader does not read, and categorical splits compare a category code against a threshold as if it were a number. Both are rejected at load time rather than approximated, because a scorer that half-understands a model produces plausible wrong numbers in production instead of an error at deploy time.
- Models are held in memory, all of them. There is no eviction, so the working set has to fit. Fine for tens of gradient-boosted models; not the design for a tenant-per-model service with thousands.
- Drift is monitored on the first output only for multi-class models. Comparing a whole probability vector against one baseline needs a joint distribution, which is a different statistic.
- The rollout guard reacts to errors, not to quality. It catches a version that is throwing, not one that is quietly predicting badly — that needs labels, which arrive long after the request.
- Browser sessions live in memory, so a restart signs everybody out. That is the deliberate trade: the alternative writes credential-equivalent material into a database that is backed up, replicated and read by people debugging models. Losing a session costs one click.
- Deploy actions have no approval step beyond the person clicking. The confirmation shows what will change, but one admin can move production traffic alone. Four-eyes review would need a request-and-approve workflow, which is a different feature.
- It polls rather than streams. A meta refresh every 15 seconds, because live updates would need JavaScript and that is the one thing this page deliberately does not have.
- A stolen credential file is worth more than it used to be. It now holds a refresh token, which outlives the ID token it renews. Rotation, 0600 permissions and revocation on logout are the mitigations; none of them make it free.
- Renewal needs the provider reachable. An expired session on a machine that cannot reach the identity provider cannot be renewed, where a static token would still work.
- Static-token revocation is as fast as the reload. A withdrawn token stops
working on the next
SIGHUPor within the poll interval, not instantly across a fleet. OIDC logins do not have this problem — they expire on their own. - Group membership is read from the token, not the directory. If somebody leaves a group, their access changes when their token is next reissued, not the moment the directory does.
- Rate limiting keys on the client address, which is a shared resource. Behind a NAT everyone shares a bucket, so a burst of 10 is per egress rather than per person. A valid credential is never refused, which is what keeps that from being an outage — but it does mean the limit is coarser than it looks.
- It is not a substitute for volumetric defence. It bounds the cost of failures reaching this process; it does nothing about the traffic arriving. A gateway is still the right place for that.
| Path | What lives there |
|---|---|
internal/runtime/xgboost |
the pure-Go scorer and the XGBoost JSON loader |
internal/registry |
models, versions, policies — Postgres |
internal/artifact |
content-addressed blob store |
internal/batch |
the dynamic batching scheduler |
internal/routing |
canary splits, shadow mirroring, the rollout guard |
internal/drift |
PSI baselines and the sliding-window monitor |
internal/serving |
loaded versions, schema binding, the data plane |
internal/auth |
bearer tokens, scopes, the audit middleware |
internal/api |
HTTP surface |
internal/app |
wiring, startup restore, shutdown ordering |
internal/cli |
modelforgectl |
tools/fixtures |
the XGBoost fixture generator |
MIT — see LICENSE.