feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes) - #487

Closed
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable
Closed

feat(stack): paid services survive stack down/up (sell-inference + sell-http resume + storefront fixes)#487
bussyjd wants to merge 7 commits into
mainfrom
feat/sell-paid-services-stable

Conversation

@bussyjd

Copy link
Copy Markdown
Contributor

Summary

Supersedes #485 and #486. Single PR for the full "operator does obol stack up once and every paid service comes back, including its storefront entry, ERC-8004 well-known route, and host-side gateway" surface.

The work spans three originally-separate threads that turned out to be one coherent feature:

  1. obol sell inference registration parity--register-* flags, auto-register on-chain, signer ≠ payTo allowed per the ERC-8004 spec, and obol sell update --pay-to patch builder extracted so operators can re-point payees post-creation.
  2. Stack-up resumeobol stack up walks the on-disk seller state and replays every offer it finds, plus spawns the inference foreground gateway as a detached subprocess so UpstreamHealthy flips True without operator intervention.
  3. Storefront filter relax + sell-http persistence — the controller's Ready=True requirement gated on the ERC-8004 on-chain tx (which often takes minutes or never lands if the agent wallet is unfunded), silently hiding usable offers from the operator's own storefront. The relaxed filter treats Registered as informational and adds registrationPending=true to /api/services.json so UIs can badge the offer. Same store-and-resume pattern wired up for obol sell http too.

Commits (chronological story for review)

4708407fix(sell inference): emit registration spec + honor operator overrides — adds --register-*, fixes buildInferenceServiceOfferSpec to honor operator description / model name
b92b78efix(sell inference): auto-register on-chain to match sell http parityshouldAutoRegisterSell helper, source-level guard for the call site
dbf6c89fix(sell): allow signer != payTo on ERC-8004 register (not a spec constraint) — drops the over-strict obol-CLI guard, adds signerPayeeDelegationNote, extracts buildSellUpdatePatch for obol sell update --pay-to
2d36745feat(stack): resume sell-inference offers on stack up — descriptor → cluster artifacts replay
6efdf3ffeat(stack): zero-command resume — auto-start gateway + relax storefront filter — detached subprocess + offerOperationallyReady + RegistrationPending field
9947bb0fix(stack): three cosmetic fixes on top of the zero-command resume — PID-before-Release, footer wording, skill.md filter aligned with storefront
702d7c6feat(stack): resume obol sell http offers on stack up — sell-http manifest store + matching resume branch

Tests added across the PR

SurfaceTest
Registration flag parityTestSellInference_Flags extended to require the six --register-* flags + --no-register
Spec builderTestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault, _NoRegisterOmitsRegistration, _OperatorOverridesWin, _ModelNameNotHardcoded
Auto-registerTestShouldAutoRegisterSell (6 cases), TestSellInferenceAction_InvokesAutoRegister (source guard)
Signer ≠ payeeTestSignerPayeeDelegationNote (6 cases), TestAutoRegister_AllowsSignerPayeeMismatch (source guard)
obol sell updateTestBuildSellUpdatePatch_PayToOnly, _PriceSwitchNullsOldKeys, _NoFieldsErrors, TestSellUpdate_PayToFlagSurface
Store contractTestStoreCreate_PersistsResumeFields, TestStoreCreate_LegacyDescriptorWithoutResumeFields
Resume guardsTestResumeSellOffers_EmptyStoreNoOp, _DescriptorPresentButNoCluster, TestResumeOneInferenceOffer_RequiresModelName, _NilDescriptor, TestStackUpAction_CallsResumeSellOffers
Detached gatewayTestBuildResumeGatewayArgs (3 cases), TestReadGatewayPID (8 cases), TestProcessAlive_SelfAndBogus
Controller relaxTestOfferOperationallyReady_IncludesAwaitingExternalRegistration, _RejectsRealNotReady, TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers, _RegistrationPendingFalseForFullyReady
sell-http persistenceTestPersistSellHTTPOffer_RoundTrip, _NamespaceIsolation, TestRemoveSellHTTPOffer_DropsPersistedManifest, TestResumeSellHTTPOffers_EmptyStoreNoOp, TestSellDeleteAction_CallsRemoveSellHTTPOffer, TestResumeSellOffers_HTTPOnlyStore

Live spark2 validation

Walked through end-to-end multiple times during development:

  • obol sell inference aeon … --register-* … writes a descriptor with model_name, service_namespace, registration fields (PR feat(stack): resume sell-inference offers on stack up #486 commit 1).
  • obol stack down then obol stack up (with controller image rebuilt to pick up the storefront filter change): resume fires, gateway re-spawns detached with PID + log file, ServiceOffer + Service + Endpoints come back, /api/services.json returns 1 service with registrationPending=true, and the storefront landing page no longer says "No services are currently available."
  • obol sell delete aeon -n llm --force removes the on-disk inference descriptor and the sell-http manifest (if applicable). Re-obol stack up does NOT resurrect the offer.
  • obol sell inference aeon --pay-to 0xCold… against an agent signer of 0xA5d4aF96… no longer errors at the auto-register pre-flight; an informational note explains the delegation.

Follow-ups noted in the code

  • Inference gateway as a real cluster Pod — the detached host subprocess is the deliberate near-term shape. A helm-managed Pod (gateway image + Deployment) would survive stack down/up the same way Traefik / LiteLLM do, and the controller could observe it as a first-class workload instead of a docker-bridge bounce. Comment in startDetachedInferenceGateway calls this out.
  • Unified sell-offer store — sell-inference and sell-http persist to separate dirs because their descriptor shapes diverge. A future schema merge is fine; not the right MVP.
  • obol sell delete doesn't drop the inference descriptor — by design (the descriptor is what obol sell inference list/status reads). The HTTP cleanup is one-shot because there's no equivalent post-delete state to preserve.

Closes

Generated with Claude Code

bussyjd added 7 commits May 12, 2026 17:31
`obol sell inference` was producing ServiceOffers with three latent
defects that together stopped /.well-known/agent-registration.json
from ever being routed for inference-typed sells:
1. The inference subcommand never exposed `--register-*` flags
(compare with `obol sell http` which does), even though the help
text on `--register` says "registration is enabled by default".
2. `buildInferenceServiceOfferSpec` never wrote `spec.registration`
onto the offer at all, so the controller's
`reconcileRegistrationStatus` saw `Enabled=false` (zero value) and
emitted `Registered=True Disabled` with no RegistrationRequest CR
and no /.well-known HTTPRoute.
3. `buildInferenceServiceOfferSpec` hardcoded
`spec.model.name = "ollama"` regardless of the actual `--model`
value, so anything downstream that keyed off the model id (the
controller's description default included) was looking at the
wrong string.
Surfaced today on spark2 while trying to fetch
`https://inference.v1337.org/.well-known/agent-registration.json` —
got the Traefik fall-through 404. Manual `kubectl patch` enabled
registration, exposed a fourth defect:
4. `buildActiveRegistrationDocument` in the serviceoffer-controller
unconditionally overwrote `Spec.Registration.Description` for
inference offers with `"<model.name> inference via x402
micropayments"`, even when the operator had supplied an explicit
description at sell time.
This PR fixes all four:
- Add `--no-register`, `--register-name`, `--register-description`,
`--register-image`, `--register-skills`, `--register-domains`,
`--register-metadata` to `obol sell inference`.
- Rename `sellHTTPRegistrationInput` / `buildSellHTTPRegistrationConfig`
to the unqualified `sellRegistrationInput` /
`buildSellRegistrationConfig` since they now serve both inference
and http call sites.
- Extend `buildInferenceServiceOfferSpec` to accept the resolved
model name and the registration block, write
`spec.model.name = <real model id>`, and merge the registration
block into `spec.registration` when non-empty.
- In the controller's `buildActiveRegistrationDocument`, only fall
back to the model-aware description string when the operator left
`Spec.Registration.Description` empty.
Tests that would have caught the regression earlier:
- `TestSellInference_Flags` now requires the six registration flags;
their absence on `main` was the bug.
- `TestBuildInferenceServiceOfferSpec_RegistrationEnabledByDefault`
pins that defaults produce `spec.registration.enabled = true` and
the offer name as `spec.registration.name`.
- `TestBuildInferenceServiceOfferSpec_NoRegisterOmitsRegistration`
pins the `--no-register` opt-out.
- `TestBuildInferenceServiceOfferSpec_OperatorOverridesWin` pins
that operator-supplied name/description/image/skills/domains all
survive into the spec verbatim.
- `TestBuildInferenceServiceOfferSpec_ModelNameNotHardcoded` pins
that `spec.model.name` reflects `--model`, not the historical
"ollama" literal.
- `TestBuildActiveRegistrationDocument_KeepsOperatorDescription`
pins the controller-side fix: an operator description survives
the buildActiveRegistrationDocument pass.
- `TestBuildActiveRegistrationDocument_FallsBackToModelDescriptionForInference`
pins the other branch — inference offers with no operator
description still get the model-aware default, not the generic
one.
Drive-by: `TestSellInference_Flags` was failing on `main` after #470
removed the `--price` default but didn't update the corresponding
assertion. Updated to assert `--price` default = "" with a comment
explaining the contract.
The first revision of #485 stopped at "produce a ServiceOffer with a
populated spec.registration block". That made
/.well-known/agent-registration.json get routed, but left the offer in
Registered=False AwaitingExternalRegistration until someone manually
ran `obol sell register`. The serviceoffer-controller's storefront
filter (buildServiceCatalogJSON in render.go) requires Ready=True,
which transitively requires Registered=True, so the offer was silently
excluded from /api/services.json — the very feed the operator's own
storefront UI consumes.
obol sell http already auto-registers on the same code path. Mirror
that here so the inference path reaches the same end state without a
follow-up obol sell register step.
Changes:
- Extract shouldAutoRegisterSell(spec, tunnelURL) as the shared
decision predicate. The same gate now drives both the http and
inference auto-register call sites; defensively returns false when
the registration block is missing, disabled, malformed, or the
tunnel URL is empty.
- sellInferenceCommand Action: after kubectlApply + EnsureTunnelForSell
succeed, if shouldAutoRegisterSell says yes, call
autoRegisterServiceOffer with the resolved name/description/wallet
pulled from the spec. Surface failures as warnings + a re-run hint
rather than aborting the gateway start, because the underlying call
needs gas on the target chain and we do not want a one-off RPC
hiccup to block local dev.
Test coverage that would have caught the regression:
- TestShouldAutoRegisterSell - table-driven over six scenarios
including the defensive cases (registration not a map,
registration.enabled not a bool). Both call sites use the helper.
- TestSellInferenceAction_InvokesAutoRegister - source-level guard
that scans the sellInferenceCommand body for shouldAutoRegisterSell
and autoRegisterServiceOffer. The bug we just fixed was "Action
calls neither"; an innocent refactor could remove the calls without
any unit-level signal otherwise.
Operational note: the agent's remote-signer wallet needs a small ETH
balance on the target chain (~0.20-0.50 USD typical) for the on-chain
register tx. If the wallet is unfunded the warning fires and the
offer stays in AwaitingExternalRegistration; the operator can fund
the wallet and re-run obol sell register to finish.
…straint)
The autoRegisterServiceOffer pre-flight check rejected any registration
whose signer didn't match the offer's payTo wallet:
registration signer 0xA... does not match the payment wallet 0xB...
Use a matching signer, omit --wallet so the remote-signer wallet is
used, or pass --no-register
The error wording read like an ERC-8004 limitation but isn't. ERC-8004
treats the agent OWNER (msg.sender at register time) and the agent
WALLET (settable post-mint via setAgentWallet) as independent
addresses. x402 settlement honors the offer's spec.payment.payTo
directly — buyers pay that address regardless of what the registry's
getAgentWallet returns. The "hot signer, cold/multisig payee" split is
the canonical pattern.
The historic guard existed because the obol CLI never exposed
setAgentWallet, so a mismatched registration left operators with no
in-CLI recovery path. This change instead surfaces the split as an
informational note + adds `obol sell update <name> --pay-to <new>` as
the recovery surface (already in tree; just needed test coverage and
the connection wired into the diagnostic).
Changes:
- signerPayeeDelegationNote(signer, payTo) returns a human-readable
note when the two diverge (case-insensitive, whitespace-tolerant,
empty on either side) and "" otherwise. Used by
autoRegisterServiceOffer instead of the previous early-return.
- buildSellUpdatePatch(payTo, chain, price) extracted from the inline
sellUpdateCommand Action so the patch shape — the thing that
actually hits the cluster — is testable without a live offer.
Action calls the helper instead of inlining the same logic.
Tests:
- TestSignerPayeeDelegationNote — 6-case table: match,
case-insensitive, whitespace, empty payTo, empty signer (defensive),
true mismatch (assertions name the addresses + advise sell update).
- TestAutoRegister_AllowsSignerPayeeMismatch — source-level guard
asserting the banned error wording is gone from
autoRegisterServiceOffer and the soft-notice path is wired. Anyone
re-introducing the check has to delete this test too, which forces
them to read the rationale.
- TestBuildSellUpdatePatch_PayToOnly — `obol sell update <name>
--pay-to 0xBooB` builds a patch that touches only
spec.payment.payTo, not network or price.
- TestBuildSellUpdatePatch_PriceSwitchNullsOldKeys — table over
perRequest/perMTok/perHour: the unused keys are explicitly null so
a switch (e.g. perRequest → perMTok) doesn't leave the previous key
fighting through merge semantics.
- TestBuildSellUpdatePatch_NoFieldsErrors — error fires when no
fields are set, and the error names the flags the operator should
pass.
- TestSellUpdate_PayToFlagSurface — `obol sell update` exposes
--pay-to (with --wallet/--recipient/-w aliases via payToFlag), and
--namespace is Required.
Closes the "cluster comes back but the seller offers don't" gap.
After `obol stack down` destroys the k3d cluster, all ServiceOffer
custom resources are wiped from etcd. The descriptors on disk at
~/.workspace/config/inference/<name>/ survive, but nothing replays
them when the cluster comes back, so operators had to manually re-run
`obol sell inference <name>` for every offer after every stack up.
This wires a resume step into `obol stack up`: after stack.Up returns
successfully, the action iterates inference.Store, rebuilds the
Service+Endpoints+ServiceOffer for each persisted deployment from the
on-disk descriptor, and kubectl-applies them. The foreground host
gateway is NOT auto-started — it is an interactive operator action and
stack-up shouldn't launch long-running processes — but the cluster
side is fully reattached so the operator's eventual `obol sell
inference <name>` rerun hits a "service already healthy" reconcile
instead of "build from scratch."
Storage model:
- inference.Deployment gains ModelName, ServiceNamespace, and
Registration fields, persisted with `omitempty` so legacy descriptors
written by older binaries still load (the new fields come back as
zero values; the resume path either defaults sensibly or refuses
with an actionable message).
- The inference Action now resolves the registration block once and
passes the same map to both store.Create() and the in-process
ServiceOffer apply — guaranteeing on-disk state matches what the
cluster sees.
Resume path:
- resumeSellOffers(ctx, cfg, u): lists inference.Store, skips when no
kubeconfig (no cluster yet), warns + continues per-offer on errors.
- resumeOneInferenceOffer: createHostService + buildInferenceServiceOfferSpec
+ kubectlApply, no foreground process. Returns an actionable error
on missing ModelName so legacy descriptors surface a clear
"recreate the offer" prompt rather than producing a broken
ServiceOffer.
- Wired from cmd/obol/main.go's `stack up` Action after stack.Up. A
resume failure is logged as a warning, not propagated — stack-up
must succeed even if one descriptor is malformed.
Tests:
- TestStoreCreate_PersistsResumeFields: pins ModelName,
ServiceNamespace, and Registration round-trip through JSON. Without
this round-trip, the resume path silently loses operator
customizations.
- TestStoreCreate_LegacyDescriptorWithoutResumeFields: pins
backwards-compatibility — a JSON written by an older binary still
loads, with the new fields as zero values.
- TestResumeSellOffers_EmptyStoreNoOp: empty store, resume returns nil.
- TestResumeSellOffers_DescriptorPresentButNoCluster: descriptor on
disk, kubeconfig absent (post-purge, pre-up state) — resume must
skip cleanly.
- TestResumeOneInferenceOffer_RequiresModelName: legacy descriptor
with empty ModelName surfaces an actionable error naming the
missing field and the recovery command.
- TestResumeOneInferenceOffer_NilDescriptor: defensive nil/empty
descriptor guard so one bad entry can't crash the loop.
- TestStackUpAction_CallsResumeSellOffers: source-level guard that
cmd/obol/main.go's `stack up` Action calls resumeSellOffers AFTER
stack.Up. Pinning the order — running resume before stack.Up would
see no kubeconfig and skip every offer silently.
Operational note: `obol sell http` offers don't yet have an on-disk
descriptor (only `obol sell inference` persists), so they aren't
covered by this resume pass. Filed as a follow-up — adding a parallel
manifest store for `sell http` plus a third resume branch is its own
PR.
…ont filter
The first cut of stack-up resume (prior commit) reattached cluster-side
artifacts but stopped short of two things that together left the
operator's storefront empty after every stack-up cycle:
1. The foreground x402 gateway never restarted, so UpstreamHealthy=False.
2. Even after a manual gateway restart, the controller's storefront
filter required Ready=True — which itself requires Registered=True
— and an unfunded agent wallet (or a deliberate "register later"
choice) leaves Registered=False AwaitingExternalRegistration. The
operationally-usable offer was invisible to the operator's own UI.
Both addressed here so `obol stack up` is the only command needed.
1. Auto-start the gateway as a detached host subprocess:
- startDetachedInferenceGateway reconstructs the original
`obol sell inference <name> --model … --register-* …` invocation
from the persisted inference.Deployment and spawns it with
Setsid + Process.Release so it survives the parent's exit.
- PID + log path live under <StateDir>/sell-inference/<name>/.
readGatewayPID + processAlive let subsequent stack-ups detect a
still-running gateway and skip the relaunch.
- Surfaced via a u.Successf with the PID + log path so operators can
`tail -f` or `kill $(cat .pid)` without parsing helpers.
- sell_proc_unix.go isolates the platform-specific SysProcAttr
behind the unix build tag, keeping the rest of the code portable.
The long-term shape is to ship the inference gateway as a real
helm-managed cluster Pod (it would survive stack-down/up like every
other infra piece, and the controller could observe it as a
first-class workload). Comment in startDetachedInferenceGateway
spells that out so the next person looking at this knows the host
subprocess + PID-file plumbing is the deliberate near-term step,
not the final form.
2. Relax the storefront filter:
- offerOperationallyReady = ModelReady + UpstreamHealthy +
PaymentGateReady + RoutePublished. Registered is intentionally
NOT in the AND — on-chain ERC-8004 registration is publication
metadata, not operational readiness. Aggregate Ready=True is
also accepted as a shortcut so existing test fixtures and any
external callers that only emit the aggregate signal still work.
- offerAwaitingRegistration flags
Registered=False AwaitingExternalRegistration specifically, and
buildServiceCatalogJSON propagates it as
ServiceCatalogEntry.RegistrationPending=true so storefront UIs
can badge "registration pending" alongside the usable offer.
Tests:
- TestBuildResumeGatewayArgs (3 cases): full descriptor incl.
registration map, --no-register path, legacy descriptor with no
Registration map. Pins positional-name-before-flags ordering so a
CLI surface tweak can't silently desync the resume relaunch.
- TestReadGatewayPID (8 cases): clean integer, trailing newline,
surrounding whitespace, zero rejected, negative rejected,
non-numeric rejected, empty rejected, missing-file rejected.
Format is one decimal int in ASCII so external tooling
(`kill $(cat .pid)`) works without parsing.
- TestProcessAlive_SelfAndBogus: self is alive, absurd PID is not.
- TestOfferOperationallyReady_IncludesAwaitingExternalRegistration:
the headline behavioral fix — an offer with all four ops conditions
True + Registered=False(AwaitingExternalRegistration) is
operationally ready.
- TestOfferOperationallyReady_RejectsRealNotReady: an offer with
UpstreamHealthy=False is still excluded — the relax is narrow to
the registration gate only.
- TestBuildServiceCatalogJSON_IncludesPendingRegistrationOffers:
end-to-end through /api/services.json — AwaitingExternalRegistration
offers appear with RegistrationPending=true.
- TestBuildServiceCatalogJSON_RegistrationPendingFalseForFullyReady:
the negative — fully Ready=True offers must NOT carry
RegistrationPending or the storefront badge would stick around on
healthy offers.
Operational footnote: the auto-spawned gateway picks up the same
inference.Deployment that originally created the offer. If the
descriptor predates this PR and has no ModelName, the resume per-offer
warning fires (already pinned by TestResumeOneInferenceOffer_RequiresModelName
in the prior commit) — recreate the offer once with the new code so
the descriptor has the resume-side fields, after which every
subsequent stack down/up cycle reattaches the offer with zero extra
commands.
Surfaced during the live spark2 walkthrough of the prior commit:
1. Gateway PID printed as -1.
cmd.Process.Release() on Unix sets p.Pid to -1 as part of its handle
teardown, so reading cmd.Process.Pid AFTER Release prints a bogus -1
to the operator and pins -1 in the PID file. Snapshot the PID before
the Release call.
2. Stale "Host gateways are not auto-started" footer.
Carried over from the pre-auto-spawn version of the resume function.
With the gateway now auto-spawned, the message is misleading.
Replaced with a pointer to the gateway log path so operators can
`tail -f` immediately.
3. /skill.md catalog used the strict Ready=True filter while
/api/services.json used the new operationally-ready filter.
The two surfaces should stay consistent — an offer that's usable for
x402 payments shows up in BOTH the storefront feed and the operator-
facing skill catalog. Switched buildSkillCatalogMarkdown to the same
offerOperationallyReady predicate.
No new tests; the existing TestOfferOperationallyReady_* / catalog
tests cover the filter relax for both call sites since they share the
predicate. The PID-reading order is a one-line semantic fix; the
footer wording isn't worth a test.
Closes the second half of the "all paid services come back automatically
after stack up" promise. The inference resume in earlier commits only
handled `obol sell inference` because that's the only sell command
whose state was persisted on disk. `obol sell http` always built its
ServiceOffer manifest fresh from CLI flags and kubectl-applied it; no
on-disk trace, so the resume loop had nothing to replay for the http
case.
On-disk schema:
<ConfigDir>/sell-http/<namespace>__<name>.yaml
One YAML file per offer, holding the rendered ServiceOffer manifest
verbatim. The `<namespace>__<name>` filename ensures two offers with
the same name in different namespaces never collide. We keep this
separate from the inference store on purpose: inference descriptors
carry host-side fields (ListenAddr, ModelName, Registration, …) that
HTTP offers don't have, and folding them into one schema would force
empty/optional fields onto every HTTP descriptor. A future
unification is fine but not the right MVP shape.
Lifecycle:
- `obol sell http` (both --from-json and flag-driven paths) calls
persistSellHTTPOffer right after kubectlApply succeeds. Persistence
failures emit a warning but don't abort — the cluster-side offer is
already in place, the disk state is the recovery aid, not the source
of truth.
- `obol sell delete` calls removeSellHTTPOffer so a stack-up after a
delete doesn't resurrect the offer. Idempotent; missing-file
removals are silent no-ops.
- `obol stack up`'s resumeSellOffers calls resumeSellHTTPOffers after
the inference branch. The early-return for "no inference offers"
was rewritten to fall through to the http branch — operators with
only HTTP offers and no inference offers must still get their
storefront back.
resumeSellHTTPOffers reads each YAML, unmarshals, and kubectl-applies.
Per-offer errors are surfaced as warnings; the loop keeps going so
one corrupt YAML can't block the rest.
Tests:
- TestPersistSellHTTPOffer_RoundTrip — pins the on-disk shape (path
layout + YAML body content) by writing a manifest, reading it back,
and asserting metadata.{name,namespace} + spec.payment.payTo survive.
- TestPersistSellHTTPOffer_NamespaceIsolation — two offers with the
same name in different namespaces produce distinct files.
- TestRemoveSellHTTPOffer_DropsPersistedManifest — the symmetric
teardown contract: file goes away on delete, idempotent on double-
delete, defensive on empty inputs.
- TestResumeSellHTTPOffers_EmptyStoreNoOp — no offers ever persisted
must be a no-op, not an error.
- TestSellDeleteAction_CallsRemoveSellHTTPOffer — source-level guard
that the delete handler still calls the cleanup. Without it, an
innocent refactor could leak persisted manifests and the only
downstream symptom would be "deleted offers spookily come back" on
the next stack-up — hard to attribute.
- TestResumeSellOffers_HTTPOnlyStore — pins that the early-return
rewrite works: an http-only workspace still gets its offers
replayed.
The sell-inference resume tests from the prior commits still cover
their respective paths; the http-side additions are independent so
neither store interferes with the other.
@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Filed the longer-term direction as #488 — 'architecture: seller offers as first-class declarative state'. This PR is the deliberate near-term step that #488 explicitly supersedes; the migration path is captured there so the helm-chart-future intent doesn't only live in the startDetachedInferenceGateway comment.

@bussyjd

Copy link
Copy Markdown
ContributorAuthor

Superseded by #492 — folded into integration/post-490-cleanups as merge commit 7125662. Closing to keep the open-PR list tidy. Re-open if anything was missed in the fold-in.

@bussyjdbussyjd closed this May 14, 2026
bussyjd added a commit that referenced this pull request May 15, 2026
post-490: fold #487 + #489, strip debug logs, drop SKIP_PULL (smoke 13/13)
@OisinKyne
OisinKyne deleted the feat/sell-paid-services-stable branch July 1, 2026 12:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode)

1 participant

@bussyjd