Skip to content

Add instance expiration controls - #434

Merged
sjmiller609 merged 2 commits into
mainfrom
hypeship/instance-ttl-reaper
Aug 21, 2026
Merged

Add instance expiration controls#434
sjmiller609 merged 2 commits into
mainfrom
hypeship/instance-ttl-reaper

Conversation

@sjmiller609

@sjmiller609sjmiller609 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • accept either a relative ttl or absolute expires_at when creating or updating an instance
  • treat update TTLs as relative to commit time and allow ttl: "0s" to disable expiration
  • persist and return the canonical expires_at deadline, returning null when disabled
  • serialize creation and expiration updates with reaper deletion so successful operations cannot race stale deletion decisions
  • bound each reaper deletion to 30 seconds and skip busy lifecycle locks until the next sweep
  • sweep once at startup, then once per minute
  • return 409 instance_expired once the current deadline has passed and 404 if deletion already completed
  • clear expiration metadata from snapshots and forked instances

Instances without an expiration are unchanged. The reaper uses the normal instance deletion path.

Testing

  • targeted API tests for relative TTL, absolute expiration, disabling, validation, consistent error mapping, and explicit null responses
  • targeted manager tests for persistence, deadline boundaries, startup sweeping, delete timeout continuation, and race ordering
  • snapshot and instance-fork expiration clearing tests
  • targeted race-detector tests for reaper lifecycle coordination
  • go test ./... -run '^$'
  • make oapi-generate (verified deterministic output)

The complete API suite was also attempted, but environment-dependent tests require mkfs.erofs and network bridge permissions unavailable locally. The targeted tests and repository-wide compile pass.


Note

Medium Risk
Introduces automatic instance deletion and new create/update locking around expiration, which can destroy running workloads if deadlines are set incorrectly. Race handling between updates and the reaper is security-adjacent for data loss but is opt-in and covered by tests.

Overview
Adds optional instance expiration on create and update: callers may set a relative ttl (Go duration) or an absolute expires_at, but not both. ttl: "0s" disables expiration. Responses always include canonical expires_at (null when disabled).

A background TTL reaper starts at process init, sweeps immediately, then every minute. It deletes expired instances through the normal delete path, skips busy lifecycle locks, rechecks the deadline under lock, and times out stuck deletes at 30s so the sweep can continue.

Update TTLs are resolved at commit time. Once the current deadline has passed, expiration updates return 409 instance_expired. Forks and snapshots drop expiration; restoring a snapshot keeps the live instance’s existing deadline.

Reviewed by Cursor Bugbot for commit 127bc50. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actionsBot commented Aug 20, 2026

Copy link
Copy Markdown
-->

✱ stlc build

gocode · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

116 files generated at 5091857 (pushed)

go get github.com/kernel/hypeman-go-staging@50918576be65b0d9aebca23e958638a2c43ac56e
pythoncode · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

230 files generated at 641b19a (pushed)

typescriptcode · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

138 files generated at c0d6363 (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_76BZf799-gentle-wolf
Timestamp2026-08-21T18:19:08.418Z
stlc8413509
Spec hash0f15511e31a6
Config hash55e15f6f4434

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-21 18:19:36 UTC

@sjmiller609
sjmiller609 marked this pull request as ready for review August 20, 2026 14:11

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 843a3cd. Configure here.

Comment threadlib/instances/ttl_reaper.go
@sjmiller609
sjmiller609force-pushed the hypeship/instance-ttl-reaper branch from 843a3cd to 469a295CompareAugust 20, 2026 14:51
@sjmiller609sjmiller609 changed the title Add instance TTL expirationAdd instance expiration controlsAug 20, 2026
@sjmiller609
sjmiller609force-pushed the hypeship/instance-ttl-reaper branch from 469a295 to 0336f8bCompareAugust 20, 2026 15:10
@chruffins

Copy link
Copy Markdown
Contributor

reviewed — one correctness issue and three follow-ups worth addressing.

Bugs

  • lib/instances/update.go:56-65,111 — TTL deadlines are calculated before environment/proxy updates and metadata persistence complete. A slow update can commit an already-expired deadline, violating the API contract that TTL starts when the update is committed. Calculate the relative deadline at the final metadata commit point.

  • lib/instances/ttl_reaper.go:63-66 — the 30-second delete timeout is not a hard bound. The normal delete path calls context-insensitive operations such as killHypervisor and deleteInstanceData/os.RemoveAll, so a stuck delete can block this sequential sweep and delay all later expirations.

Structural / Maintainability

  • cmd/api/api/instances.go:137-152, lib/instances/create.go:624-628 — expiration validation is split between the API and domain layers and already differs for ttl: "0s" plus expires_at: HTTP rejects any combination, while the domain request accepts zero TTL and uses the absolute deadline. Make TTL presence explicit in the domain request or centralize the policy so callers cannot observe different rules.

Questions

  • lib/instances/fork.go:281, lib/instances/snapshot.go:302,426 — is the differing expiration behavior intentional? Instance forks and snapshot forks clear expiration, while restoring a snapshot preserves the current instance deadline. The public API does not document these inheritance rules.

@sjmiller609

Copy link
Copy Markdown
CollaboratorAuthor
  • TTL deadlines now resolve at the final metadata save, with expiration revalidated immediately before commit.
  • Reaper deletions run asynchronously, so the 30-second timeout bounds each sweep while preserving per-instance locking.
  • Create TTL is now presence-aware, making 0s plus expires_at consistently invalid.
  • Confirmed fork behavior is intentional and documented that instance TTL is cleared on fork.

@sjmiller609
sjmiller609 merged commit 6213d9f into mainAug 21, 2026
12 checks passed
@sjmiller609
sjmiller609 deleted the hypeship/instance-ttl-reaper branch August 21, 2026 18:37
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@sjmiller609@chruffins