Skip to content

fix: stop op autolaunching a session bus on every call - #1

Merged
GraysonCAdams merged 1 commit into
masterfrom
fix/session-bus-leak
Sep 14, 2026
Merged

fix: stop op autolaunching a session bus on every call#1
GraysonCAdams merged 1 commit into
masterfrom
fix/session-bus-leak

Conversation

@GraysonCAdams

Copy link
Copy Markdown
Contributor

The sandbroker user owned 103 processes tonight. Ninety-nine were dbus-daemon --session, one per op invocation, about 312 MB. A list_fields call hung past 123 seconds and had to be backgrounded, which is how it surfaced.

op links the 99designs/keyring secret-service backend, and its go-dbus client runs dbus-launch --autolaunch when DBUS_SESSION_BUS_ADDRESS is unset. The daemon that starts calls setsid() and outlives the op process that caused it. The service user has no login session and therefore no bus, so every single call started one. Caught directly: a /proc poller recorded op item list at 20:43:52.56 and a new sandbroker-owned session bus at 20:43:52.57, and twenty brokered calls against an idle vault took its cgroup from one bus to twenty-one, exactly.

The fix sets an inert DBUS_SESSION_BUS_ADDRESS on every child environment. go-dbus autolaunches only when that variable is empty or the literal autolaunch:; given a real address it attempts a connect, fails with ENOENT, and reports no session bus, which is the truth here. ProtectSystem=strict keeps the filesystem read-only so nothing can later appear at that path. Nothing is lost by it: authentication is a service-account token read from a file, never a stored login session, so the keyring falling back to its file backend costs nothing. Twenty Dev calls also got faster, 7.1s to 5.9s, dbus-launch having cost roughly 60ms each.

It is applied to all four child-environment builders rather than only the op one, since each could invoke something that wants a bus, and DBUS_SESSION_BUS_ADDRESS joins RESERVED_ENV so a caller cannot rebind it and undo the fix.

One correction to the initial diagnosis, which claimed the daemons escaped the unit's cgroup. They did not. Every orphan was inside the slice and dbus-daemon catches SIGTERM; a restart drained 44 processes to 2. That survival could not be reproduced, and the likely explanation is that Production was taking a call every twenty seconds, so the count regrew within seconds. Growth was linear per call and bounded by TasksMax=256, not unbounded, and containment was never the missing piece.

Not claimed: this does not explain the 123-second hang. At 99 daemons the unit sat around 121 tasks against a 256 limit, and 60ms per call does not add up to 123 seconds. It may be a separate problem, now easier to see.

Test plan

  • tests/test_childenv.py, six tests in tests/run.sh; the load-bearing one runs twenty commands through the real runner.run() against a stub applying go-dbus's own rule and asserts the process count is flat
  • A sibling test strips the guard and asserts the stub does leak three of three, so the counting test cannot pass vacuously
  • Perturbed: replacing without_session_bus with return env gives AssertionError: 20 != 0 : 20 call(s) out of 20 left a detached process behind; restored from a file copy, md5 identical across all five sources
  • Live: zero orphaned session buses after 40+ op invocations across both vaults, steady state four processes and about 95 MB
  • Both vaults verified end to end, no value reaching the caller; vault locks untouched
  • Full suite green with bash tests/run.sh
  • All five changed files byte-identical to what is deployed at /opt/sandbroker/lib/sandbroker/

op links the 99designs/keyring secret-service backend, and go-dbus
autolaunches a private `dbus-daemon --fork --session` whenever
DBUS_SESSION_BUS_ADDRESS is unset. The daemon user has no login session and
so has no bus, so every single op invocation started one. Each daemon calls
setsid() and outlives the op process that caused it: twenty list_items calls
left exactly twenty behind, and an evening of ordinary use had 99 sitting at
about 2.4 MB each. They do stay inside the unit's cgroup, so a restart
collects them, but between restarts the unit walks toward TasksMax for
nothing.

Pointing the variable at an address that cannot connect is the whole fix.
go-dbus autolaunches only when it is empty or the literal "autolaunch:";
given anything else it tries that address, fails, and reports no session bus,
which is the truth. The keyring falls back to its file backend and nothing is
lost, because authentication here is a service-account token read from a file
rather than a stored login session.

The guard goes on all four env builders, not just the op one. keeper, the
notify hook and a caller's own command each build their own child environment
and each could invoke something that wants a bus.
@GraysonCAdams
GraysonCAdams merged commit 928958a into master Sep 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant