Uh oh!
There was an error while loading. Please reload this page.
rfc(decision): Distroless base images - #157
Conversation
21cb505 to
75cc51eCompare| # Resolved questions | ||
| - **Long-term commitment to DHI:** Despite Docker Inc having a history of unexpected licensing and policy changes (Hub rate limiting, Desktop licensing, etc.), DHI was recently made public under Apache 2.0, and a rollback of that decision seems unlikely. If needed, Google Distroless is a practical drop-in fallback — it lags a few patch versions behind but is otherwise compatible. Other solutions may also emerge over time. We can go with DHI images as a default. |
There was a problem hiding this comment.
Do you think we would be able to maintain the images if we had to?
There was a problem hiding this comment.
While we really don't want to do this (as it's not Sentry business) but I tried doing this literally from scratch in a couple of hours, and drop-in replacement of the base image worked fine to the point of passing all smoke tests on snuba:
- https://github.com/oioki/python-base-image (own base images from scratch)
- build: try ghcr.io/oioki/python-base-image for distroless targets snuba#7959 (testing own base images)
So, if we had to, it is doable nowadays.
| - **Snuba and getsentry:** These are the largest remaining Python services. The Snuba PoC (https://github.com/getsentry/snuba/pull/7753, https://github.com/getsentry/snuba/pull/7821, https://github.com/getsentry/snuba/pull/7829, https://github.com/getsentry/ops/pull/19824) showed it is feasible. What is the sequencing and who owns driving this to completion? | ||
| - **Local development compatibility:** Are there any blockers that might disrupt local development workflows when switching to distroless? So far this appears to be a non-issue — for example, Snuba distroless containers work fine in `sentry devservices` (https://github.com/getsentry/snuba/pull/7829). | ||
| - **Services with non-trivial runtime deps:** Some services (e.g. uptime-checker with OpenSSL for certificate validation, or services using external libraries) may need extra work. Are there any blockers that make distroless infeasible for them? | ||
| - **Public mirrors for anonymous access:** Pulling directly from `dhi.io` requires a Docker login, which complicates CI pipelines and local image builds for contributors. Should we commit to maintaining public mirrors at `ghcr.io/getsentry/dhi` to allow unauthenticated pulls? See current PoC: https://github.com/getsentry/dhi. |
There was a problem hiding this comment.
Needing to login could be disruptive to self-hosted users.
There was a problem hiding this comment.
We are going to mirror the limited set of images (for start, python and node) on artifact registries controlled by us: either GHCR (could be more native for self-hosted) or Public Artifact Registry in GCP (could be faster for SaaS builds?). In both cases, pulling base images from those won't require authentication.
There was a problem hiding this comment.
Update: this is now implemented and the login requirement is gone for downstream consumers. We run a public pull-through mirror on GCP Artifact Registry at us-docker.pkg.dev/sentryio/dhi-mirror, serving python and node with no authentication. It transparently caches from dhi.io on demand — no hand-maintained mirroring pipeline — so self-hosted users and CI pull base images without logging in. Infra PR: getsentry/ops#21183. I've moved this to the RFC's Resolved questions section.
| Distroless containers have no shell. You cannot `exec` into a running container and run arbitrary commands. Debugging requires: | ||
| - Attaching an ephemeral debug container with a shell to the running pod (e.g. [`sentry-kube debug`](https://github.com/getsentry/sentry-infra-tools/blob/main/sentry_kube/cli/debug.py)) | ||
| - Using application-level tooling (e.g. interactive shells provided by the framework) rather than OS-level tools, e.g. `getsentry shell` | ||
| - Investing in proper observability (logs, metrics, tracing) instead of ad-hoc inspection |
There was a problem hiding this comment.
I'd like us to put in some effort ahead of time to validate that the debugging flow is very smooth - we've definitely run into various issues attempting to attach debugger pods in some of the places we've already swapped out more minimal images.
There was a problem hiding this comment.
sentry-kube debug is a full replacement of the kubectl exec workflow. I'd even consider it a better experience as it allows elevating permissions to allow attaching runtime profilers, which isn't possible with just kubectl exec.
There was a problem hiding this comment.
+1, our existing application images are already very slim, so exec is not very useful. debug is a better experience today, and anytime I use exec, i immediately go into getsentry shell.
There was a problem hiding this comment.
I guess the only downside for debugging is docker-compose in dev setups and self-hosted where sentry-kube debug doesn't work.
There was a problem hiding this comment.
sentry uses /dev/shm for multiprocess IPC, and /tmp for random stuff related to release artifacts and other file uploads.
some of these images explicitly say that /tmp is "hardened" (so basically unusuable). while we generally mount tmpfs on /tmp into the container (making this a non-issue), i'm not sure that we do it consistently in all pods that require it. also no idea if self-hosted has this kind of setup at all.
it may be desirable to enforce in the ops repo that tmpfs is mounted in absolutely every container in /dev/shm and /tmp, rather than relying on smoke tests. we have had incidents where new deployments were missing those mounts leading to really weird issues in multiprocess consumers, so some systematic enforcement would be nice to have for other reasons.
| Distroless containers have no shell. You cannot `exec` into a running container and run arbitrary commands. Debugging requires: | ||
| - Attaching an ephemeral debug container with a shell to the running pod (e.g. [`sentry-kube debug`](https://github.com/getsentry/sentry-infra-tools/blob/main/sentry_kube/cli/debug.py)) | ||
| - Using application-level tooling (e.g. interactive shells provided by the framework) rather than OS-level tools, e.g. `getsentry shell` | ||
| - Investing in proper observability (logs, metrics, tracing) instead of ad-hoc inspection |
There was a problem hiding this comment.
+1, our existing application images are already very slim, so exec is not very useful. debug is a better experience today, and anytime I use exec, i immediately go into getsentry shell.
Move the "public mirrors for anonymous access" item to Resolved questions: the login concern (disruptive to self-hosted users and CI) is addressed by the public GCP Artifact Registry pull-through mirror at us-docker.pkg.dev/sentryio/dhi-mirror, which serves python/node without authentication and needs no hand-maintained mirroring pipeline. Refs getsentry/ops#21183 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dav1dde
left a comment
There was a problem hiding this comment.
Self-Hosted needs curl and potentially other tools available for docker-compose health checks, since docker compose does not have any way of natively using HTTP for health checks.
How do we deal with this?
Possibly out of scope for this RFC.
oioki
commented
Jun 17, 2026
For snuba, we re-implemented health checks in python: getsentry/self-hosted#4352 |
@oioki that doesn't work for distroless containers which do not have a Python interpreter available, which is very similar to the curl/bash issue (just now with Python). |
oioki
commented
Jun 17, 2026
We can still add curl binary if we really need to, not a big deal. |
Rendered RFC