Skip to content

Update Prometheus fork to start-timestamps branch - #7

Open
dasbus wants to merge 1 commit into
invoca-0.42.0/replace-rate-funcsfrom
invoca-0.42.0/start-timestamps
Open

Update Prometheus fork to start-timestamps branch#7
dasbus wants to merge 1 commit into
invoca-0.42.0/replace-rate-funcsfrom
invoca-0.42.0/start-timestamps

Conversation

@dasbus

Copy link
Copy Markdown

Summary

  • Updates go.mod replace directive to point at Invoca/prometheus@307a9562c (invoca-3.13/start-timestamps branch)
  • Picks up Colin's xrate removal + yrate start-timestamp support + temporary debug logging for integration testing

Context

Companion to Invoca/prometheus#34. The Prometheus fork now includes start-timestamp-aware yrate/yincrease logic. This updates Thanos's embedded Prometheus dependency so the Thanos sidecar, Query, Store, and Compactor all use the new PromQL engine.

Once both images are built, they'll be deployed to red.prodref.us-east-1 via kubernetes-observability#17 for integration testing with st-synthesis feature flags enabled.

Test plan

  • go build ./... passes (verified locally)
  • Thanos image builds via Dockerfile.invoca
  • yrate/yincrease assertions pass when queried through Thanos Query on red.prodref
NONE

🤖 Generated with Claude Code

Point go.mod replace directive at Invoca/prometheus@307a9562c
(invoca-3.13/start-timestamps) which includes:
- xrate/xincrease/xdelta removal
- yrate/yincrease start-timestamp support
- temporary debug logging for integration testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Shannon Price <sprice@invoca.com>
@lhaven-invoca

Copy link
Copy Markdown

This PR will not unblock the red.prodref integration testing it is aimed at, because the blocker is in Thanos's own code rather than in the embedded Prometheus dependency.

The compactor is already crashlooping on red.prodref

thanos-compactor-0: 730 restarts, not Ready. Compaction and cleanup succeed; the downsample pass dies every iteration:

level=error err="4 errors: downsampling to 5 min: downsample block
01M01YD9T8M7HQ9W8EWGQSVX5E to window 300000: expand chunk 778, series 15091:
unexpected chunk encoding 4 XOR2; ..."

Four blocks: 01M01YD9T8M7HQ9W8EWGQSVX5E, 01M0738D1KEWJC8Y3FWVCSWFYZ, 01M01YCKG4S9P7Z0FY847KHS8P, 01M0738CTNNX0RH9CW51K6PTYK. It retries the same blocks on every restart, so it does not self-heal.

Bumping the Prometheus replace directive does not fix it

I checked both commits in this diff. Both already have full XOR2 support in the embedded TSDB:

Invoca/prometheus commitDateVERSIONXOR2 refs in tsdb/chunkenc/chunk.go
ec2ecb44fa29 (current)2026-07-093.13.022
307a9562c26d (this PR)2026-08-133.13.122

So the dependency can read XOR2 either way. The failure is in pkg/compact/downsample/downsample.go:

funcexpandChunkIterator(it chunkenc.Iterator, encoding chunkenc.Encoding, samples*[]sample) error {
switchencoding {
casechunkenc.EncXOR:
returnexpandXorChunkIterator(it, samples)
casechunkenc.EncFloatHistogram:
returnexpandFloatHistogramChunkIterator(it, samples)
casechunkenc.EncHistogram:
returnexpandHistogramChunkIterator(it, samples)
default:
returnerrors.Errorf("unexpected chunk encoding %d %s", encoding, encoding)
}
}

EncXOR2 is encoding 4 and has no case, so it hits default. This is Thanos code enumerating encodings explicitly, which is why a code search for xor2 in either Invoca/thanos or thanos-io/thanos returns nothing. Upstream latest is v0.42.4 and does not handle it either, so rebasing onto a newer upstream Thanos is not available as a route today.

Suggested scope

Adding a case looks small, assuming expandXorChunkIterator's it.Next() loop works unchanged against an XOR2 chunk:

casechunkenc.EncXOR, chunkenc.EncXOR2:
returnexpandXorChunkIterator(it, samples)

Worth checking whether anything else in the downsample path enumerates encodings the same way, and whether the aggregate chunk writer needs to emit XOR2 or can keep writing XOR.

Separately

The four blocks already written stay poisoned after any code fix, so the compactor needs them deleted or marked for deletion in the bucket before it recovers.

Also worth knowing: the enableFeatures list including xor2-encoding is live on red.prodref now, applied out-of-band. managedFields shows manager='kubectl-patch' owning spec.enableFeatures at 2026-08-13T14:31:49Z, and origin/main of kubernetes-observability has no enableFeatures block. The prometheus Argo CD Application has no selfHeal, so the drift persists and reports Synced. Context on kubernetes-observability#17.

🤖 Generated with LLM

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@dasbus@lhaven-invoca