Scope RSS hydration summaries to each source

[Floating-Y]

Summary

This is the first implementation cut for #417.

It carries decision record 0026 together with the query implementation, as
requested in review. The record status now says that the query cut is
implemented while the nested public contract remains pending.

The public API remains unchanged in this PR.

Problem

sources::list previously wrapped the complete ENTRY_SELECT projection in a
CTE referenced six times.

Because the CTE is referenced more than once, PostgreSQL materializes it. Since
ENTRY_SELECT has no source, knowledge-base or generation filter, this projects
every row in rss_full_content_entries, including rows belonging to other
knowledge bases and older generations.

Each correlated count then scans the unindexed materialized result for every
listed source.

The resulting shape was:

O(sources × all entries × 6)

Implementation

The query now:

• uses a LEFT JOIN LATERAL aggregate;
• scopes the projection by the outer source ID and current
  rss_generation;
• uses listed_source as the outer alias so it cannot be shadowed by the
  sources s join inside ENTRY_SELECT;
• calculates the existing baseline count and five hydration counts in one
  pass with count(*) FILTER;
• preserves the canonical state classification in ENTRY_SELECT;
• excludes non-RSS sources at the lateral join boundary;
• preserves the existing flat SourceView fields in this cut;
• preserves source ordering, document and missing counts, secret removal,
  SOURCE_SECRET_KEYS, and config - $2::text[].

The source/generation lookup is served by the implicit btree behind:

UNIQUE (source_id, activation_generation, external_key)

PostgreSQL names that index:

rss_full_content_entries_source_id_activation_generation_ex_key

The partial rss_full_content_entries_pending_idx does not serve this
aggregate.

baseline observations remain outside the five hydration work counts. The
existing flat baseline_count is still calculated in this first cut so the
public API remains unchanged.

Tests

The store tests cover:

• non-RSS sources returning null RSS summary fields;
• RSS source summary state and counts;
• pending observations;
• baseline activation;
• disabled full-content mode;
• current-generation filtering;
• exclusion of observations from older generations;
• isolation between sources and knowledge bases;
• merging queued and hydrating;
• merging terminal, deleted, and superseded;
• source ordering and unrelated list fields;
• source credential removal.

The database-backed suite was run with:

UTOPIA_TEST_REQUIRE_DB=1

against PostgreSQL running in Docker, so database tests could not silently
skip.

Validation completed:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo build --workspace
cargo test --workspace
UTOPIA_TEST_REQUIRE_DB=1 cargo test -p utopia-store
git diff --check

EXPLAIN ANALYZE

The comparison used PostgreSQL 16.15 with 22,000 RSS observations:

• 5,000 rows for the target source's current generation;
• 4,000 rows for an older generation of the target source;
• 6,000 rows for another source in the same knowledge base;
• 7,000 rows in another knowledge base;
• a non-RSS source in the listed knowledge base.

Both plans were captured with:

EXPLAIN (ANALYZE, BUFFERS, VERBOSE)

A warm-cache run with JIT disabled produced:

Measurement	Before	After
Deployment-wide entry rows projected	22,000	0
Current-generation rows scanned for the target source	included in global 22,000	5,000
Current-generation rows scanned across listed RSS sources	included in global 22,000	11,000
Index used	none on materialized CTE	rss_full_content_entries_source_id_activation_generation_ex_key
Shared buffer hits	384	337
Shared reads	0	0
Execution time	48.266 ms	43.921 ms

The lateral aggregate ran once per listed RSS source. For the non-RSS source,
the entry index scan was reported as never executed.

The important result is the change in scan scope: observations from older
generations and other knowledge bases are no longer projected for the target
source. Execution time is included for completeness but is secondary because
it is sensitive to cache state.

Scope

This PR includes:

• decision record 0026;
• the source-scoped query;
• database tests.

It does not include:

• the nested rss_full_content public contract;
• frontend changes;
• database migrations;
• new Rust or npm dependencies.

The nested API and Library update will follow in a separate PR after this query
cut lands.

Part of #417.

[WaylandYang]

Read this against the current query. The diagnosis holds, and it is worse than the record states.

Three things compound in WITH projected AS (ENTRY_SELECT) plus six correlated counts:

1. projected is referenced six times. Since PG 12 only a non-recursive, side-effect-free CTE referenced once is inlined; more than one reference materialises by default. So there is no version of this where the planner pushes e.source_id = s.id inside — the materialisation is guaranteed, not incidental.
2. ENTRY_SELECT carries no WHERE. What gets materialised is the whole rss_full_content_entries table, across every knowledge base and every generation.
3. The materialised result has no index. Each of the six correlated subqueries scans it in full, once per source.

And ENTRY_SELECT is not cheap per row: three joins, an EXISTS against document_deletions, and a nine-branch state CASE. That cost is paid across the full table, times the number of sources, times six. The shape is O(sources × all entries × 6) where it should be O(sources × own entries).

So: the direction is right, and this is a real defect on a hot path rather than speculative tuning.

One thing that needs an answer before this lands

Where do baseline rows go?

ENTRY_SELECT produces nine states. The proposed aggregate covers five groups — pending, queued (+hydrating), retrying, complete, terminal (+deleted +superseded). baseline is in none of them, and decision 2 removes baseline_count as internal state.

That may well be intended — a baseline entry is pre-activation stock, not outstanding work — but the record does not say so. As written, a source whose entries are all baseline reports five zeros while the table plainly holds rows, and nothing in the response explains the gap. Worth one sentence either way.

Two implementation notes

count(*) FILTER (WHERE state = …) gets all five numbers from a single pass over the source's own entries, instead of five subqueries over the same rows.

The s.kind = 'rss' predicate inside the lateral is harmless but the stated reason does not hold. A lateral subquery referencing an outer column is evaluated per outer row; there is no one-time short circuit. What actually keeps a non-RSS source out of the lateral is the predicate on the join. Keeping both is fine as belt-and-braces — the EXPLAIN this record already requires will show which one does the work.

On scope

Decision 1 (the aggregation) and decision 2 (the nested public contract) are independent changes travelling in one record. The contract change is reasonable on its own — rss_full_content: null says more than six null columns, and generation / baseline_count are implementation state that does not belong in a public response. v0.1 is also the cheapest moment this project will ever have for a breaking change, so the timing argues for doing it rather than deferring.

The coupling is what I would separate. Nothing in the aggregation needs the contract to change, and nothing in the contract needs the aggregation. Landed together, a problem found in either one takes the other back with it. Suggest two implementation PRs against this one record: the lateral aggregation first (internal, independently measurable, independently revertable), then the contract with its frontend change.

Otherwise

This is a more rigorous record than most of the internal ones — four rejected alternatives with reasons, an explicit evidence requirement on the implementation PR, and what was left out sitting in Open questions rather than going unmentioned. Thanks for writing it up this way.

[Floating-Y]

Thanks — updated the record to address all three points.

• The diagnosis now states that the six CTE references force
  materialization, that ENTRY_SELECT has no source or generation filter,
  and that each correlated count scans the unindexed materialized result.
• baseline rows are explicitly excluded by design. They represent
  pre-activation feed stock rather than outstanding hydration work, so the
  five counts describe hydration work rather than observation-ledger
  cardinality. baseline_count remains internal where activation logic
  needs it.
• The record no longer claims that the inner kind predicate causes a
  one-time short circuit. The join predicate is the exclusion boundary,
  while the inner predicate remains a defensive guard.
• The aggregation uses count(*) FILTER expressions in one source-scoped
  pass.
• Implementation is split into two independently revertible PRs: the query
  optimization first while preserving the flat API, followed by the nested
  public contract and frontend update.

The branch has been updated in b9f2daf.

[WaylandYang]

The revision answers all four points, and answers them better than I asked. The baseline paragraph states the consequence plainly — a source holding only baseline stock reports five zeros — instead of leaving the reader to infer it. The s.kind = 'rss' reasoning is now something an EXPLAIN can actually settle. Decision 4 is the right split. Nothing here needs rewriting.

What I want to change is when it lands, not what it says.

On merging this as a standalone record

I'd rather this file travelled with its first implementation PR than merged on its own. Three reasons, all from this directory's own conventions:

Every record 0001–0025 sits at a built status — In progress, Built, Implemented, Cut 1 implemented. The index is scrupulous about what is not built, which is what makes it worth reading. Proposed for #417 would be the first entry with no code behind it, and the convention that "the PR that implements a record updates its status line" assumes the record arrives with, or just ahead of, the work.

Decision 1 does not meet the test for writing a record. The test is: someone looks at the code in six months, asks "why not simply…", and the answer is not in the code. Nobody looks at a lateral aggregate and asks why the whole table wasn't materialised six times. That is a defect fix, and #417 already specifies it down to the shape of the response object. What does meet the test is decision 2 (why the public contract broke) and the rejected alternatives (why not simply a cache table). That residue is worth keeping — it just doesn't need its own merge.

Decisions 2 and 4 describe the second cut. If the whole record lands with the first PR, the status line can't be written honestly. Either the record travels with cut one and its status line says which decisions are implemented and which are still ahead, or it updates twice, once per cut.

To be clear about what this is not: it isn't a request to hold the work or to re-litigate the design. The direction is approved. Write the query PR against this file, carry the file on that branch, and fill in the status line when it lands.

Two notes for the query implementation

Both are things I'd want in the record before it lands, since they bear on decision 1 as written.

The outer alias has to change. sources::list selects FROM sources s, and ENTRY_SELECT itself contains JOIN sources s ON s.id = e.source_id. Embed ENTRY_SELECT in a lateral that references the outer s and the inner one shadows it: e.source_id = s.id binds to the entry's own joined source row and is trivially true, so the aggregate counts every source's entries. This does not raise an error, and a test fixture with a single source still passes. The outer table needs a different alias (src, say). Note that e.activation_generation = s.rss_generation survives the shadowing — the inner s is the entry's source — which is exactly what makes the failure quiet.

Name the index the plan should use. The only named index on rss_full_content_entries is rss_full_content_entries_pending_idx, and it is partial: WHERE entry_kind='candidate' AND current_job_id IS NULL. It will not serve this aggregate. What does serve it is the implicit btree behind UNIQUE (source_id, activation_generation, external_key), whose leading prefix matches the lateral's two equality predicates. Worth saying so in the record, so "the existing source/generation index" doesn't send the next reader to the partial one — and worth having the required EXPLAIN confirm which index the plan actually picks.

[Floating-Y]

Thanks — I have updated this PR to carry the first implementation cut together with the decision record.

The revision now:

• changes the record status to state that the query cut is implemented while the nested public contract remains pending;
• includes the source- and current-generation-scoped lateral aggregate;
• preserves the existing flat public API for this cut;
• uses listed_source as the outer alias, avoiding the s alias shadowing inside ENTRY_SELECT;
• names and verifies the implicit btree rss_full_content_entries_source_id_activation_generation_ex_key;
• documents why the partial rss_full_content_entries_pending_idx cannot serve this aggregate;
• includes database coverage for multiple sources, knowledge bases, generations and the existing state groupings;
• includes the before/after EXPLAIN (ANALYZE, BUFFERS, VERBOSE) evidence.

The branch is now based on the latest dev, and backend, web, migrations and DCO checks are passing.

I will hold the nested API and frontend cut until this first cut has landed.

[WaylandYang]

Merged, with three additions pushed to your branch first so it could land today rather than wait another round:

• The record is now 0033. dev took 0026 through 0032 while this was open; the file, its title line and the README row say 0033. Nothing else in the record changed.
• dev merged in. Only the README row conflicted.
• One more test, source_list_counts_only_the_listed_source. Two RSS sources in one base and a third in another, holding one, two and three pending observations; the lists report 1 and 2, then 3. This is the fixture that catches the alias-shadowing failure discussed above, which a single-source fixture cannot. The whole rss_full_content suite ran green against a database at dev's 41 migrations with UTOPIA_TEST_REQUIRE_DB=1.

The query is exactly as you last pushed it. #417 stays open for the nested contract and the Library change; same shape as this one, and welcome.