✨ catalogd graphql shift to file-based cache

[grokspawn]

Description

This PR proposes a shift in the graphql service endpoint from in-memory to on-disk caching which leverages common architectural conventions like fan-out catalog index generation (graphql-schema.json) which is then accessed to fulfill queries to eliminate RSS increases from the feature initial implementation.

Included is a structured graphql validation package as a focus for future limits enforcement and adds several cold-cache/cache-miss concurrency tests.

Referenced against a diverse set of existing catalogs, this change reduces RSS 39X on average:

┌───────────────┬────────┬─────────┬──────────┬──────────┬───────────┐
│ Catalog │ Raw │ Objects │ OLD │ NEW │ Reduction │
│ │ Size │ │ Memory │ Memory │ │
├───────────────┼────────┼─────────┼──────────┼──────────┼───────────┤
│ r-o-i/4.23 │ 31MB │ 1,244 │ ~126MB │ ~2MB │ 63x │
├───────────────┼────────┼─────────┼──────────┼──────────┼───────────┤
│ ce-o-i/4.22 │ 8.6MB │ 954 │ ~35MB │ ~2MB │ 17x │
├───────────────┼────────┼─────────┼──────────┼──────────┼───────────┤
│ co-o-i/4.22 │ 20MB │ 4,958 │ ~79MB │ ~2MB │ 39x │
└───────────────┴────────┴─────────┴──────────┴──────────┴───────────┘

The new file-system storage is approximately 200KB/catalog for the new file offsets index data.

Query latency shifts from microsecond access to low millisecond for the same access approach, with a transient per-query allocation between ~1-10MB depending on the size of the object(s) returned in the query.

NB: I ran some benchmarks after investigating repeated failures of the st2ex e2e test, and determined that unbounded FBC evaluation during schema detection phase was causing excessive delays in catalogd service readiness. Instituted a bound of ten (10) samples of a given Meta to determine the appropriate GraphQL schema type, which allows the functionality to complete operation on example catalogs above with a worst-case of 250ms on test hardware. These changes are in a separate commit to help them stand out.

Reviewer Checklist

• API Go Documentation
• Tests: Unit Tests (and E2E Tests, if appropriate)
• Comprehensive Commit Messages
• Links to related GitHub Issue(s)

[netlify]

✅ Deploy Preview for olmv1 ready!

Name	Link
🔨 Latest commit	a5db5c5
🔍 Latest deploy log	https://app.netlify.com/projects/olmv1/deploys/6a511eb7bbe1a90008be48a5
😎 Deploy Preview	https://deploy-preview-2732--olmv1.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR shifts catalogd’s GraphQL implementation from building/caching schemas off in-memory catalog metas to a file-based approach: schema metadata is discovered during Store() and persisted to disk, while query execution loads requested objects from catalog.jsonl on-demand using byte offsets from index.json. It also introduces a query complexity validation helper and updates handlers/tests to use the new GraphQL service interface.

Changes:

• Persist GraphQL schema metadata to graphql-schema.json during catalog storage and load it from disk for schema building.
• Switch query execution to disk-backed object loading using index-based byte offsets (reducing RSS growth from caching parsed objects).
• Add a GraphQL query complexity validation helper and expand concurrency/singleflight tests around cache misses/builds.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
internal/catalogd/storage/localdir.go	Writes graphql-schema.json on store, adds disk-backed object loader, and adjusts GraphQL pre-warm behavior.
internal/catalogd/storage/index.go	Exposes schema-section byte ranges for disk-backed GraphQL pagination.
internal/catalogd/service/graphql_service.go	Refactors service to use a storage-provided data provider and adds query validation/timeout.
internal/catalogd/service/graphql_service_test.go	Updates tests to use the new provider-based GraphQL service and adds singleflight coverage via provider counting.
internal/catalogd/server/handlers.go	Updates GraphQL handler to execute queries without needing a catalog FS.
internal/catalogd/server/handlers_test.go	Updates mocks/tests for the new GraphQL service interface and error behavior.
internal/catalogd/graphql/validation.go	Adds AST-based query complexity validation (depth/aliases/fields).
internal/catalogd/graphql/graphql.go	Adds schema serialization/deserialization and switches to loader-based query-time object retrieval.
hack/demo/graphql-demo-server/main.go	Updates demo server to build schema once and serve GraphQL directly from the generated schema.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 76.32242% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.39%. Comparing base (e4879a3) to head (77e8ecb).
⚠️ Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/catalogd/storage/localdir.go	67.02%	21 Missing and 10 partials ⚠️
hack/demo/graphql-demo-server/main.go	0.00%	27 Missing ⚠️
internal/catalogd/graphql/graphql.go	88.11%	15 Missing and 9 partials ⚠️
internal/catalogd/service/graphql_service.go	61.90%	7 Missing and 1 partial ⚠️
internal/catalogd/graphql/validation.go	90.90%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #2732 +/- ##
==========================================
+ Coverage 70.54% 71.39% +0.85% 
==========================================
Files 143 144 +1 Lines 10617 10888 +271 ==========================================
+ Hits 7490 7774 +284 + Misses 2568 2535 -33 - Partials 559 579 +20 

Flag	Coverage Δ
e2e	34.16% <1.35%> (-0.99%)	⬇️
experimental-e2e	52.20% <44.05%> (-0.39%)	⬇️
unit	61.07% <75.81%> (+1.57%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• internal/testutil/mock/catalogdservice/mock_graphqlservice.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• internal/testutil/mock/catalogdservice/mock_graphqlservice.go: Generated file

[perdasilva]

Code Review — 10 findings (3 correctness, 1 performance, 6 cleanup/structural)

The most impactful fix would be keeping the write lock held through the entire Store() operation (or using per-catalog locks), which would resolve findings 1–3 in one change.

Finding not placeable inline:

Performance — O(N×regex) field resolution on every query (graphql.go:678)

createFieldResolver iterates all source keys calling remapFieldName (3 regex replacements each) to find the matching key. For a query returning 100 objects with 20 fields, requesting 5 fields: 5 × 100 × 20 × 3 = 30,000 regex operations per query.

FieldInfo.OriginalName is already stored during schema discovery — the resolver could capture it and do a direct O(1) map lookup:

funccreateFieldResolver(fieldName, originalNamestring) graphql.FieldResolveFn {
returnfunc(p graphql.ResolveParams) (interface{}, error) {
ifsource, ok:=p.Source.(map[string]interface{}); ok {
ifvalue, ok:=source[originalName]; ok {
returnmarshalComplexValue(value), nil
}
}
returnnil, nil
}
}

Same issue in createNestedFieldResolver (line 698) and createNestedObjectType resolver (line 653).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• internal/testutil/mock/catalogdservice/mock_graphqlservice.go: Generated file

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• internal/testutil/mock/catalogdservice/mock_graphqlservice.go: Generated file

[perdasilva]

/approve

[tmshort]

/lgtm

[tmshort]

Nit: loss of context comment

[tmshort]

Nit: loss of context comment

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva, rashmigottipati, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [perdasilva,tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment