feat(release): authenticated terminal RAG, accessible citations, and restart-safe corpus ingestion - #192
Merged
Conversation
The retrieval corpus needs authoritative, version-pinned sources with complete provenance. Use Wget2, safe archive extraction, and deterministic Javadoc seeding for exact source coverage. - register canonical source citations and archive-backed API documentation
A fresh configured Qdrant store needs its collections and indexes before CLI ingestion starts. Invoke the existing initializer in the command runner and propagate initialization failures. - cover initializer ordering and failure propagation
Add exact, complete mirrors for Docker, Dokploy, Infisical, and Doppler. Preserve canonical citation routes and source-specific fetch limits.
Add exact, complete mirrors for Docker, Dokploy, Infisical, and Doppler. Preserve canonical citation routes and source-specific fetch limits.
Register official Anthropic, Claude Code, Amp, and Tinker documentation sources. Preserve canonical citations and index Langfuse MDX documentation.
CI requires the documentation source assertion to use the repository formatter. Preserve behavior while applying Spotless-owned line wrapping. - format the citation URL expectation in the registry test
…the feature A configured CLERK_SECRET_KEY was treated as proof that API-key login could work, but Clerk gates API keys per instance. With the feature off, the availability endpoint answered 204 while every key operation failed, sending clients into a login that could never complete. Recognise Clerk's feature_not_enabled code on a 403, latch it, and report the deployment unavailable from then on. Match on the error code rather than the bare status so transient credential and edge 403s stay retryable.
The assistant emits {{hint:}}, {{background:}}, {{reminder:}}, {{warning:}},
and {{example:}} markers that the web client renders as titled callouts. The
terminal client printed them verbatim, showing internal markup where the
website shows a formatted block.
Convert each marker to a titled terminal block during streaming, holding text
back only from an opening brace pair until its close arrives because markers
routinely straddle SSE chunks. Unrecognised tokens pass through unchanged, and
an unterminated marker is released verbatim so no answer text is lost.Use the official Groovy archive without publishing its distribution envelope, canonicalize Quarkus guide URLs before redirect-rejecting fetches, and document Fedora's mutool package. - preserve existing Groovy and Quarkus citation roots - cover archive projection, discovery URLs, citations, and Fedora guidance
Add the pinned wafer-ai resource repository to the durable local embeddings queue. Keep the enforced queue census and its contract test synchronized at 23 repositories. - preserve dynamic repository discovery and sole-writer behavior
Preserve the existing local-only ingestion evidence and its remaining completion checks. Keep the run ledger reviewable alongside the queue implementation it measures. - retain completed and pending gate evidence without changing its claims
Use content fingerprints and exact Qdrant point identities as the unchanged-file contract. Keep size and modification time as marker diagnostics without triggering duplicate embeddings.
Queue every remaining documentation mirror behind the local embedding backlog. - Preserve durable predecessor proof and reject malformed selectors before Qdrant ingestion
Avoid maintaining a duplicate mirror-path list after the queued sources are fetched. - Delegate full canonical selection to the CLI all selector
Contributor
Important Review skippedToo many files! This PR contains 119 files, which is 19 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (119)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 4 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 4 issues found in the latest run.
- ✅ Fixed: Archive resume discards unpublished staging
- The archive short-circuit now flows through the shared completion tail, which discards staging only when it was seeded from the published mirror and otherwise validates and publishes the resumed archive extract.
- ✅ Fixed: Ingestion image omits writer lease
- Added scripts/qdrant_writer_lease.py to the ingestion stage COPY so acquire_qdrant_writer_lease can exec the lease helper inside the container.
- ✅ Fixed: API-key budget counts local rejects
- The filter now refunds the client verification budget when verify throws ApiKeyOperationUnavailableException, so failures where Clerk never answered no longer consume the 20/min budget.
- ✅ Fixed: Queue contract compares raw grep cuts
- The guard-order check now captures each grep result set, requires exactly one match per anchor, and extracts line numbers only afterward, matching the local-staging contract test pattern.
Or push these changes by commenting:
@cursor push 434cad135b
Preview (434cad135b)
diff --git a/Dockerfile b/Dockerfile--- a/Dockerfile+++ b/Dockerfile@@ -78,7 +78,7 @@
&& groupadd --gid 1001 appuser \
&& useradd --uid 1001 --gid 1001 --create-home --shell /bin/bash appuser
-COPY scripts/process_all_to_qdrant.sh scripts/+COPY scripts/process_all_to_qdrant.sh scripts/qdrant_writer_lease.py scripts/
COPY scripts/with_build_state_lock.sh scripts/BuildStateLock.java scripts/
COPY scripts/lib/common_qdrant.sh scripts/lib/shell_bootstrap.sh \
scripts/lib/env_loader.sh scripts/lib/embedding_preflight.sh scripts/lib/
diff --git a/scripts/fetch_all_docs.sh b/scripts/fetch_all_docs.sh--- a/scripts/fetch_all_docs.sh+++ b/scripts/fetch_all_docs.sh@@ -607,6 +607,10 @@
if [ "$existing_count" -gt 0 ]; then
log "${BLUE}ℹ Existing mirror: $existing_count HTML files${NC}"
fi
+ # A complete source-matched archive staging skips re-extraction. Whether it is+ # a disposable copy of the published mirror or an unpublished resume is decided+ # by staging_seeded_from_published_mirror in the shared completion tail below.+ local archive_source_already_complete="false"
if [ -n "$archive_format" ] \
&& [ "$FORCE_REFRESH" != "true" ] \
&& [ "$existing_count" -ge "$min_files" ] \
@@ -614,12 +618,8 @@
"$staging_directory" "$name" "$min_files" "$identity_regex" "$forbidden_identity_regex" \
&& validate_staged_documentation_identity \
"$staging_directory" "$name" "$required_identity_page" "$required_identity_text" "$expected_meta_version"; then
+ archive_source_already_complete="true"
log "${GREEN}✓ $name already fetched: $existing_count HTML files (validated pinned archive)${NC}"
- if ! discard_documentation_fetch_staging_directory "$staging_directory"; then- log "${RED}✗ Could not discard validated archive staging for $name${NC}"- return 1- fi- return 0
fi
# Proactive cleanup for known legacy Spring mirror layouts that otherwise mask incomplete fetches.
if [[ "$name" == *"Spring Framework Javadoc"* ]]; then
@@ -632,7 +632,9 @@
quarantine_versioned_reference_subdirs "$fetch_target_directory" "$name" "^1\\.1$"
fi
- log "${YELLOW}Fetching $name...${NC}"+ if [ "$archive_source_already_complete" != "true" ]; then+ log "${YELLOW}Fetching $name...${NC}"+ fi
if ! mkdir -p "$fetch_target_directory" || ! cd "$fetch_target_directory"; then
if [ -n "$staging_directory" ] \
&& ! discard_documentation_fetch_staging_directory "$staging_directory"; then
@@ -664,14 +666,18 @@
"$partial_mirror_allowed" \
"${plain_text_required_fragments[@]}" || documentation_fetch_status=$?
elif [ -n "$archive_format" ]; then
- fetch_documentation_archive \- "$url" \- "$fetch_target_directory" \- "$name" \- "$min_files" \- "$archive_format" \- "$archive_strip_components" \- "$archive_publication_root" || documentation_fetch_status=$?+ if [ "$archive_source_already_complete" = "true" ]; then+ DOCUMENTATION_SOURCE_ALREADY_COMPLETE="true"+ else+ fetch_documentation_archive \+ "$url" \+ "$fetch_target_directory" \+ "$name" \+ "$min_files" \+ "$archive_format" \+ "$archive_strip_components" \+ "$archive_publication_root" || documentation_fetch_status=$?+ fi
elif [ -n "$java_release" ] || [ "$javadoc_seed" = "true" ]; then
local java_api_fetch_required="true"
if ! generate_javadoc_seed "$url" "$fetch_target_directory" \
diff --git a/scripts/test_queued_platform_documentation_staging_contract.sh b/scripts/test_queued_platform_documentation_staging_contract.sh--- a/scripts/test_queued_platform_documentation_staging_contract.sh+++ b/scripts/test_queued_platform_documentation_staging_contract.sh@@ -74,11 +74,26 @@
grep -Fq './scripts/process_all_to_qdrant.sh --doc-sets=all' "$queued_launcher" \
|| fail_queued_platform_test "queued job does not invoke the canonical documentation registry"
-completion_gate_line="$(grep -n "expected_staging_invocation_journal | grep -Fq 'LOCAL_STAGING_COMPLETE'" "$queued_launcher" | tail -1 | cut -d: -f1)"-invocation_receipt_line="$(grep -n '^mv -- "$STAGING_INVOCATION_RECEIPT.next"' "$queued_launcher" | cut -d: -f1)"-writer_lease_line="$(grep -n '^acquire_qdrant_writer_lease$' "$queued_launcher" | cut -d: -f1)"-source_refresh_line="$(grep -n './scripts/fetch_all_docs.sh --doc-sets=' "$queued_launcher" | cut -d: -f1)"-ingestion_start_line="$(grep -n './scripts/process_all_to_qdrant.sh --doc-sets=' "$queued_launcher" | cut -d: -f1)"+completion_gate_locations="$(grep -n "expected_staging_invocation_journal | grep -Fq 'LOCAL_STAGING_COMPLETE'" "$queued_launcher" || true)"+invocation_receipt_locations="$(grep -n '^mv -- "$STAGING_INVOCATION_RECEIPT.next"' "$queued_launcher" || true)"+writer_lease_locations="$(grep -n '^acquire_qdrant_writer_lease$' "$queued_launcher" || true)"+source_refresh_locations="$(grep -n './scripts/fetch_all_docs.sh --doc-sets=' "$queued_launcher" || true)"+ingestion_start_locations="$(grep -n './scripts/process_all_to_qdrant.sh --doc-sets=' "$queued_launcher" || true)"+for guard_order_locations in \+ "$completion_gate_locations" \+ "$invocation_receipt_locations" \+ "$writer_lease_locations" \+ "$source_refresh_locations" \+ "$ingestion_start_locations"; do+ if [ "$(printf '%s\n' "$guard_order_locations" | sed '/^$/d' | wc -l | tr -d ' ')" -ne 1 ]; then+ fail_queued_platform_test "queued guard-order anchors do not each match exactly once"+ fi+done+completion_gate_line="${completion_gate_locations%%:*}"+invocation_receipt_line="${invocation_receipt_locations%%:*}"+writer_lease_line="${writer_lease_locations%%:*}"+source_refresh_line="${source_refresh_locations%%:*}"+ingestion_start_line="${ingestion_start_locations%%:*}"
if [ "$completion_gate_line" -ge "$invocation_receipt_line" ] \
|| [ "$invocation_receipt_line" -ge "$source_refresh_line" ] \
|| [ "$source_refresh_line" -ge "$writer_lease_line" ] \
diff --git a/src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationFilter.java b/src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationFilter.java--- a/src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationFilter.java+++ b/src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationFilter.java@@ -105,6 +105,7 @@
try {
verifiedKey = apiKeyLifecycle.verify(presentedSecret.get());
} catch (ApiKeyOperationUnavailableException verificationFailure) {
+ refundClientVerification(request.getRemoteAddr());
log.error("Clerk API key verification was unavailable", verificationFailure);
writeError(response, HttpStatus.SERVICE_UNAVAILABLE, VERIFICATION_UNAVAILABLE_MESSAGE);
return;
@@ -134,6 +135,17 @@
return verificationCount.incrementAndGet() <= CLERK_VERIFICATION_MAXIMUM_REQUESTS_PER_MINUTE;
}
+ // The budget tracks answered Clerk verifications only. An unavailable outcome+ // (missing credential, latched-off feature, exhausted capacity, or an+ // unanswered provider call) must not consume it, or purely local failures+ // could 429 later real verifications.+ private void refundClientVerification(String clientAddress) {+ AtomicInteger verificationCount = clientVerificationCounts.getIfPresent(clientAddress);+ if (verificationCount != null) {+ verificationCount.decrementAndGet();+ }+ }+
private static Optional<String> presentedApiKey(HttpServletRequest request) {
if (!AUTHENTICATED_USER_PATH.equals(request.getRequestURI())
&& !API_KEY_REVOCATION_PATH.equals(request.getRequestURI())You can send follow-ups to the cloud agent here.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:990f5ea223
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces security-sensitive authentication changes plus critical ingestion/write-coordination behavior across many layers, requiring careful human verification beyond this automated review.
Pull request overview
This PR expands Java Chat’s authenticated surfaces and ingestion pipeline reliability: it adds Clerk-backed API-key authentication for a packaged CLI, improves “source record” fidelity in streamed answers/citations, and hardens long-running documentation ingestion so it survives restarts and enforces a single Qdrant writer.
Changes:
- Add Clerk API-key lifecycle + auth filter path (API key availability probe,
/api/meidentity,/api/me/api-keyrevocation, CSRF exemption only for API-key bearers). - Strengthen ingestion correctness and durability (exact Qdrant point identity checks, writer lease, restart-safe staging/queue scripts, fail-closed doc-set selection + Qdrant readiness gate).
- Improve UX/accessibility and testing coverage (CLI authorization route, citation/source semantics in prompts, keyboard-focusable message regions, Playwright+axe checks).
File summaries
| File | Description |
|---|---|
| src/test/java/com/williamcallahan/javachat/web/BrowserErrorResponseIntegrationTest.java | Adds SPA forwarding coverage for /cli/authorize. |
| src/test/java/com/williamcallahan/javachat/web/AuthenticatedUserEndpointSecurityIntegrationTest.java | Adds integration coverage for API-key auth + revocation. |
| src/test/java/com/williamcallahan/javachat/web/AuthenticatedUserControllerTest.java | Updates controller unit tests for Authentication-based identity + key revocation. |
| src/test/java/com/williamcallahan/javachat/web/ApiKeyAvailabilityControllerTest.java | New unit test for API-key availability endpoint behavior. |
| src/test/java/com/williamcallahan/javachat/service/RetrievalServiceCitationTest.java | Adds regression coverage for legacy Spring Boot SQL URL normalization. |
| src/test/java/com/williamcallahan/javachat/service/ingestion/SourceCodeFileIngestionProcessorTest.java | Updates ingestion tests for exact-point identity checks and replacement behavior. |
| src/test/java/com/williamcallahan/javachat/service/HybridVectorServiceTest.java | Adjusts exact-point identity tests for new overload usage. |
| src/test/java/com/williamcallahan/javachat/service/HtmlContentExtractorTest.java | Adds frameset-navigation exclusion coverage for Java API extraction. |
| src/test/java/com/williamcallahan/javachat/service/ChunkerTest.java | New coverage for tokenizer special-token quoting scenarios. |
| src/test/java/com/williamcallahan/javachat/service/ChatServiceTest.java | Adds coverage for “unspecified” source record version behavior. |
| src/test/java/com/williamcallahan/javachat/domain/ingestion/SourceFileLanguageTest.java | New coverage for .mdx classification. |
| src/test/java/com/williamcallahan/javachat/config/SystemPromptConfigTest.java | Adds tests for new system-prompt clauses (control flow + source fidelity). |
| src/test/java/com/williamcallahan/javachat/config/SecurityConfigTest.java | Adds tests for CSRF exemption narrowness and API-key bearer auth. |
| src/test/java/com/williamcallahan/javachat/config/QdrantIndexInitializerTest.java | Updates initializer test for fail-closed batch readiness requirement. |
| src/test/java/com/williamcallahan/javachat/cli/DocumentProcessorSelectionTest.java | Expands selector rejection semantics + verifies Qdrant readiness gating. |
| src/main/java/com/williamcallahan/javachat/web/AuthenticatedUserController.java | Adds Authentication-based identity and API-key self-revocation endpoint. |
| src/main/java/com/williamcallahan/javachat/web/ApiKeyAvailabilityController.java | Adds deployment readiness endpoint for CLI key creation flow. |
| src/main/java/com/williamcallahan/javachat/support/JavaApiPageExtractor.java | Excludes frameset navigation shells from Java API ingestion. |
| src/main/java/com/williamcallahan/javachat/service/JavaApiPageExtraction.java | Generalizes “excluded” semantics and adds navigation-page exclusion constructor. |
| src/main/java/com/williamcallahan/javachat/service/JavaApiPageDisposition.java | Adds navigation-page exclusion enum value. |
| src/main/java/com/williamcallahan/javachat/service/ingestion/SourceCodeFileIngestionProcessor.java | Switches to exact stored point identity checks (vs. count-based coverage). |
| src/main/java/com/williamcallahan/javachat/service/ingestion/LocalDocsFileIngestionProcessor.java | Excludes navigation shells, increases batching, and aligns marker checks with fingerprints. |
| src/main/java/com/williamcallahan/javachat/service/ingestion/LocalDocsDirectoryIngestionService.java | Increases local ingestion batching. |
| src/main/java/com/williamcallahan/javachat/service/HybridVectorService.java | Adds hasExactPointIdsForUrl overload for collection kind + strengthens validation. |
| src/main/java/com/williamcallahan/javachat/service/EmbeddingBatchEmbedder.java | Adds bounded request waves + transient retry behavior for embedding batches. |
| src/main/java/com/williamcallahan/javachat/service/Chunker.java | Uses ordinary encoding to handle quoted special tokens safely. |
| src/main/java/com/williamcallahan/javachat/service/ChatService.java | Prepends structured “SOURCE RECORD” headers to retained context documents. |
| src/main/java/com/williamcallahan/javachat/domain/ingestion/SourceFileLanguage.java | Adds .mdx support to documentation classification. |
| src/main/java/com/williamcallahan/javachat/config/WebMvcConfig.java | Forwards /cli/authorize to SPA index shell. |
| src/main/java/com/williamcallahan/javachat/config/SystemPromptConfig.java | Adds generated-control-flow and source-fidelity clauses to core prompt. |
| src/main/java/com/williamcallahan/javachat/config/SecurityConfig.java | Adds Clerk API-key auth filter, narrows CSRF exemption, and avoids session lane for API keys. |
| src/main/java/com/williamcallahan/javachat/config/QdrantIndexInitializer.java | Adds fail-closed readiness requirement for batch ingestion. |
| src/main/java/com/williamcallahan/javachat/cli/DocumentProcessor.java | Fail-closed doc-set selection, --app-jar, and Qdrant readiness gating. |
| src/main/java/com/williamcallahan/javachat/application/auth/VerifiedApiKey.java | New verified API-key identity record (no secret). |
| src/main/java/com/williamcallahan/javachat/application/auth/ApiKeyOperationUnavailableException.java | New exception type for provider-unavailable key operations. |
| src/main/java/com/williamcallahan/javachat/application/auth/ApiKeyLifecycle.java | New provider-agnostic API-key lifecycle interface. |
| src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationToken.java | New Authentication token for verified API keys. |
| src/main/java/com/williamcallahan/javachat/adapters/in/web/security/ClerkApiKeyAuthenticationFilter.java | New filter for API-key bearer authentication + rate limiting. |
| scripts/test_queued_platform_documentation_staging_contract.sh | Contract test ensuring queued staging remains behind sole writer. |
| scripts/test_prune_retired_java_api_vectors.sh | Extends contract to ensure writer lease usage only on validated paths. |
| scripts/test_process_all_to_qdrant_environment.sh | Extends environment contract tests (log archival, lease ordering, help/invalid args). |
| scripts/test_local_embedding_staging_contract.sh | New contract test for local-only staging + cloud credential masking. |
| scripts/test_github_sync_failure_contract.sh | Ensures no lease acquisition / network access on invalid inputs. |
| scripts/test_embedding_preflight.sh | Updates expected embedding batch sizes for new batching behavior. |
| scripts/test_documentation_fetch_publication.sh | Ensures staging copies don’t share mutable inodes with published mirrors. |
| scripts/test_documentation_archives.sh | New offline archive extraction tests (zip/tar + unsafe path rejection). |
| scripts/run_queued_platform_documentation_staging.sh | New queued staging runner gated behind predecessor completion + writer lease. |
| scripts/run_local_embedding_staging.sh | New local staging runner enforcing local Qdrant-only + pinned repo cleanliness. |
| scripts/qdrant_writer_lease.py | New Python helper for exclusive writer lease acquisition. |
| scripts/prune_retired_java_api_vectors.sh | Acquires writer lease before pruning. |
| scripts/process_github_repo.sh | Adds writer lease acquisition before cache refresh/sync/ingest. |
| scripts/process_all_to_qdrant.sh | Adds --app-jar, log archival, fail-closed selector, and lease ordering. |
| scripts/lib/embedding_preflight.sh | Updates probe batch size to align with new batch behavior. |
| scripts/lib/documentation_seed_mirrors.sh | Adds safe copy-to-staging helper; generalizes seed mirror verification names. |
| scripts/lib/documentation_archives.sh | New shared archive fetch/extract utility with safety checks. |
| scripts/lib/common_qdrant.sh | Adds shared writer lease acquisition helpers. |
| scripts/javadoc_seed.py | Generalizes deterministic Javadoc seed generation without regex HTML parsing. |
| Makefile | Adds CLI tests and additional shell contract tests. |
| infra/systemd/user/java-chat-queued-platform-documentation-staging.service | Adds queued staging unit with durable predecessor proof. |
| infra/systemd/user/java-chat-local-embedding-staging.service | Adds local staging unit with completion proof + queue trigger. |
| GATES.md | Adds run ledger for local staging completion evidence. |
| frontend/src/vite-env.d.ts | Updates deployment docs reference to Dokploy/Infisical. |
| frontend/src/styles/global.css | Adjusts accent colors and improves code/citation readability. |
| frontend/src/lib/validation/schemas.ts | Adds Zod schemas for CLI authorization and API-key creation response. |
| frontend/src/lib/services/pageMetadata.ts | Adds metadata for CLI authorization route. |
| frontend/src/lib/services/pageMetadata.test.ts | Adds tests for CLI authorization route resolution behavior. |
| frontend/src/lib/composables/clerkAuthentication.svelte.ts | Adds auth lifecycle phase + CLI API-key creation with schema validation. |
| frontend/src/lib/composables/clerkAuthentication.svelte.test.ts | Adds test ensuring malformed secret isn’t logged; updates phase expectations. |
| frontend/src/lib/components/ToastContainer.svelte | Adjusts toast container semantics (toasts provide status roles). |
| frontend/src/lib/components/MessageBubble.test.ts | Adds keyboard-focusability coverage for user message regions. |
| frontend/src/lib/components/MessageBubble.svelte | Makes user message bubble focusable region for keyboard scrolling. |
| frontend/src/lib/components/Header.test.ts | Updates tests to use auth “phase” state. |
| frontend/src/lib/components/Header.svelte | Switches auth gating from isLoaded to lifecycle phase. |
| frontend/src/lib/components/CliAuthorizationPage.test.ts | New tests for CLI authorization query validation and sign-in gating. |
| frontend/src/lib/components/CliAuthorizationPage.svelte | New CLI authorization SPA page and loopback redirect. |
| frontend/src/lib/components/CitationPanel.svelte | Improves expanded-trigger color contrast. |
| frontend/src/lib/components/ChatInput.svelte | Improves hint text contrast. |
| frontend/src/App.svelte | Adds CLI authorization view and skips CSRF refresh on that route. |
| frontend/playwright.config.ts | Adds Playwright E2E configuration for accessibility runs. |
| frontend/package.json | Adds test:browser and Playwright/axe dev deps; adjusts oxfmt ignore-path. |
| frontend/package-lock.json | Locks Playwright and axe dependencies. |
| frontend/e2e/answer-accessibility.spec.ts | New axe-based E2E checks for cited answers + CLI authorization landmarks. |
| docs/pipeline-commands.md | Updates ingestion/fetch docs (Wget2, selector semantics, seed generator naming). |
| docs/ingestion.md | Updates pipeline overview terminology and prerequisites (Wget2). |
| docs/getting-started.md | Updates prerequisites (Wget2). |
| docs/development.md | Updates deployment references (Dokploy). |
| Dockerfile | Adds “ingestion” target stage and updates ignore-file copy path. |
| cli/README.md | Adds CLI install and usage docs. |
| cli/package.json | Adds publishable CLI package manifest and commands. |
| cli/package-lock.json | Locks CLI package metadata. |
| cli/LICENSE.md | Adds CLI license text. |
| AGENTS.md | Updates repository rules summary (Dokploy/Infisical + expanded rule references). |
| .gitignore | Ignores Playwright test output and ensures cli/bin/ is tracked. |
| .github/workflows/build.yml | Adds macOS portability job, CLI pack/test checks, Playwright+axe CI runs, ingestion image build. |
| .dockerignore | Excludes Playwright output directories from Docker context. |
Review details
Files not reviewed (2)
- cli/package-lock.json: Generated file
- frontend/package-lock.json: Generated file
- Files reviewed: 102/109 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
A pinned detached-HEAD repository clone with no refs/remotes/origin/HEAD (wafer-ai gpu-perf-engineering-resources) made the branch fallback's symbolic-ref exit 128 inside a command substitution assignment, which set -e escalated into aborting the entire staging run. Branch identity is already best-effort downstream (envOrEmpty, guarded echo), so resolve it to empty instead of failing the run.
Keep singleton documentation on canonical URLs while disambiguating only files that share one citation. Migrate collided vectors and markers once, then skip unchanged files by exact point identity.
Keep one existing canonical identity per shared citation and disambiguate only the remaining pages. Avoid redundant legacy deletes while retaining collision-free markers and exact skip behavior.
Recognize the dedicated embedding batch component as the approved virtual-thread fan-out owner.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Persist runtime-owned batch settings, recover stale PID state only under the writer lease, and resume queued work against the live invocation. Keep collision-aware audit and embedding fan-out behavior consistent across retries while removing the obsolete run ledger.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
There are 4 total unresolved issues (including 1 from previous review).
Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issues. You can view the agent here.
Reviewed by Cursor Bugbot for commit 3c269b8. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Bugbot Autofix prepared fixes for both issues found in the latest run.
Or push these changes by commenting: Preview (31905be67e)diff --git a/src/main/java/com/williamcallahan/javachat/config/DocsSourceRegistry.java b/src/main/java/com/williamcallahan/javachat/config/DocsSourceRegistry.java--- a/src/main/java/com/williamcallahan/javachat/config/DocsSourceRegistry.java+++ b/src/main/java/com/williamcallahan/javachat/config/DocsSourceRegistry.java@@ -38,7 +38,13 @@
private static final String PATH_SEPARATOR_TEXT = "/";
private static final String GITHUB_BLOB_PATH = "/blob/";
private static final String GITHUB_TREE_PATH = "/tree/";
- private static final String INGESTION_IDENTITY_QUERY_PREFIX = "?java-chat-mirror=";+ /**+ * Starts with a literal dot because {@code LocalStoreService} safe names keep dots while rewriting other+ * separators to underscores; a disambiguated identity therefore never extends the canonical URL's+ * {@code safeName + "_"} parsed-chunk prefix, so pruning the canonical owner cannot sweep sibling files.+ */+ private static final String INGESTION_IDENTITY_MARKER = ".java-chat-mirror.";+
private static final String SPRING_FRAMEWORK_MARKER = "spring-framework";
private static final String SPRING_FRAMEWORK_LEGACY_DUPLICATE_JAVADOC_PREFIX =
"docs/current/api/current/javadoc-api/";
@@ -688,11 +694,11 @@
}
}
- private static String ingestionIdentityQuery(String mirroredRelativePath) {+ private static String ingestionIdentitySuffix(String mirroredRelativePath) {
String encodedMirrorPath = Base64.getUrlEncoder()
.withoutPadding()
.encodeToString(mirroredRelativePath.getBytes(StandardCharsets.UTF_8));
- return INGESTION_IDENTITY_QUERY_PREFIX + encodedMirrorPath;+ return INGESTION_IDENTITY_MARKER + encodedMirrorPath;
}
private static Optional<String> resolveJavaSourceCitation(String sourceBaseUrl, String mirroredRelativePath) {
@@ -756,44 +762,54 @@
/**
* Resolves per-file storage identities, disambiguating only canonical URLs shared by multiple mirror files.
+ *+ * <p>Each shared citation keeps exactly one file on the bare canonical URL: the file with the shortest+ * path (lexicographic tie-break). Nested-type pages always extend the outer type page's file name, so+ * the page that actually names the cited source owns the URL regardless of inventory order, and+ * later-added siblings never reassign an existing owner's identity.</p>
*/
public static Map<Path, String> resolveMirroredIngestionIdentities(Path mirrorRoot, List<Path> documentFiles) {
Objects.requireNonNull(mirrorRoot, "mirrorRoot");
List<Path> requiredDocumentFiles = List.copyOf(Objects.requireNonNull(documentFiles, "documentFiles"));
Path absoluteMirrorRoot = mirrorRoot.toAbsolutePath().normalize();
Map<Path, String> canonicalUrlsByFile = new LinkedHashMap<>();
+ Map<String, Path> canonicalIdentityOwners = new LinkedHashMap<>();
Map<String, Integer> canonicalUrlCounts = new LinkedHashMap<>();
for (Path documentFile : requiredDocumentFiles) {
Path absoluteDocumentFile = documentFile.toAbsolutePath().normalize();
resolveMirroredPath(absoluteMirrorRoot, absoluteDocumentFile).ifPresent(canonicalUrl -> {
canonicalUrlsByFile.put(absoluteDocumentFile, canonicalUrl);
+ canonicalIdentityOwners.merge(+ canonicalUrl, absoluteDocumentFile, DocsSourceRegistry::preferredCanonicalIdentityOwner);
canonicalUrlCounts.merge(canonicalUrl, 1, Integer::sum);
});
}
- Map<String, Path> canonicalIdentityOwners = new LinkedHashMap<>();
Map<Path, String> ingestionIdentities = new LinkedHashMap<>();
canonicalUrlsByFile.forEach((documentFile, canonicalUrl) -> {
boolean citationCollision = canonicalUrlCounts.getOrDefault(canonicalUrl, 0) > 1;
- if (!citationCollision) {+ if (!citationCollision || documentFile.equals(canonicalIdentityOwners.get(canonicalUrl))) {
ingestionIdentities.put(documentFile, canonicalUrl);
return;
}
- Path canonicalIdentityOwner =- canonicalIdentityOwners.computeIfAbsent(canonicalUrl, ignoredUrl -> documentFile);- if (canonicalIdentityOwner.equals(documentFile)) {- ingestionIdentities.put(documentFile, canonicalUrl);- return;- }
String mirroredRelativePath = absoluteMirrorRoot
.relativize(documentFile)
.toString()
.replace(WINDOWS_PATH_SEPARATOR, UNIX_PATH_SEPARATOR);
- ingestionIdentities.put(documentFile, canonicalUrl + ingestionIdentityQuery(mirroredRelativePath));+ ingestionIdentities.put(documentFile, canonicalUrl + ingestionIdentitySuffix(mirroredRelativePath));
});
return Map.copyOf(ingestionIdentities);
}
+ private static Path preferredCanonicalIdentityOwner(Path currentOwner, Path candidateOwner) {+ String currentOwnerPath = currentOwner.toString();+ String candidateOwnerPath = candidateOwner.toString();+ if (candidateOwnerPath.length() != currentOwnerPath.length()) {+ return candidateOwnerPath.length() < currentOwnerPath.length() ? candidateOwner : currentOwner;+ }+ return candidateOwnerPath.compareTo(currentOwnerPath) < 0 ? candidateOwner : currentOwner;+ }+
private static boolean pathEndsWith(String normalizedRoot, String relativeMirrorPath) {
return normalizedRoot.equals(relativeMirrorPath)
|| normalizedRoot.endsWith(PATH_SEPARATOR_TEXT + relativeMirrorPath);
@@ -1129,7 +1145,7 @@
}
private static String stripIngestionIdentity(String url) {
- int identityStart = url.indexOf(INGESTION_IDENTITY_QUERY_PREFIX);+ int identityStart = url.indexOf(INGESTION_IDENTITY_MARKER);
return identityStart < 0 ? url : url.substring(0, identityStart);
}
}
diff --git a/src/test/java/com/williamcallahan/javachat/config/DocsSourceRegistryTest.java b/src/test/java/com/williamcallahan/javachat/config/DocsSourceRegistryTest.java--- a/src/test/java/com/williamcallahan/javachat/config/DocsSourceRegistryTest.java+++ b/src/test/java/com/williamcallahan/javachat/config/DocsSourceRegistryTest.java@@ -3,6 +3,7 @@
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import com.williamcallahan.javachat.config.DocsSourceRegistry.DocumentationCitationPathStyle;
import com.williamcallahan.javachat.config.DocsSourceRegistry.DocumentationSource;
@@ -147,7 +148,7 @@
Path singletonTypePage = mirrorRoot.resolve("com/fasterxml/jackson/databind/JsonNode.html");
Map<Path, String> ingestionIdentities = DocsSourceRegistry.resolveMirroredIngestionIdentities(
- mirrorRoot, List.of(outerTypePage, nestedTypePage, singletonTypePage));+ mirrorRoot, List.of(nestedTypePage, outerTypePage, singletonTypePage));
String outerStorageUrl =
ingestionIdentities.get(outerTypePage.toAbsolutePath().normalize());
String nestedStorageUrl =
@@ -156,13 +157,14 @@
ingestionIdentities.get(singletonTypePage.toAbsolutePath().normalize());
assertNotEquals(outerStorageUrl, nestedStorageUrl);
+ assertTrue(nestedStorageUrl.startsWith(outerStorageUrl + ".java-chat-mirror."));
assertEquals(
DocsSourceRegistry.normalizeDocUrl(outerStorageUrl),
DocsSourceRegistry.normalizeDocUrl(nestedStorageUrl));
assertEquals(
DocsSourceRegistry.resolveMirroredPath(mirrorRoot, outerTypePage)
.orElseThrow(),
- DocsSourceRegistry.normalizeDocUrl(outerStorageUrl));+ outerStorageUrl);
assertEquals(
DocsSourceRegistry.resolveMirroredPath(mirrorRoot, singletonTypePage)
.orElseThrow(),You can send follow-ups to the cloud agent here. |
Mirror URLs with the same public citation could leave Qdrant chunks unaccounted. Persist citation families, index their payload key, and query both identities during auditing. - Re-index document extraction when citation metadata changes and prune matching GitHub tree URLs.
A failed embedding request could leave a blocked sibling running for the full request duration. Submit cancellable tasks directly and interrupt the executor when a wave fails. - Cover interruption of a blocked sibling request.
The documentation queue was not enabled in the user manager, so it did not resume after restart. Install it under default.target and enforce that durable lifecycle in the contract test. - Keep predecessor handoff and completion proof intact.
Uh oh!
There was an error while loading. Please reload this page.
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Java Chat now supports authenticated terminal RAG queries, renders source-grounded answers accessibly, and durably fetches and embeds the complete documentation corpus without concurrent writers, lost restart state, stale removed pages, or citation-identity collisions.
Changes
Features
cli/bin/javachat.js,ClerkApiKeyVerifier,CliAuthorizationPage.svelte).DocsSourceRegistry,fetch_all_docs.sh).MessageBubble.svelte,CitationPanel.svelte,answer-accessibility.spec.ts).Bug Fixes
documentation_fetch_sources.sh,run_queued_platform_documentation_staging.sh,qdrant_writer_lease.py).DocsSourceRegistry.resolveMirroredIngestionIdentities,LocalDocsFileIngestionProcessor.pruneRemovedSourceUrls,AuditService.auditByUrl).DocumentProcessor.selectDocumentationSets,process_all_to_qdrant.sh).ApiKeyAvailabilityController,AuthenticatedUserController,clerkAuthentication.svelte.ts).ChatService,SystemPromptConfig,MessageBubble.svelte).Performance
EmbeddingBatchEmbedder.embedDocuments).QdrantPayloadFieldSchema.CITATION_URL_FIELD,FileIngestionMarkerStore.storageUrlsForCanonicalCitation).Documentation
docs/development.md,docs/ingestion.md,docs/pipeline-commands.md).Breaking Changes
None.
Validation
make testmake buildmake lint32908160098— success, including Docker ingestion/runtime smokeADVERSARIAL: Deep — independent audit-synthesizer — 6 refuted, 0 retained, 6 added — dispositions: 18 fixed, 0 filed, 0 user-decision