Skip to content

fix: ground AI analysis in SDK documentation - #64

Merged
nicknisi merged 2 commits into
mainfrom
nicknisi/ai-analysis-fixes
Feb 18, 2026
Merged

fix: ground AI analysis in SDK documentation#64
nicknisi merged 2 commits into
mainfrom
nicknisi/ai-analysis-fixes

Conversation

@nicknisi

Copy link
Copy Markdown
Member

Summary

The AI analysis in workos doctor was generating inaccurate findings because it had no access to actual SDK documentation or project files — it was guessing based on training data and hallucinating SDK options, config properties, and "anti-patterns" that were actually correct code.

Before

  • AI only received metadata (framework name, SDK version, env type)
  • Hardcoded "correct patterns" for Next.js in the prompt — didn't scale
  • Generated findings like "AuthKitProvider needs 'use client'" (wrong), "middleware matcher too broad" (wrong), "missing error handling in handleAuth" (wrong)

After

  • Fetches the SDK's README from GitHub at runtime — the actual source of truth for correct patterns, imports, and configuration
  • Includes project files (middleware, layout, page, callback route, env) in the prompt so the AI sees real code
  • Includes .env.local with redacted secrets so the AI can verify env var names and redirect URIs
  • Task reframed: "compare code against documentation, report deviations" instead of "guess what's wrong"
  • Scales to all SDKs — 13 package-to-repo mappings covering all AuthKit JS SDKs
  • Zero maintenance — always fetches latest README, no hardcoded patterns to update

Files Changed

  • src/doctor/agent-prompt.ts — Async README fetch, project file collection, env redaction, rewritten prompt
  • src/doctor/checks/ai-analysis.ts — Await async prompt builder
  • src/doctor/checks/ai-analysis.spec.ts — Async test updates
  • src/doctor/index.ts — Pass installDir to analysis context

Instead of hardcoding correct patterns for each SDK, fetch the actual
README from GitHub and include it in the prompt alongside project files.
The AI compares code against documentation rather than guessing.
- Fetch README for all AuthKit JS SDKs (13 package→repo mappings)
- Include .env.local with redacted secrets so AI can verify config
- Strengthen rules: drop findings that admit code is correct
- buildDoctorPrompt is now async (README fetch)
@nicknisi
nicknisiforce-pushed the nicknisi/ai-analysis-fixes branch from 91f8f46 to 6f667d5CompareFebruary 18, 2026 21:11
@nicknisinicknisi changed the title fix: ground AI analysis in SDK documentation instead of hallucinatingfix: ground AI analysis in SDK documentation insteadFeb 18, 2026
@nicknisinicknisi changed the title fix: ground AI analysis in SDK documentation insteadfix: ground AI analysis in SDK documentationFeb 18, 2026
@nicknisi
nicknisi merged commit db8d6e3 into mainFeb 18, 2026
5 checks passed
@nicknisi
nicknisi deleted the nicknisi/ai-analysis-fixes branch February 18, 2026 21:13
@github-actionsgithub-actionsBot mentioned this pull request Feb 18, 2026
lucasmotta added a commit that referenced this pull request Feb 20, 2026
…ts-skills
* origin/main: (21 commits)
chore(main): release 0.7.2 (#67)
fix: Correct issue submission links (#66)
chore(main): release 0.7.1 (#65)
fix: ground AI analysis in SDK documentation (#64)
chore(main): release 0.7.0 (#60)
fix: improve installer skill and remove shell: true from spawn calls (#63)
feat: major workos doctor overhaul — visual refresh, multi-language, AI analysis (#62)
fix: replace dotenv devDependency with inline env parser in doctor (#61)
feat: add environment, organization, and user management commands (#59)
chore(main): release 0.6.0 (#58)
feat: agent self-correction via validation feedback loop (#57)
chore(main): release 0.5.4 (#56)
fix: restore workflow_call and remove registry-url for OIDC
chore(main): release 0.5.3 (#55)
fix: trigger release.yml directly via release event for OIDC match
fix: remove registry-url from setup-node to unblock OIDC auth
chore(main): release 0.5.2 (#54)
fix: use npm publish for OIDC trusted publishing support
chore(main): release 0.5.1 (#53)
fix: remove duplicate release trigger causing publish race condition
...
# Conflicts:
#	src/lib/adapters/cli-adapter.ts
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@nicknisi