Skip to content

Ingest Quest professor teaching data - #190

Open
jerryzhou196 wants to merge 1 commit into
mainfrom
codex/prof-teaches-ingestion
Open

Ingest Quest professor teaching data#190
jerryzhou196 wants to merge 1 commit into
mainfrom
codex/prof-teaches-ingestion

Conversation

@jerryzhou196

@jerryzhou196 jerryzhou196 commented Jun 21, 2026

Copy link
Copy Markdown
Member

Summary

  • add a server-authenticated POST /admin/prof-teaches/ingest endpoint for the Quest scraper's prof_teaches.json
  • validate the full payload and referenced term/course IDs before any write, with path-specific client errors and in-file deduplication
  • create or resolve professors through UWFlow's existing professor-code/remap identity, then idempotently upsert term-aware scrape provenance
  • fold ingested associations into the materialized professor/course relationship and refresh both search indexes
  • keep every /api/admin/ route off the public Nginx proxy while retaining header authentication on the private API
  • include reversible migrations, focused tests, and operator documentation

This consolidates the useful behavior from #194 and the older Flow-42 ingestion branches into one implementation. It intentionally drops the obsolete fuzzy-match review files, generated match dumps, and duplicated upload endpoint.

Verification

  • CGO_CPPFLAGS="-I/opt/homebrew/Cellar/poppler/25.03.0/include" CGO_LDFLAGS="-L/opt/homebrew/Cellar/poppler/25.03.0/lib -lpoppler-cpp" GOCACHE=/private/tmp/uwflow-prof-ingest-go-cache go test ./...
  • same environment with go vet ./...
  • applied every migration from an empty PostgreSQL 15 database, then rolled back and reapplied the new migration
  • uploaded the real 3-term, 6,172-row scrape: 6,172 provenance rows became 5,414 distinct search links across 2,073 professors and 3,521 courses
  • repeated the same upload and received records_written: 0
  • verified all 2,073 professors and all 3,521 courses have associations in the materialized indexes and GET /data/search
  • rendered the Nginx template and passed nginx -t
  • git diff --check

@jerryzhou196
jerryzhou196 force-pushed the codex/prof-teaches-ingestion branch from bbfbe21 to ac84a0e Compare August 2, 2026 22:06
@jerryzhou196 jerryzhou196 changed the title Add protected professor teaching ingestion Ingest Quest professor teaching data Aug 2, 2026
@jerryzhou196
jerryzhou196 marked this pull request as ready for review August 2, 2026 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant