[FEAT#149] LLM 자동 parsing rule 생성 wiring (1차: cmd + llmgen + parser_worker fallback)

[juhy0987]

연관 이슈

• Closes [FEATURE] rule.Parser ErrNoRule fallback — LLM 자동 규칙 생성 wiring (이슈 #100 follow-up) #149

구현 내용

이슈 #149 의 1차 PR 입니다 — Section 1 + 2 + 3 + 기본 validation + 단위 테스트 범위.

1. LLM provider 와이어링 (cmd/issuetracker/main.go)

• pkg/llm/providers blank-import 으로 gemini/openai/anthropic 자동 등록
• buildLLMGenerator 헬퍼: LoadLLM → llm.New → policy.NewFixedOrder → chain.NewWithPolicy → llmgen.New
• LLM_ENABLED=false / API key 누락 / provider 생성 실패 시 nil 반환 (graceful degrade — 기존 host 정상 파싱 유지)

2. LLM rule generator 컴포넌트 (internal/crawler/parser/rule/llmgen/)

• prompt.go: target_type 별 프롬프트 + JSON 추출 (markdown 펜스 무시, brace balancing)
• generator.go: 비동기 Enqueue → LLM 호출 → SelectorMap 파싱 → goquery validation → INSERT (enabled=false) → Resolver.Invalidate
• dedup.go: in-process inflightSet — 동일 (host, type) 동시 호출 1회로 제한
• INSERT 시 source_name="llm-auto" + enabled=false (운영자 spot-check 게이트)

3. ErrNoRule fallback 진입점 (internal/parser/worker/parser_worker.go)

• ParserWorker.llmGen 필드 추가 (nil 허용)
• handleRuleError 시그니처에 raw + storage.TargetType 추가
• rule.ErrNoRule + llmGen 활성화 → 비동기 enqueue + raw 잔존 + commit
• 다른 rule.Error (parse_failure / empty_selector) 는 기존 동작 유지

4. 기본 validation (안전망 일부)

• 생성된 selector 가 실제 HTML 에 매칭되는지 goquery.Find().Length() > 0 검증
• TargetTypePage: Title + MainContent 둘 다 매칭 필수
• TargetTypeList: ItemContainer + ItemLink 둘 다 매칭 필수
• 매칭 0건 → INSERT skip (hallucination 방어)

5. 단위 테스트 (race 검증 포함)

• 페이지/리스트 정상 INSERT (enabled=false 확인)
• validation 실패 → INSERT 안 됨
• LLM 에러 → INSERT 안 됨
• in-flight dedup: 동시 100 enqueue → LLM 1회만 호출
• 다른 host 두 개는 둘 다 호출
• 빈 HTML → LLM 호출 skip

⚠️ 본 PR 의 정책 제약 (별도 이슈로 추적)

현재 policy.NewFixedOrder("gemini") 단일 provider 만 사용 — 1000회/일 무료 한도 내 검증 목적.
운영 배포 전에 chain (gemini → openai → anthropic) 으로 확장해야 함 — 후속 이슈에서 추적.

CI / 머지 게이트 점검

변경 영향 범위

• 영향 패키지/모듈:
  • pkg/config (LLMConfig + LoadLLM 추가)
  • internal/crawler/parser/rule/llmgen (신규)
  • internal/parser/worker (handleRuleError 시그니처)
  • cmd/issuetracker (LLM provider 와이어링)
• 위험도: Medium
  • llmGen 비활성 시 (default) 기존 동작 완전 동일 — 회귀 위험 0
  • llmGen 활성 시 INSERT 가 enabled=false 라 hot path 영향 0 (운영자가 enable 전까지 lookup 미사용)
  • LLM 호출은 비동기 background goroutine — parser worker 슬롯 점유 X

Required Status Checks

• 통과 확인 대상:
  • Commit Lint
  • PR Title Lint
  • Linked Issue Check
  • Format Check
  • Build
  • Test
  • Lint

롤백 계획

• 환경변수 LLM_ENABLED=false 설정 → 즉시 비활성 (배포 재기동만 필요)
• 코드 롤백: 본 PR revert — 기존 동작 (raw 잔존만) 으로 복귀
• DB 영향: 자동 생성된 row 는 source_name="llm-auto" 로 식별 가능 — 필요 시 DELETE FROM parsing_rules WHERE source_name='llm-auto' 로 정리

TODO (후속 이슈로 추적)

• 2차 PR: Section 4 안전망 (비용 cap, audit log) + Section 5 metric
• chain 정책 변경: FixedOrder("gemini") → Hybrid 또는 명시 chain (gemini → openai → anthropic)

논의 사항

• LLM 호출 ctx 정책: 본 PR 은 background ctx 사용 — parser worker 의 message commit 과 LLM generation 의 lifecycle 을 분리. 논의 여지 있음
• INSERT 시 enabled=false 정책: 운영자 review 게이트. 자동 enable 정책은 후속 이슈에서 검토 (validation 강도와 연동)

Summary by CodeRabbit

Release Notes

• New Features
  • Added optional LLM-backed parsing rule generation with configurable provider, API key, model, and timeout settings
  • Implemented in-flight deduplication to limit concurrent LLM calls and optimize API usage
  • Added automatic HTML validation for generated parsing rules

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0e10c044-6f7e-4f93-86bd-ff046d9fb6cb

📥 Commits

Reviewing files that changed from the base of the PR and between 3aa2c70 and 3cdd9a0.

📒 Files selected for processing (7)

• cmd/issuetracker/main.go
• internal/crawler/parser/rule/llmgen/dedup.go
• internal/crawler/parser/rule/llmgen/generator.go
• internal/crawler/parser/rule/llmgen/prompt.go
• internal/parser/worker/parser_worker.go
• pkg/config/config.go
• test/internal/parser/rule/llmgen/generator_test.go

---

📝 Walkthrough

Walkthrough

This PR implements an LLM-driven automatic rule generator that generates CSS selectors for unsupported websites when the parser encounters ErrNoRule, wiring the generator into the parser worker alongside LLM configuration loading. It includes deduplication to limit concurrent LLM calls per host and validation to ensure generated selectors match the target HTML before insertion.

Changes

Cohort / File(s)	Summary
LLM Configuration pkg/config/config.go	Adds LLMConfig struct with Enabled, Provider, APIKey, Model, Timeout fields; implements LoadLLM to load config from env files and environment variables with provider-specific API key fallback (GEMINI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY).
Main Wiring cmd/issuetracker/main.go	Introduces buildLLMGenerator helper to construct a single-provider LLM chain using FixedOrder policy; initializes generator with provider, persists during runtime, and explicitly stops during shutdown to allow in-flight calls to complete; blank imports issuetracker/pkg/llm/providers.
Generator Core internal/crawler/parser/rule/llmgen/generator.go, prompt.go, dedup.go	Implements Generator with Enqueue (async spawning via context.WithoutCancel), Stop (graceful shutdown), and goroutine-safe dedup per (host, targetType) tuple; BuildPrompt creates system/user prompt pair with truncated HTML (32KB max, UTF-8 safe); extractJSON scans LLM response for first valid JSON object handling quoted strings and escapes.
Generator Validation & Insertion internal/crawler/parser/rule/llmgen/generator.go (runOnce)	Validates generated selectors by parsing HTML with goquery and matching against required fields; inserts disabled parsing-rule record on success; calls resolver.Invalidate(host, targetType) for cache flush; logs errors rather than propagating.
Parser Worker Integration internal/parser/worker/parser_worker.go	Adds optional llmGen dependency to ParserWorker; extends handleRuleError to accept raw payload and storage.TargetType; triggers asynchronous LLM enqueue via llmGen.Enqueue(...) when ErrNoRule is encountered; threads target type through category and article failure paths.
Tests test/internal/parser/rule/llmgen/generator_test.go	Comprehensive test suite validating selector extraction for page/list HTML, suppression on validation/generation failures, in-flight dedup (same host → single LLM call), concurrent independence (different hosts → independent calls), and lifecycle behavior (Stop waits for in-flight work, prevents future enqueues, allows best-effort completion even on context cancellation).

Sequence Diagram

sequenceDiagram
    participant ParserWorker
    participant Generator
    participant InflightSet
    participant LLMProvider
    participant HTML Parser
    participant Repository
    participant Resolver

    ParserWorker->>ParserWorker: handleRuleError (ErrNoRule detected)
    ParserWorker->>Generator: Enqueue(ctx, host, targetType, rawContent)
    Generator->>InflightSet: tryAcquire(host, targetType)
    alt Already in-flight
        InflightSet-->>Generator: false
        Generator-->>ParserWorker: return (dedup skipped)
    else Not in-flight
        InflightSet-->>Generator: true
        Generator->>Generator: spawn background goroutine<br/>(context.WithoutCancel)
        Generator-->>ParserWorker: return (async)
        Generator->>Generator: BuildPrompt(host, targetType, html)
        Generator->>LLMProvider: Generate(prompt, taskHint="json")
        LLMProvider-->>Generator: response
        Generator->>Generator: extractJSON(response)
        Generator->>Generator: parseSelectorMap(jsonStr)
        Generator->>HTML Parser: Parse HTML with goquery
        Generator->>Generator: validateSelectors(selectorMap, parsed HTML)
        alt Validation succeeds
            Generator->>Repository: Insert(disabled ParsingRuleRecord)
            Repository-->>Generator: ok
            Generator->>Resolver: Invalidate(host, targetType)
            Resolver-->>Generator: ok
        else Validation fails
            Generator->>Generator: log error
        end
        Generator->>InflightSet: release(host, targetType)
        InflightSet-->>Generator: ok
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• [FEATURE] host 단위 fetcher rule + validation 기반 자동 chromedp 전환 + 실패 raw republish #175: Modifies the same handleRuleError code path in parser worker to add failure counting and rule upgrade logic; this PR wires LLM enqueue into that same handler, so coordination on the failure handling flow is necessary.
• [FEATURE] parsing_rules review/enable 운영 도구 — LLM 자동 생성 row 검토 + enable 워크플로 (이슈 #149 후속) #172: Requests admin tooling to review and enable LLM-generated disabled parsing-rule rows; this PR creates those disabled rows via Generator.Insert, providing the exact data that issue's tooling will need to manage.
• [FEATURE] parsing rule URI 패턴 매칭 + 점진적 자동 정밀화 (host-only → host+path) #173: Proposes path-pattern and LLM refinement workflows that depend on the llmgen infrastructure (Generator, Enqueue, Stop); this PR delivers the foundational llmgen component that issue needs.
• [FEATURE] LLM rule generator 안전망 + metric (이슈 #149 2차) #169: Plans to extend llmgen with cost caps, audit logging, and Prometheus/Redis integration; this PR provides the core llmgen.Generator that issue will augment with observability and control-plane features.

Possibly related PRs

• [REFAC#100] 파싱 규칙 DB 일원화 — 단일 page parser engine (모든 웹페이지 도메인 중립) #145: Introduces parsing-rule types, ParsingRuleRepository.Insert, and rule.Resolver.Invalidate used throughout this PR's generator implementation; this PR directly depends on its interfaces and semantics.
• [FEAT#144] LLM 비용/성능 기반 routing 정책 + 동적 chain 합성 #167: Provides the LLM chain construction (FixedOrder policy) and TaskHint feature used in buildLLMGenerator and LLM request construction; this PR leverages both for provider initialization and prompt-aware generation.

Suggested labels

enhancement

Poem

🐰 A rule for every site now blooms,
No more the void, no more the glooms!
When selectors hide and LLM calls,
New parsing rules fill empty halls. ✨
One dedup lock, no wasteful spin—
The automation begins! 🎯

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/#149/llm-rule-generator-wiring

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 60 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request introduces an LLM-powered parsing rule generator that automatically creates selectors for hosts lacking registered rules. The implementation includes an in-process deduplication set, prompt construction logic, and integration into the parser worker. The review feedback highlights several improvement opportunities: preserving observability metadata in background tasks by using context.WithoutCancel, implementing a graceful shutdown mechanism for asynchronous generation tasks, and removing redundant code.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.