[REFAC#100] 파싱 규칙 DB 일원화 — 단일 page parser engine (모든 웹페이지 도메인 중립)

[juhy0987]

연관 이슈

• Refs [REFACTOR] 사이트별 파싱 규칙을 DB로 일원화하고 미지원 페이지는 LLM API로 자동 생성 #100
• 본 PR 은 [REFACTOR] 사이트별 파싱 규칙을 DB로 일원화하고 미지원 페이지는 LLM API로 자동 생성 #100 의 backbone (DB schema + rule-based page parser engine) 만 다루며, `Closes` 키워드는 사용하지 않음. 사이트별 parser 마이그레이션 / chain_handler 교체 / 기존 사이트별 파일 제거는 후속 PR.

Note: 위 사유로 `Linked Issue Check` 가 실패할 수 있습니다. 후속 PR 들이 누적되면 마지막 PR 에서 `Closes #100` 적용 예정.

구현 내용

#100 의 "사이트별 파싱 규칙을 DB로 일원화하고 미지원 페이지는 LLM API로 자동 생성" 작업의 backbone. 사용자 요구사항을 반영하여:

1. 인터페이스는 기존 fetcher / parser 와 동일하게 구성 (`ContentParser` / `LinkListParser` 명명, fetcher 와 통일)
2. 모든 웹페이지 처리 (뉴스 한정 X) — `parser.Page` / `parser.LinkItem` 도메인 중립 모델

6개 논리적 commit:

1. [FEAT] migration 006 — `parsing_rules` 테이블

   • 자연키: `(source_name, host_pattern, target_type, version)`
   • target_type: `'page'` (단일 컨텐츠 페이지) / `'list'` (링크-허브 페이지)
   • `selectors JSONB` — 진화 친화 (새 필드 추가 시 migration 불필요)
   • `enabled` 플래그 + `updated_at` auto-touch trigger
   • 인덱스: `(host_pattern, target_type) WHERE enabled` (Resolver 핫패스)

2. [FEAT] ParsingRuleRepository — `internal/storage/{,postgres/}parsing_rule.go`

   • `SelectorMap` — page (Title/MainContent/Author/PublishedAt/Summary/Category/Tags/Images) + list (ItemContainer/ItemLink/ItemTitle/ItemSnippet)
   • `FieldSelector{ CSS, Attribute, Multi }`
   • 인터페이스: Insert / Update / GetByID / FindActive(host, type) / List / Delete
   • 자연키 충돌 → `storage.ErrDuplicate` (pgerrcode.UniqueViolation 매핑)

3. [FEAT] rule.Resolver — `internal/crawler/parser/rule/resolver.go`

   • URL/host → 활성 Rule lookup, TTL cache (양성 5min / negative 30s)
   • goroutine-safe (sync.RWMutex)
   • 매칭 없음 → `*rule.Error{Code: ErrNoRule}` — LLM 자동 생성 fallback 진입점
   • `Invalidate(host, type)` / `InvalidateAll()` — 운영자 hook

4. [FEAT] rule.Parser — `internal/crawler/parser/rule/parser.go`

   • 단일 engine 이 `parser.ContentParser` + `parser.LinkListParser` 둘 다 구현 (compile-time interface assertion)
   • stateless / goroutine-safe — 모든 worker 단일 인스턴스 공유
   • page: Title 누락 → ErrEmptySelector / MainContent 매칭 0건 → ErrParseFailure
   • list: ItemContainer + ItemLink 필수, 상대 URL → raw.URL base 로 절대화
   • PublishedAt: 9가지 layout 순회 (RFC3339 / Korean / ISO 8601 등)

5. [FEAT] 단위 테스트 (22 케이스) — `test/internal/parser/rule/`

   • Resolver 10: cache hit/miss/negative/Invalidate/TTL 만료/host 정규화/invalid URL/nil panic
   • Parser 12: 성공 추출 (모든 필드) / NoRule / EmptySelector / ParseFailure / 빈 raw / list 절대화

6. [REFAC] 도메인 일반화 — news → 모든 웹페이지 (사용자 요구 반영)

   • 위치 이동: `internal/crawler/domain/news/rule/` → `internal/crawler/parser/rule/`
   • 신규 `internal/crawler/parser` 패키지 — 도메인 중립 인터페이스 + 모델
   • `parser.Page` / `parser.LinkItem` 모델 추가 (Title/MainContent/Author/PublishedAt/Tags/Images/Metadata)
   • `parser.ContentParser` / `parser.LinkListParser` 인터페이스
   • SelectorMap 필드 일반화: Body→MainContent, ImageURLs→Images, Date→PublishedAt, ItemSummary→ItemSnippet
   • `TargetTypeArticle` → `TargetTypePage` (호환 alias 유지)
   • news 도메인 의존 완전 제거 — 호출자가 필요 시 Page → news.NewsArticle 어댑터 작성

도메인 모델

type Page struct {
    URL         string
    Title       string
    MainContent string             // 페이지 핵심 본문
    Summary     string
    Author      string
    PublishedAt time.Time
    Language    string
    Category    string
    Tags        []string
    Images      []string
    Metadata    map[string]string  // canonical_url / og:* 등 확장
}

type LinkItem struct {
    URL     string
    Title   string
    Snippet string
}

type ContentParser interface {
    ParsePage(raw *core.RawContent) (*Page, error)
}

type LinkListParser interface {
    ParseLinks(raw *core.RawContent) ([]LinkItem, error)
}

사용 예시

import (
    \"issuetracker/internal/crawler/parser\"
    \"issuetracker/internal/crawler/parser/rule\"
    pgstore \"issuetracker/internal/storage/postgres\"
)

repo := pgstore.NewParsingRuleRepository(pool, log)
resolver := rule.NewResolver(repo)
p := rule.NewParser(resolver)

// 두 인터페이스 모두 구현 — 어느 쪽 의존성에도 주입 가능
var _ parser.ContentParser  = p
var _ parser.LinkListParser = p

page, err := p.ParsePage(raw)        // 임의 웹페이지의 핵심 내용
links, err := p.ParseLinks(rawList)  // 링크-허브 페이지의 링크 목록

CI / 머지 게이트 점검

변경 영향 범위

• 신규 패키지: `internal/crawler/parser/`, `internal/crawler/parser/rule/`, 신규 테이블 `parsing_rules`
• 신규 storage 인터페이스: `ParsingRuleRepository`
• 신규 의존성: `github.com/jackc/pgerrcode`
• 위험도: Low — 신규 코드만 추가, 기존 사이트별 parser 미변경 (후속 PR 에서 처리)

Required Status Checks

• 통과 확인 대상:
  • `Commit Lint` / `PR Title Lint` / `Format Check` / `Build` / `Test` / `Lint`
  • `Linked Issue Check` ← partial scope 라 실패 가능 (논의)

로컬 검증:

• `go test -race -count=1 ./...` 23 패키지 통과 (rule 22 케이스 + 기존 모두)
• `gofmt -l .` clean / `go build ./...` 통과
• 라이브 DB 에 migration 적용 후 INSERT/SELECT sanity check 통과

롤백 계획

• 코드: 단일 PR revert (기존 사이트별 parser 미변경이라 안전)
• DB: `migrations/down/006_create_parsing_rules.sql`

TODO (후속 PR)

본 PR 은 backbone 만. 후속 작업:

• 마이그레이션 PR — naver/daum/yonhap/cnn hardcode selector → `parsing_rules` row
• chain_handler 교체 PR — 사이트별 NaverParser/... 호출 → `rule.Parser` 일괄 교체
• 사이트별 parser.go 제거 PR
• 운영자 도구 PR — `parsing_rules` 관리 CLI / admin endpoint
• LLM 자동 생성 PR — `ErrNoRule` 시 LLM ([FEATURE] LLM 모듈 추가 — Gemini / ChatGPT / Claude 공통 인터페이스 #140 모듈) 으로 selector 초안 생성 + 휴먼 리뷰
• 마지막 PR 에서 `Closes [REFACTOR] 사이트별 파싱 규칙을 DB로 일원화하고 미지원 페이지는 LLM API로 자동 생성 #100` 명시

논의 사항

• 인터페이스 명명 — fetcher 와 통일하기 위해 `parser` 패키지명 사용 (사용자 요구). `ContentParser` / `LinkListParser` 가 `NewsFetcher` / `NewsRSSFetcher` 와 같은 형태.
• 도메인 일반화 — 처음 `news.NewsArticle` 의존이었으나 사용자 변경 요청으로 `parser.Page` 도메인 중립 모델로 재작성. 뉴스 도메인은 호출자가 어댑터로 변환 (Page → NewsArticle).
• target_type 명명 — `'article'` → `'page'` (모든 웹페이지 의미 반영). 호환을 위해 Go 측 `TargetTypeArticle` const 는 `TargetTypePage` 별칭으로 유지 — 후속 PR 에서 점진적 정리 가능.
• selectors JSONB — schema validation 부재 단점 vs 진화 친화 장점. 운영 중 새 필드 (예: `og_image`, `twitter_card`) 추가 시 migration 불필요.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores

  • Updated development configuration for VS Code and debug logging.
  • Added Go module dependency.

• Tests

  • Added comprehensive test coverage for parsing and rule resolution.

• Infrastructure

  • Implemented parsing rule system supporting CSS selector-based content extraction with caching.
  • Added database schema and persistence layer for parsing rules.

[gemini-code-assist]

Code Review

This pull request introduces a database-driven parsing system for news articles and lists, replacing hardcoded site-specific parsers. It includes a rule-based Parser using goquery, a Resolver with TTL caching, and a PostgreSQL implementation for rule storage. Review feedback recommends improving robustness by using timeout contexts for database lookups, refining custom error matching logic, enhancing diagnostic details in error messages, and ensuring consistent error handling for empty parsing results.

[juhy0987]

@gemini-code-assist review again

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[gemini-code-assist]

Code Review

This pull request introduces a database-driven parsing system designed to replace site-specific hardcoded parsers with a flexible, rule-based engine. It includes new interfaces for content and link extraction, a rule resolver with in-memory caching, and a Postgres repository implementation. The reviewer feedback identifies several critical improvements, including the necessity of passing context.Context through the parser interfaces for better execution control and tracing, addressing potential OOM risks in the unbounded cache, and removing unnecessary files and unused logger fields.

[coderabbitai]

Warning

Rate limit exceeded

@juhy0987 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 53 minutes and 40 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b4fc5e3b-569d-4314-ba5e-44462954f02b

📥 Commits

Reviewing files that changed from the base of the PR and between 544e18c and 61ba04d.

📒 Files selected for processing (1)

• internal/crawler/parser/rule/parser.go

📝 Walkthrough

Walkthrough

This pull request introduces a database-driven parsing rule system for the crawler. It adds domain models (Page, LinkItem), storage contracts for parsing rules, PostgreSQL repository implementation, a rule resolver with TTL caching, and a rule-based parser engine that normalizes content extraction across sites.

Changes

Cohort / File(s)	Summary
Core Domain Models internal/crawler/parser/parser.go	Defines Page struct with title, content, metadata fields and LinkItem struct for normalized links. Introduces ContentParser and LinkListParser interfaces for transforming raw content.
Rule Parsing Engine internal/crawler/parser/rule/errors.go, internal/crawler/parser/rule/parser.go, internal/crawler/parser/rule/resolver.go	Error classification system with ErrorCode constants (ErrInvalidURL, ErrNoRule, ErrEmptySelector, ErrParseFailure). Rule-based parser implementing both parser interfaces with CSS selector extraction, datetime parsing, and relative URL resolution. Resolver mapping URLs to active rules via host pattern matching with configurable TTL caching for both positive and negative results.
Storage Layer internal/storage/parsing_rule.go, internal/storage/postgres/parsing_rule.go	Storage contracts defining TargetType, FieldSelector, SelectorMap, ParsingRuleRecord, and ParsingRuleRepository interface. PostgreSQL implementation using pgx with JSON marshaling for selectors, constraint mapping, and repository methods for CRUD and rule discovery.
Database Migrations migrations/up/006_create_parsing_rules.sql, migrations/down/006_create_parsing_rules.sql	Creates parsing_rules table with host pattern, target type, version, enabled flag, JSONB selectors, and timestamps. Adds uniqueness constraint and indexes for lookups. Provides rollback that drops table, trigger, and indexes.
Test Suites test/internal/parser/rule/parser_test.go, test/internal/parser/rule/resolver_test.go	Page parsing tests covering title/author/content/tags/images extraction, datetime parsing, error cases, and URL absolutization. Resolver tests validating rule resolution, host normalization, caching behavior, TTL expiration, cache invalidation, and panic conditions.
Project Configuration .gitignore, go.mod	Uncomments .vscode/ ignore pattern and adds debug log files and Claude scheduler lock file. Adds indirect dependency on github.com/jackc/pgerrcode.

Sequence Diagram(s)

sequenceDiagram
    participant Client as Parser Client
    participant Parser as Rule Parser<br/>(parser.go)
    participant Resolver as Rule Resolver<br/>(resolver.go)
    participant Cache as TTL Cache
    participant Repo as PostgreSQL Repo<br/>(parsing_rule.go)
    participant DB as Database

    Client->>Parser: ParsePage(ctx, RawContent)
    Parser->>Resolver: ResolveByURL(ctx, url, TargetTypePage)
    Resolver->>Cache: Check (host, type)
    alt Cache Hit (positive)
        Cache-->>Resolver: ParsingRuleRecord
    else Cache Miss or Expired
        Resolver->>Repo: FindActive(ctx, host, type)
        Repo->>DB: SELECT * FROM parsing_rules<br/>WHERE host_pattern, target_type, enabled<br/>ORDER BY version DESC LIMIT 1
        DB-->>Repo: ParsingRuleRecord
        Repo-->>Resolver: ParsingRuleRecord
        Resolver->>Cache: Store with TTL
        Cache-->>Resolver: Done
    else Not Found
        Resolver->>Cache: Store ErrNoRule (negative cache)
        Cache-->>Resolver: Done
        Resolver-->>Parser: ErrNoRule
    end
    Resolver-->>Parser: ParsingRuleRecord
    Parser->>Parser: Extract via CSS selectors<br/>(title, content, metadata)
    Parser->>Parser: Parse datetime layouts<br/>Absolutize URLs
    Parser-->>Client: Page{Title, MainContent, Metadata}

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 Rules from the database, now we sing!
No more code for every site—
Just selectors, clean and bright,
Let the resolver cache take wing,
One parser rules them all just right! 🚀

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 51.22% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The Korean title correctly identifies the core change: centralizing parsing rules in DB with a unified, domain-neutral page parser engine, matching the PR's primary objective.
Linked Issues check	✅ Passed	Code implements DB schema, repository, resolver with TTL cache, generic parser engine with selector validation and URL absolutization, parsing rule models, tests—directly addressing issue #100's primary objectives.
Out of Scope Changes check	✅ Passed	All changes are tightly scoped to issue #100 backbone: .gitignore/go.mod updates, parser domain models, rule resolver/parser engines, storage contracts/implementation, migrations, and tests. No unrelated refactoring.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/#100/parsing-rule-db-engine

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 53 minutes and 40 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.vscode/settings.json:
- Around line 2-3: The workspace setting "chat.tools.terminal.autoApprove"
currently enables auto-approval for the "gh" terminal tool; change this to
disable committed auto-approval by removing the "gh": true entry or setting it
to false in .vscode/settings.json, and, if needed, recommend moving any personal
preference to user-level settings instead of committing it to the repo so
contributors don't inherit auto-approve behavior for the GitHub CLI.

In `@internal/crawler/parser/rule/parser.go`:
- Around line 86-93: The selector validation currently only checks for nil
pointers (e.g., rule.Selectors.Title) but must also treat zero-value selectors
(empty CSS string) as missing; update the validation in parser.go so each
required selector (Title, MainContent and the selectors checked by the list/page
guards) is rejected when selector == nil OR selector.CSS == "" and return the
existing &Error{Code: ErrEmptySelector, Message: "...", URL: raw.URL,
TargetType: string(storage.TargetTypePage)} (or the appropriate TargetType for
list rules) instead of letting empty-CSS selectors fall through to
ErrParseFailure/stale-rule handling; apply the same CSS-empty check to the other
validation sites mentioned (the checks around lines 112-120, 145-152, and
184-191) so all required selectors fail with ErrEmptySelector when CSS is empty.

In `@internal/crawler/parser/rule/resolver.go`:
- Around line 118-123: ResolveByURL currently reduces the input to u.Hostname(),
preventing any host_pattern or path-scoped/wildcard rules from matching; update
ResolveByURL to pass a normalized full host/path (or both host and path) instead
of only the hostname and adjust extractHost (or add extractHostAndPath) to
return the components needed; then modify Resolver.Resolve to accept and forward
the full host pattern or URL-path pair to storage lookup logic (so it can
evaluate wildcard subdomains and path rules against host_pattern and any
path_pattern fields) and ensure storage lookup functions (used by Resolve)
perform pattern matching instead of exact equality.

In `@internal/storage/parsing_rule.go`:
- Around line 82-84: The struct field comment for TargetType is stale—update the
inline comment for the TargetType field (seen next to the TargetType symbol) to
reflect current valid values "page" | "list" instead of `"article" | "list"`,
keeping the rest of the field comments (HostPattern, Version) unchanged so
maintainers see the accurate schema in the parsing rule declaration.

In `@internal/storage/postgres/parsing_rule.go`:
- Around line 109-130: FindActive can return a nondeterministic row when
multiple enabled rules share host_pattern, target_type and version because the
SQL only filters by host and type; change the lookup to include source_name as
part of the key. Update the sqlFindActiveParsingRule SQL constant to add
source_name = $1 in the WHERE clause (shifting other params to $2/$3) and change
the pgParsingRuleRepository.FindActive signature to accept sourceName (e.g.,
func (r *pgParsingRuleRepository) FindActive(ctx context.Context, sourceName,
host string, targetType storage.TargetType) ) and pass sourceName into
r.pool.QueryRow so the query deterministically returns the intended rule;
alternatively, if you cannot change the signature now, add a deterministic
tie-breaker like ORDER BY version DESC, source_name ASC to the SQL constant.

In `@migrations/up/006_create_parsing_rules.sql`:
- Around line 41-43: The UNIQUE constraint parsing_rules_natural_key_unique
currently includes source_name but your runtime lookup only uses (host_pattern,
target_type) and selects the highest version, so duplicates across source_name
can cause wrong matches; update the constraint to enforce uniqueness on the
columns used by the resolver by replacing UNIQUE (source_name, host_pattern,
target_type, version) with UNIQUE (host_pattern, target_type, version) so that
only one rule exists per host_pattern/target_type/version (and adjust any
related index or migration statements to match).

In `@test/internal/parser/rule/resolver_test.go`:
- Around line 145-152: The test currently discards errors returned by r.Resolve
calls which can mask regressions; change each call like the two initial Resolve
calls and the post-Invalidate Resolve to capture the returned error and assert
it with assert.NoError(t, err) (or require.NoError), e.g., call r.Resolve(...),
check NoError before asserting repo.calls(), and do the same for the other
similar blocks (the ones around lines 162-170 and 178-184); keep the
Invalidate("news.example.com", storage.TargetTypePage) and repo.calls()
assertions unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8a314cbe-d06f-42fc-a1b5-37af3b758548

📥 Commits

Reviewing files that changed from the base of the PR and between 08ad2d4 and 237fed0.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (13)

• .gitignore
• .vscode/settings.json
• go.mod
• internal/crawler/parser/parser.go
• internal/crawler/parser/rule/errors.go
• internal/crawler/parser/rule/parser.go
• internal/crawler/parser/rule/resolver.go
• internal/storage/parsing_rule.go
• internal/storage/postgres/parsing_rule.go
• migrations/down/006_create_parsing_rules.sql
• migrations/up/006_create_parsing_rules.sql
• test/internal/parser/rule/parser_test.go
• test/internal/parser/rule/resolver_test.go

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

migrations/up/006_create_parsing_rules.sql (2)

29-30: Constrain selectors to JSON objects.

selectors currently accepts any JSONB value. A manual SQL/admin insert like [] or "foo" would satisfy the schema but break once the repository reads it back into storage.SelectorMap. Adding a top-level shape check now will harden the DB-backed contract before admin tooling lands.

💡 Suggested schema hardening

   CONSTRAINT parsing_rules_version_positive
     CHECK (version > 0),
+  CONSTRAINT parsing_rules_selectors_object
+    CHECK (jsonb_typeof(selectors) = 'object'),
   -- Lookup 키 (host_pattern, target_type, version) 와 동일하게 UNIQUE.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@migrations/up/006_create_parsing_rules.sql` around lines 29 - 30, Add a CHECK
constraint to ensure the selectors JSONB column is always a JSON object: modify
the migration that defines the selectors column (column name selectors in the
CREATE TABLE / ALTER TABLE block) to include a constraint such as CHECK
(jsonb_typeof(selectors) = 'object') and give the constraint a clear name (e.g.,
selectors_is_object) so future migrations/rollbacks can reference it; keep the
DEFAULT '{}::jsonb' and ensure any existing rows are validated/converted before
applying the constraint in a migration that can run safely in production.

---

50-53: Align the lookup index with FindActive's ORDER BY version DESC.

The resolver hot path filters by (host_pattern, target_type) and then picks the highest enabled version. This partial index still stores enabled as a key and omits version, so PostgreSQL may still scan/sort multiple enabled rows for the same host/type. Indexing version DESC here would better match the query shape.

⚙️ Suggested index shape

 CREATE INDEX IF NOT EXISTS idx_parsing_rules_lookup
-  ON parsing_rules (host_pattern, target_type, enabled)
+  ON parsing_rules (host_pattern, target_type, version DESC)
   WHERE enabled = TRUE;

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@migrations/up/006_create_parsing_rules.sql` around lines 50 - 53, The partial
index idx_parsing_rules_lookup should include version DESC to match the resolver
hot path (FindActive) which filters by host_pattern and target_type and orders
by version DESC; update the migration to create the index on parsing_rules using
columns (host_pattern, target_type, version DESC) with the same WHERE enabled =
TRUE predicate so PostgreSQL can satisfy the ORDER BY without additional sorting
when FindActive queries for the highest enabled version.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@internal/crawler/parser/rule/parser.go`:
- Around line 205-216: validateRaw currently treats whitespace-only HTML as
non-empty; update the check in validateRaw (function name validateRaw, return
&Error{...} logic) to use strings.TrimSpace(raw.HTML) and treat trimmed == "" as
empty so it returns the ErrParseFailure error with raw.URL; add the strings
import if missing and keep the same error construction path.
- Around line 168-195: The current loop appends to items only when
extractFieldFromSelection(container, rule.Selectors.ItemLink) is non-empty, so
when ItemContainer matches but every ItemLink is empty we still hit the
"ItemContainer selector matched 0 elements" error; change the parsing logic in
the doc.Find(... rule.Selectors.ItemContainer ...) block to track both
containerCount and validLinkCount (or separately count containers and appended
items): if containerCount == 0 keep the existing ErrParseFailure message about
ItemContainer, but if containerCount > 0 and validLinkCount == 0 return a
distinct ErrParseFailure (or clearer Error) whose Message explicitly states that
ItemContainer matched elements but ItemLink resolved to empty values (include
raw.URL and TargetType as the existing error does), using the same Error struct
and preserve other fields.

---

Nitpick comments:
In `@migrations/up/006_create_parsing_rules.sql`:
- Around line 29-30: Add a CHECK constraint to ensure the selectors JSONB column
is always a JSON object: modify the migration that defines the selectors column
(column name selectors in the CREATE TABLE / ALTER TABLE block) to include a
constraint such as CHECK (jsonb_typeof(selectors) = 'object') and give the
constraint a clear name (e.g., selectors_is_object) so future
migrations/rollbacks can reference it; keep the DEFAULT '{}::jsonb' and ensure
any existing rows are validated/converted before applying the constraint in a
migration that can run safely in production.
- Around line 50-53: The partial index idx_parsing_rules_lookup should include
version DESC to match the resolver hot path (FindActive) which filters by
host_pattern and target_type and orders by version DESC; update the migration to
create the index on parsing_rules using columns (host_pattern, target_type,
version DESC) with the same WHERE enabled = TRUE predicate so PostgreSQL can
satisfy the ORDER BY without additional sorting when FindActive queries for the
highest enabled version.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bedd3622-4b8d-4a44-816d-0ab57006b4e6

📥 Commits

Reviewing files that changed from the base of the PR and between 237fed0 and 544e18c.

📒 Files selected for processing (5)

• .gitignore
• internal/crawler/parser/rule/parser.go
• internal/storage/parsing_rule.go
• migrations/up/006_create_parsing_rules.sql
• test/internal/parser/rule/resolver_test.go

✅ Files skipped from review due to trivial changes (2)

• .gitignore
• internal/storage/parsing_rule.go