Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions .claude/rules/01-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,24 +104,24 @@ issuetracker/
│ └── main.go
├── internal/ # Private application code
│ ├── parser/ # ✅ Domain-agnostic parser + DB-driven rule engine (이슈 #100, #196)
│ │ ├── parser.go # ContentParser / LinkListParser interfaces + Page model
│ │ ├── rule/ # parsing_rules 기반 단일 engine
│ │ │ ├── llmgen/ # LLM 기반 selector 자동 생성 (이슈 #149)
│ │ │ ├── pathinfer/ # path_pattern 추론 알고리즘 (이슈 #173)
│ │ │ └── refiner/ # path_pattern 정밀화 polling
│ │ └── worker/ # Claim Check 기반 ParserWorker (Kafka consumer)
│ ├── locks/ # ✅ 단계 무관 distributed lock — fetcher/parser/validator 공유 (이슈 #197)
│ │ ├── ingestion_lock.go # IngestionLock (Publisher 가 Kafka enqueue 직전 사용)
│ │ └── processing_lock.go # ProcessingLock + ProcessingKey(stage, url)
│ ├── processor/ # ✅ 파이프라인 단계별 정렬 (이슈 #195)
│ ├── processor/ # ✅ 파이프라인 단계별 정렬 — 모든 stage 가 본 디렉토리 하위 (이슈 #195)
│ │ ├── fetcher/ # ✅ Web fetch + DB-driven parse 라우팅 + worker pool (이슈 #198)
│ │ │ ├── core/ # 인터페이스 + 모델 + 에러 + HTTP client + retry
│ │ │ ├── handler/ # crawler_name → Handler registry
│ │ │ ├── implementation/# chromedp / goquery 구현체
│ │ │ ├── domain/ # 사이트 chain handler + 사이트별 등록 (sources/)
│ │ │ ├── rate_limiter/ # IP 단위 token bucket
│ │ │ └── worker/ # PoolManager + KafkaConsumerPool + RetryScheduler + CircuitBreaker
│ │ ├── parser/ # ✅ Domain-agnostic parser + DB-driven rule engine + ParserWorker (이슈 #100, #196, #204)
│ │ │ ├── parser.go # ContentParser / LinkListParser interfaces + Page model
│ │ │ ├── rule/ # parsing_rules 기반 단일 engine
│ │ │ │ ├── llmgen/ # LLM 기반 selector 자동 생성 (이슈 #149)
│ │ │ │ ├── pathinfer/ # path_pattern 추론 알고리즘 (이슈 #173)
│ │ │ │ └── refiner/ # path_pattern 정밀화 polling
│ │ │ └── worker/ # Claim Check 기반 ParserWorker (Kafka consumer) + RawContentCleaner
│ │ └── validate/ # Validate worker (Validation logic)
│ ├── embedding/ # Embedding & ML (planned)
│ │ ├── model/ # Embedding models
Expand All @@ -142,8 +142,7 @@ issuetracker/
│ ├── internal/ # ✅ internal/ 패키지 테스트
│ │ ├── classifier/ # ← internal/classifier/
│ │ ├── locks/ # ← internal/locks/
│ │ ├── parser/ # ← internal/parser/
│ │ ├── processor/ # ← internal/processor/ (fetcher/{core,worker,...} + validate)
│ │ ├── processor/ # ← internal/processor/ (fetcher/{core,worker,...} + parser/{rule,worker} + validate)
│ │ └── storage/ # ← internal/storage/
│ └── pkg/ # ✅ pkg/ 패키지 테스트
│ ├── config/ # ← pkg/config/
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ make fmt # gofmt
## 디렉토리 구조 (요약)

```
cmd/ → 실행 바이너리 (crawler, processor, issuetracker, migrate)
cmd/ → 실행 바이너리 (issuetracker, processor, api, migrate, migrate-down, rldebug)
internal/ → 비공개 비즈니스 로직
pkg/ → 공개 유틸리티 (logger, config, queue, redis)
test/ → 테스트 (internal/, pkg/ 미러링)
Expand Down
8 changes: 4 additions & 4 deletions cmd/issuetracker/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,15 @@ import (
"github.com/prometheus/client_golang/prometheus"

"issuetracker/internal/locks"
"issuetracker/internal/parser/rule"
"issuetracker/internal/parser/rule/llmgen"
"issuetracker/internal/parser/rule/refiner"
parserWorker "issuetracker/internal/parser/worker"
"issuetracker/internal/processor/fetcher/core"
"issuetracker/internal/processor/fetcher/domain/general/sources/kr"
"issuetracker/internal/processor/fetcher/domain/general/sources/us"
"issuetracker/internal/processor/fetcher/handler"
crawlerWorker "issuetracker/internal/processor/fetcher/worker"
"issuetracker/internal/processor/parser/rule"
"issuetracker/internal/processor/parser/rule/llmgen"
"issuetracker/internal/processor/parser/rule/refiner"
parserWorker "issuetracker/internal/processor/parser/worker"
"issuetracker/internal/processor/validate"
"issuetracker/internal/publisher"
"issuetracker/internal/scheduler"
Expand Down
14 changes: 7 additions & 7 deletions docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,6 @@ docs/architecture/
│ │ └── http.md
│ ├── locks/ ← 단계 무관 distributed lock (이슈 #197)
│ │ └── README.md ← ProcessingLock + IngestionLock (Redis SETNX)
│ ├── parser/
│ │ ├── README.md ← Domain-Agnostic Parser + Claim Check Worker
│ │ └── rule.md ← rule.Parser (DB-driven) + llmgen + pathinfer + refiner
│ ├── processor/ ← 파이프라인 단계별 정렬 (이슈 #195)
│ │ ├── README.md
│ │ ├── fetcher/ ← Web fetch + DB-driven parse + rate limit + worker pool (이슈 #198)
Expand All @@ -50,6 +47,9 @@ docs/architecture/
│ │ │ ├── implementation.md ← chromedp / goquery
│ │ │ ├── rate_limiter.md
│ │ │ └── worker.md ← PoolManager + RetryScheduler + CircuitBreaker
│ │ ├── parser/ ← Domain-Agnostic Parser + Claim Check Worker (이슈 #204)
│ │ │ ├── README.md
│ │ │ └── rule.md ← rule.Parser (DB-driven) + llmgen + pathinfer + refiner
│ │ └── validate.md ← news/community Validator
│ ├── publisher.md ← chained job 발행 + IngestionLock
│ ├── scheduler.md ← seed job 주기적 발행
Expand Down Expand Up @@ -108,7 +108,7 @@ docs/architecture/
┌─────────────────────┐ rule.Parser (DB-driven, parsing_rules 테이블)
│ ParserWorker │ ├─ ErrNoRule → llmgen.Generator (async, enabled=false)
│ internal/parser/ │ └─ list 페이지 → Publisher.PublishBatch (chained CrawlJob)
│ internal/processor/parser/ │ └─ list 페이지 → Publisher.PublishBatch (chained CrawlJob)
Comment thread
juhy0987 marked this conversation as resolved.
│ worker │
└──────┬──────────────┘
│ Content → ContentService.Store → ContentRef
Expand All @@ -130,7 +130,7 @@ docs/architecture/
└──────────────────────────────────────────────────────────────────────────┘

부수 흐름:
• Refiner (internal/parser/rule/refiner) — 주기적 polling 으로 llm-auto 규칙의
• Refiner (internal/processor/parser/rule/refiner) — 주기적 polling 으로 llm-auto 규칙의
path_pattern 을 sample_urls 기반으로 정밀화 (이슈 #173).
• Classifier (internal/classifier) — gRPC 기본 + HTTP fallback 으로 ELArchive Classifier
호출 — 향후 enrich 단계에서 사용.
Expand All @@ -149,7 +149,7 @@ docs/architecture/
| `issuetracker.crawl.normal` | [scheduler](../../internal/scheduler/) / publisher | `issuetracker-crawler-workers` (동일) |
| `issuetracker.crawl.low` | [scheduler](../../internal/scheduler/) / publisher | `issuetracker-crawler-workers` (동일) |
| `issuetracker.fetched` | [processor/fetcher/worker](../../internal/processor/fetcher/worker/) | `issuetracker-parsers` (`GroupParsers`) |
| `issuetracker.normalized` | [parser/worker](../../internal/parser/worker/) | `issuetracker-validators` (`GroupValidators`) |
| `issuetracker.normalized` | [processor/parser/worker](../../internal/processor/parser/worker/) | `issuetracker-validators` (`GroupValidators`) |
| `issuetracker.validated` | [processor/validate](../../internal/processor/validate/) | (downstream — TBD) |
| `issuetracker.dlq` | 모든 stage 실패 분기 | (운영 모니터링) |

Expand Down Expand Up @@ -196,7 +196,7 @@ docs/architecture/

1. [docs/architecture/cmd/issuetracker.md](cmd/issuetracker.md) — `main()` 가 어떤 모듈을 어떤 순서로 wire 하는지
2. [docs/architecture/internal/processor/fetcher/worker.md](internal/processor/fetcher/worker.md) — Kafka consumer + lock + retry 의 핵심 패턴
3. [docs/architecture/internal/parser/README.md](internal/parser/README.md) — Claim Check + rule-driven 파싱
3. [docs/architecture/internal/processor/parser/README.md](internal/processor/parser/README.md) — Claim Check + rule-driven 파싱
4. [docs/architecture/internal/storage/README.md](internal/storage/README.md) — Repository → Service 분리 원칙
5. [docs/architecture/pkg/queue.md](pkg/queue.md) — 토픽 상수와 Kafka producer/consumer 추상

Expand Down
8 changes: 4 additions & 4 deletions docs/architecture/cmd/issuetracker.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@

- [internal/processor/fetcher/worker.md](../internal/processor/fetcher/worker.md) — PoolManager, RetryScheduler
- [internal/locks/README.md](../internal/locks/README.md) — ProcessingLock, IngestionLock
- [internal/parser/rule.md](../internal/parser/rule.md) — rule.Parser, llmgen, refiner
- [internal/parser/README.md](../internal/parser/README.md) — ParserWorker (Claim Check)
- [internal/processor/parser/rule.md](../internal/processor/parser/rule.md) — rule.Parser, llmgen, refiner
- [internal/processor/parser/README.md](../internal/processor/parser/README.md) — ParserWorker (Claim Check)
- [internal/processor/validate.md](../internal/processor/validate.md) — Validator
- [internal/scheduler.md](../internal/scheduler.md) — seed job 발행

Expand All @@ -52,8 +52,8 @@
| 함수 | 책임 | 실패 동작 |
|--------------------------|------------------------------------------------------------|------------------|
| `buildLLMProvider()` | `LLM_ENABLED` + API key 검증 → [`pkg/llm`](../../../pkg/llm/) chain provider 구성 | nil 반환 (LLM 비활성) |
| `buildLLMGenerator()` | provider nil 이면 nil — 아니면 [`llmgen.New`](../../../internal/parser/rule/llmgen/) | nil 허용 |
| `buildRefiner()` | `REFINEMENT_ENABLED` + provider 옵션 결합 → [`refiner.New`](../../../internal/parser/rule/refiner/) | nil 반환 (정밀화 비활성) |
| `buildLLMGenerator()` | provider nil 이면 nil — 아니면 [`llmgen.New`](../../../internal/processor/parser/rule/llmgen/) | nil 허용 |
| `buildRefiner()` | `REFINEMENT_ENABLED` + provider 옵션 결합 → [`refiner.New`](../../../internal/processor/parser/rule/refiner/) | nil 반환 (정밀화 비활성) |
| `verifyParsingRulesSeeded()` | 기본 파싱 규칙(seed data) 이 DB 에 정상 로드됐는지 검증 | **fail-fast (Fatal)** |

<br>
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/internal/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Go 의 `internal` 규칙에 의해 외부 모듈에서 import 할 수 없으므
| 디렉토리 | 역할 | 문서 |
|-----------------------------------------------|-----------------------------------------------------------|-----------------------------------------------------------|
| [`internal/processor/fetcher/`](../../../internal/processor/fetcher/) | 웹 크롤링 — fetch + DB-driven parse + rate limit + worker pool | [processor/fetcher/README.md](processor/fetcher/README.md) |
| [`internal/parser/`](../../../internal/parser/) | Claim Check 기반 ParserWorker (TopicFetched 소비) | [parser/README.md](parser/README.md) |
| [`internal/processor/parser/`](../../../internal/processor/parser/) | Claim Check 기반 ParserWorker (TopicFetched 소비) | [parser/README.md](processor/parser/README.md) |
Comment thread
juhy0987 marked this conversation as resolved.
| [`internal/processor/`](../../../internal/processor/) | 검증 stage (news/community Validator) | [processor/README.md](processor/README.md) |
| [`internal/classifier/`](../../../internal/classifier/) | ELArchive Classifier client (gRPC + HTTP fallback) | [classifier/README.md](classifier/README.md) |
| [`internal/publisher/`](../../../internal/publisher/) | chained CrawlJob 발행 + IngestionLock | [publisher.md](publisher.md) |
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/internal/locks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ if !acquired {

- [`internal/publisher`](../publisher.md) — `IngestionLock` (Kafka enqueue 직전 dedup)
- [`internal/processor/fetcher/worker`](../processor/fetcher/worker.md) — fetcher worker pool 의 `ProcessingLock(StageFetcher)`
- [`internal/parser/worker`](../parser/README.md) — parser worker 의 `ProcessingLock(StageParser)`
- [`internal/processor/parser/worker`](../processor/parser/README.md) — parser worker 의 `ProcessingLock(StageParser)`
- [`internal/processor/validate`](../processor/validate.md) — validator 의 `ProcessingLock(StageValidator)`

## 외부 시스템
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture/internal/processor/fetcher/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ worker ──→ handler ──→ (domain/general → fetcher) ──→ implem
└──→ core (interfaces, models, errors) ──── 모든 패키지가 의존
```

> Parser engine 은 별도 패키지 (`internal/parser/`) 로 분리됨 (이슈 #196). 상세는 [`internal/parser/README.md`](../../parser/README.md).
> Parser engine 은 별도 패키지 (`internal/processor/parser/`) 로 분리됨 (이슈 #196). 상세는 [`internal/processor/parser/README.md`](../parser/README.md).

`core` 가 sink 노드 (다른 어떤 서브패키지도 의존하지 않음 — 가장 안정적인 인터페이스 계층).

Expand Down Expand Up @@ -83,7 +83,7 @@ worker ──→ handler ──→ (domain/general → fetcher) ──→ implem
8. RawContentRef 를 issuetracker.fetched 토픽에 발행
(parser stage 로 핸드오프 — internal/parser/README.md)
(parser stage 로 핸드오프 — internal/processor/parser/README.md)
```

각 단계의 자세한 책임은 위 패키지별 문서 참조.
Expand Down
4 changes: 2 additions & 2 deletions docs/architecture/internal/processor/fetcher/core.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
| 인터페이스 | 위치 | 책임 |
|--------------------|------------------------------------------------------------------|-------------------------------------------------------|
| `Crawler` | [crawler.go](../../../../internal/processor/fetcher/core/crawler.go) | Initialize / Start / Stop / Fetch / HealthCheck |
| `Parser` | (동일 파일) | RawContent → 도메인 모델 (별도 [parser](../../parser/rule.md) 가 구현) |
| `Parser` | (동일 파일) | RawContent → 도메인 모델 (별도 [parser](../parser/rule.md) 가 구현) |
| `RateLimiter` | (동일 파일) | Wait(ctx) — 토큰 buckets 등으로 호출 빈도 제한 |
| `URLRateLimiter` | (동일 파일) | URL → IP 매핑 + per-IP rate limit (DNS 해석 필요) |
| `HTTPClient` | [http_client.go](../../../../internal/processor/fetcher/core/http_client.go) | Do(req) — 표준 `*http.Client` wrapper |
Expand Down Expand Up @@ -61,7 +61,7 @@
## 의존 관계

- **누구도 import 하지 않는** 외부 의존 외엔 standard lib + zerolog 정도
- 본 패키지를 import 하는 패키지: `domain/`, `handler/`, `worker/`, `parser/`, `rate_limiter/`, 그리고 [`internal/parser`](../../parser/README.md), [`internal/scheduler`](../../scheduler.md), [`internal/storage/service`](../../storage/service.md) 등 — 사실상 거의 모든 곳
- 본 패키지를 import 하는 패키지: `domain/`, `handler/`, `worker/`, `parser/`, `rate_limiter/`, 그리고 [`internal/processor/parser`](../parser/README.md), [`internal/scheduler`](../../scheduler.md), [`internal/storage/service`](../../storage/service.md) 등 — 사실상 거의 모든 곳

본 패키지가 변경되면 광범위한 영향이 있으므로 신중히. 새 인터페이스 추가는 자유롭지만 기존 인터페이스
시그니처 변경은 큰 PR 이 됩니다.
2 changes: 1 addition & 1 deletion docs/architecture/internal/processor/fetcher/domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,5 +100,5 @@ us.Register(registry, core.DefaultConfig(), rawSvc, crawlerProducer, log)
2. `sources/<country>/registry.go` 의 `Register` 에 site 추가
3. [`internal/scheduler.DefaultEntries`](../../scheduler.md) 에 카테고리 seed entry 추가
4. `parsing_rules` 에 host_pattern + path_pattern + selectors 시드 (마이그레이션 또는 운영자 INSERT)
— `path_pattern` 은 빈 문자열이면 catch-all 로 동작하며, [refiner](../../parser/rule.md) 가 누적 sample
— `path_pattern` 은 빈 문자열이면 catch-all 로 동작하며, [refiner](../parser/rule.md) 가 누적 sample
기반으로 정밀화합니다 (이슈 #173).
Loading
Loading