[REFAC#198] crawler/* → processor/fetcher/* 이동, 메타 이슈 #195 완료

[juhy0987]

연관 이슈

• Closes [REFAC] crawler 잔여 fetcher 영역 → processor/fetcher 이동 (이슈 #195 단계 3, 메타 이슈 close) #198

구현 내용

메타 이슈 #195 의 단계 3 (마지막) — 잔여 fetcher 영역을 internal/processor/fetcher/... 로 이동. 본 PR 머지 시 internal/crawler/ 디렉토리는 완전 제거됩니다. (메타 이슈 #195 자체는 추가 단계 4 — sub-issue #204 — 이 진행되므로 본 PR 만으로는 close 되지 않음.)

코드 변경 (commit 1)

디렉토리 이동:

• internal/crawler/core → internal/processor/fetcher/core
• internal/crawler/handler → internal/processor/fetcher/handler
• internal/crawler/implementation → internal/processor/fetcher/implementation
• internal/crawler/domain → internal/processor/fetcher/domain
• internal/crawler/rate_limiter → internal/processor/fetcher/rate_limiter
• internal/crawler/worker → internal/processor/fetcher/worker
• internal/crawler/ 자체 제거 — 메타 이슈 [REFAC] crawler → processor 정렬 — 파이프라인 단계별 디렉토리 재배치 (메타 이슈) #195 의 최종 정렬 완성

Import 갱신: 89개 .go 파일 (issuetracker/internal/crawler/ → issuetracker/internal/processor/fetcher/) sed 일괄 치환

테스트 mirror 동일 정렬:

• test/internal/crawler_core/ → test/internal/processor/fetcher/core/
• test/internal/chromedp/ → test/internal/processor/fetcher/chromedp/
• test/internal/domain/ → test/internal/processor/fetcher/domain/ (빈 디렉토리만 있어 rmdir)
• test/internal/worker/ → test/internal/processor/fetcher/worker/
• test/internal/rate_limiter/ → test/internal/processor/fetcher/rate_limiter/

Sub-issue brief 와의 차이 (의도적): sub-issue 본문은 implementation/chromedp → chromedp 같은 flatten 안을 제시했으나, 단순 prefix 교체 채택. 이유:

• package 명·sub-tree·test mirror 모두 그대로 보존 → reviewable diff 최소화
• domain/general/fetcher 가 processor/fetcher/fetcher 로 awkward 중첩되는 문제 회피
• 사이트별 register 패턴 (domain/general/sources/{kr,us}) semantic 보존
• 추가 flatten 이 필요해지면 별도 sub-issue 로 분리 (현 작업 scope 명확화)

문서 변경 (commit 2)

• docs/architecture/internal/crawler/ → docs/architecture/internal/processor/fetcher/ (디렉토리 이동)
• 모든 .md path 참조 갱신 (internal/crawler/X → internal/processor/fetcher/X)
• 이동된 docs 의 sibling-link 깊이 +1 보정 (../parser/ → ../../parser/, ../locks/ → ../../locks/, ...)
• 다른 docs 의 cross-link 갱신 ((crawler/X.md) → (processor/fetcher/X.md))
• docs/architecture/README.md 디렉토리 트리 갱신
• .claude/rules/01-architecture.md 디렉토리 트리 갱신 (cmd/ 도 현 상태로 정리)
• .cursor/rules/, .github/copilot-instructions.md, .claude/rules/06-code-style.md 의 import 예시 갱신
• CLAUDE.md cmd/ 줄 갱신 (실제 바이너리 목록 반영)

변경 후 구조

internal/
├── locks/              (이슈 #197)
│   ├── ingestion_lock.go
│   └── processing_lock.go
├── parser/             (이슈 #196)
│   ├── parser.go
│   ├── rule/
│   │   ├── llmgen/
│   │   ├── pathinfer/
│   │   └── refiner/
│   └── worker/
├── processor/          (이슈 #195 정렬 완성)
│   ├── fetcher/        (← 본 PR)
│   │   ├── core/
│   │   ├── handler/
│   │   ├── implementation/{chromedp,goquery}
│   │   ├── domain/general/{fetcher,sources}
│   │   ├── rate_limiter/
│   │   └── worker/
│   └── validate/       (기존)
├── publisher/
├── scheduler/
├── classifier/
└── storage/

internal/crawler/ 완전 제거. 모든 파이프라인 단계가 internal/processor/<stage>/ 로 정렬됨.

CI / 머지 게이트 점검

CI 운영 규약 및 Required Status Checks 단일 소스에 따라 작성합니다.

변경 영향 범위

• 영향 패키지/모듈: internal/crawler/* (전체 제거), internal/processor/fetcher/* (전체 신규), cmd/{issuetracker,processor,api}/main.go (import), 다수 docs
• 위험도(택1): Medium
  • 가장 큰 import sweep — 89개 .go 파일 + entry point 모두 영향
  • DB / Kafka / Redis / 환경변수 변경 없음
  • git mv 로 git history 보존 (renamed: 표시)
  • go build ./... && go test -race ./... 28 패키지 모두 통과 (failure 0)

Required Status Checks

• 통과 확인 대상 (PR Checks 탭에서 확인):
  • Commit Lint
  • PR Title Lint
  • Linked Issue Check
  • Format Check
  • Build
  • Test
  • Lint

롤백 계획

• git revert 2개 commit (코드 / 문서). DB / Redis / Kafka 영향 없음.
• 본 PR 머지 후 메타 이슈 [REFAC] crawler → processor 정렬 — 파이프라인 단계별 디렉토리 재배치 (메타 이슈) #195 도 함께 close — 후속 sub-issue 없음.

TODO

• (없음 — 본 PR 이 메타 이슈 [REFAC] crawler → processor 정렬 — 파이프라인 단계별 디렉토리 재배치 (메타 이슈) #195 의 마지막 단계)
• (선택) 후속 정리: internal/processor/fetcher/implementation/{chromedp,goquery} flatten / domain/general/ 단순화 등은 별도 PR 로 (이번 PR 의 reviewable diff 최소화 우선)

논의 사항

• domain/general/{fetcher,sources} 의 general 네이밍은 historical artifact — 현재 single domain group 만 있어 의미 약함. 추후 domain/news/ / domain/community/ 등 다중 도메인 도입 시 의미 회복. 본 PR 은 정렬만 (rename out of scope).
• cmd/crawler/ 가 더 이상 없으므로 (오래 전 통합됨), Makefile / docs 의 모든 잔존 reference 도 함께 정리됨.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Rate limit exceeded

@juhy0987 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 17 minutes and 16 seconds before requesting another review.

To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6edc3103-bc02-4618-8fd0-05d09a6d338f

📥 Commits

Reviewing files that changed from the base of the PR and between b03fa1b and a38e4cd.

📒 Files selected for processing (140)

• .claude/rules/01-architecture.md
• .claude/rules/02-crawler-implementation.md
• .claude/rules/04-error-handling.md
• .claude/rules/05-testing.md
• .claude/rules/06-code-style.md
• .cursor/rules/code-style.md
• .cursor/rules/development-workflow.md
• .cursor/rules/project-structure.md
• .github/CODEOWNERS
• .github/copilot-instructions.md
• README.md
• cmd/issuetracker/main.go
• docs/architecture/README.md
• docs/architecture/cmd/issuetracker.md
• docs/architecture/internal/README.md
• docs/architecture/internal/crawler/implementation.md
• docs/architecture/internal/locks/README.md
• docs/architecture/internal/parser/README.md
• docs/architecture/internal/parser/rule.md
• docs/architecture/internal/processor/fetcher/README.md
• docs/architecture/internal/processor/fetcher/core.md
• docs/architecture/internal/processor/fetcher/domain.md
• docs/architecture/internal/processor/fetcher/handler.md
• docs/architecture/internal/processor/fetcher/implementation.md
• docs/architecture/internal/processor/fetcher/rate_limiter.md
• docs/architecture/internal/processor/fetcher/worker.md
• docs/architecture/internal/processor/validate.md
• docs/architecture/internal/publisher.md
• docs/architecture/internal/scheduler.md
• docs/architecture/internal/storage/README.md
• docs/architecture/internal/storage/postgres.md
• docs/architecture/internal/storage/service.md
• docs/architecture/pkg/links.md
• docs/architecture/pkg/queue.md
• docs/architecture/pkg/redis.md
• docs/architecture/pkg/urlguard.md
• docs/ko/README.md
• examples/basic_usage/main.go
• examples/crawler_comparison/main.go
• examples/kafka_pipeline/main.go
• internal/locks/processing_lock.go
• internal/parser/parser.go
• internal/parser/rule/discovery.go
• internal/parser/rule/llmgen/generator.go
• internal/parser/rule/parser.go
• internal/parser/worker/parser_worker.go
• internal/processor/fetcher/core/crawler.go
• internal/processor/fetcher/core/errors.go
• internal/processor/fetcher/core/extractor.go
• internal/processor/fetcher/core/http2_metrics.go
• internal/processor/fetcher/core/http_client.go
• internal/processor/fetcher/core/http_status.go
• internal/processor/fetcher/core/ip_resolver.go
• internal/processor/fetcher/core/job.go
• internal/processor/fetcher/core/models.go
• internal/processor/fetcher/core/retry.go
• internal/processor/fetcher/domain/general/chain_handler.go
• internal/processor/fetcher/domain/general/convert.go
• internal/processor/fetcher/domain/general/fetcher/browser.go
• internal/processor/fetcher/domain/general/fetcher/goquery.go
• internal/processor/fetcher/domain/general/handler.go
• internal/processor/fetcher/domain/general/source_crawler.go
• internal/processor/fetcher/domain/general/sources/kr/daum/config.go
• internal/processor/fetcher/domain/general/sources/kr/naver/config.go
• internal/processor/fetcher/domain/general/sources/kr/registry.go
• internal/processor/fetcher/domain/general/sources/kr/yonhap/config.go
• internal/processor/fetcher/domain/general/sources/us/cnn/config.go
• internal/processor/fetcher/domain/general/sources/us/registry.go
• internal/processor/fetcher/domain/general/types.go
• internal/processor/fetcher/handler/handler.go
• internal/processor/fetcher/handler/noop.go
• internal/processor/fetcher/implementation/README.md
• internal/processor/fetcher/implementation/chromedp/crawler.go
• internal/processor/fetcher/implementation/chromedp/fetch.go
• internal/processor/fetcher/implementation/chromedp/graceful_timeout.go
• internal/processor/fetcher/implementation/chromedp/parse.go
• internal/processor/fetcher/implementation/chromedp/types.go
• internal/processor/fetcher/implementation/goquery/crawler.go
• internal/processor/fetcher/implementation/goquery/fetch.go
• internal/processor/fetcher/implementation/goquery/parse.go
• internal/processor/fetcher/implementation/goquery/types.go
• internal/processor/fetcher/rate_limiter/dns_resolver.go
• internal/processor/fetcher/rate_limiter/ip_registry.go
• internal/processor/fetcher/rate_limiter/token_bucket.go
• internal/processor/fetcher/worker/circuit_breaker.go
• internal/processor/fetcher/worker/manager.go
• internal/processor/fetcher/worker/pool.go
• internal/processor/fetcher/worker/resolver.go
• internal/processor/fetcher/worker/retry_scheduler.go
• internal/processor/processor.go
• internal/processor/validate/community/validator.go
• internal/processor/validate/news/validator.go
• internal/processor/validate/validator.go
• internal/processor/validate/worker.go
• internal/publisher/publisher.go
• internal/scheduler/emitter.go
• internal/scheduler/entries.go
• internal/scheduler/scheduler.go
• internal/scheduler/source.go
• internal/scheduler/throttle.go
• internal/storage/content.go
• internal/storage/postgres/content.go
• internal/storage/postgres/raw_content.go
• internal/storage/raw_content.go
• internal/storage/service/content.go
• internal/storage/service/raw_content.go
• test/internal/parser/rule/llmgen/generator_test.go
• test/internal/parser/rule/parser_test.go
• test/internal/parser/worker/cleanup_test.go
• test/internal/processor/fetcher/chromedp/graceful_timeout_test.go
• test/internal/processor/fetcher/core/errors_test.go
• test/internal/processor/fetcher/core/extractor_test.go
• test/internal/processor/fetcher/core/http2_metrics_test.go
• test/internal/processor/fetcher/core/http_client_test.go
• test/internal/processor/fetcher/core/http_status_test.go
• test/internal/processor/fetcher/core/models_test.go
• test/internal/processor/fetcher/core/raw_content_builder_test.go
• test/internal/processor/fetcher/core/retry_test.go
• test/internal/processor/fetcher/rate_limiter/dns_resolver_test.go
• test/internal/processor/fetcher/rate_limiter/ip_registry_test.go
• test/internal/processor/fetcher/rate_limiter/token_bucket_test.go
• test/internal/processor/fetcher/worker/circuit_breaker_test.go
• test/internal/processor/fetcher/worker/pool_gate_test.go
• test/internal/processor/fetcher/worker/pool_test.go
• test/internal/processor/fetcher/worker/processing_lock_pool_test.go
• test/internal/processor/fetcher/worker/retry_scheduler_test.go
• test/internal/processor/validate/community_validator_test.go
• test/internal/processor/validate/news_validator_test.go
• test/internal/processor/validate/processor_test.go
• test/internal/processor/validate/worker_test.go
• test/internal/publisher/publisher_gate_test.go
• test/internal/publisher/publisher_ingestion_lock_test.go
• test/internal/scheduler/scheduler_gate_test.go
• test/internal/scheduler/scheduler_test.go
• test/internal/scheduler/scheduler_throttle_test.go
• test/internal/scheduler/throttle_test.go
• test/internal/storage/content_service_test.go
• test/internal/storage/mock_content_repository_test.go
• test/internal/storage/mock_raw_content_repository_test.go
• test/internal/storage/raw_content_service_test.go

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch refactor/#198/processor-fetcher-migration

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Review rate limit: 0/1 reviews remaining, refill in 17 minutes and 16 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request performs a major refactoring by moving the internal/crawler package to internal/processor/fetcher. The changes include updating import paths, documentation, and test references across the entire repository to align with the new directory structure. The review feedback correctly identifies several instances in the documentation and rule files where the old test directory paths were still being referenced, providing suggestions to ensure consistency with the new architecture.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.