Skip to content

[REFAC#370] retry scheduler heartbeat 로그 압축 — 매 tick → 주기적 요약 - #371

Merged
juhy0987 merged 4 commits into
mainfrom
refactor/#370/retry-heartbeat-compression
May 12, 2026
Merged

juhy0987 merged 4 commits into
mainfrom
refactor/#370/retry-heartbeat-compression

Conversation

@juhy0987

@juhy0987 juhy0987 commented May 12, 2026

Copy link
Copy Markdown
Member

연관 이슈


구현 내용

라이브 로그 분석 결과 `retry peek returned no due items` 가 175분 동안 10,464건 (debug.log 의 24%) emit — heartbeat 의도였으나 매 tick 마다 1줄로 노이즈 큼.

변경

  • RedisRetrySchedulerConfig.HeartbeatEveryNIdleTicks 신규 필드 (default 60)
    • PollInterval 1s 기준 약 1분에 1회 heartbeat
    • 0 → legacy 동작 (매 tick) 보존
  • 메시지 명 변경: "retry peek returned no due items""retry pipeline idle heartbeat" (heartbeat 임을 명시)
  • 신규 필드:
    • heartbeat 로그: consecutive_idle_ticks — idle 누적 tick 수
    • due 로그: previous_idle_ticks — 얼마나 idle 후 처리됐는지
  • idleTicks 카운터 — due 발견 시 0 으로 reset (단일 goroutine 진입 → race-free)
  • cmd/issuetracker 와이어링 + .env.exampleRETRY_HEARTBEAT_EVERY_N_IDLE_TICKS 추가

테스트

  • TestRedisDelayedRetryScheduler_IdleHeartbeat_CompressedEveryN — N 압축 동작
  • TestRedisDelayedRetryScheduler_IdleHeartbeat_LegacyEveryTick — N=0 legacy
  • TestRedisDelayedRetryScheduler_IdleTicksResetOnDue — due 발견 시 reset + previous_idle_ticks 필드
  • safeBuffer helper — concurrent log capture race 회피

CI / 머지 게이트 점검

변경 영향 범위

  • 영향 패키지/모듈: `internal/processor/fetcher/worker` (retry_scheduler), `cmd/issuetracker`, `.env.example`
  • 위험도: Low — 로그 가시성 변경만, 실제 retry 처리 로직 불변

Required Status Checks

  • 통과 확인 대상 (PR Checks 탭에서 확인):
    • `Commit Lint`
    • `PR Title Lint`
    • `Linked Issue Check`
    • `Format Check`
    • `Build`
    • `Test`
    • `Lint`

롤백 계획

  • 운영 즉시 회귀 필요 시: `.env` 에 `RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS=0` 설정 → legacy 매 tick 로깅 복원 (재배포 불필요)

TODO


논의 사항

  • default 60 (~1분) 이 적절한지 — 너무 길면 polling alive 확인이 느려질 수 있으나 `consecutive_idle_ticks` 필드로 보완 가능

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added configurable environment variable to control retry scheduler debug logging frequency during idle polling periods, reducing log volume with tunable intervals.
  • Tests

    • Added comprehensive test coverage for idle heartbeat logging behavior across different configuration scenarios.

Review Change Stack

juhy0987 added 2 commits May 12, 2026 14:56
- RedisRetrySchedulerConfig.HeartbeatEveryNIdleTicks 신규 필드 (default 60)
- "retry peek returned no due items" → "retry pipeline idle heartbeat" 로 변경
  - 매 1s tick 마다 찍히던 메시지를 60 tick (=1분) 마다 1회로 압축
  - consecutive_idle_ticks 필드로 idle 지속 시간 노출
- due 발견 시점 로그에 previous_idle_ticks 필드 추가 — 얼마나 idle 후 처리됐는지 가시화
- cmd/issuetracker wiring + .env.example RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS 추가
- 0 = legacy 동작 보존 (매 tick)
- TestRedisDelayedRetryScheduler_IdleHeartbeat_CompressedEveryN
  - N=5 설정 시 12 tick 동안 heartbeat 가 N 압축으로 4회 이하 emit
  - 구 메시지 "retry peek returned no due items" 가 더 이상 emit 안 됨 확인
- TestRedisDelayedRetryScheduler_IdleHeartbeat_LegacyEveryTick
  - N=0 시 매 tick heartbeat (legacy 동작 보존)
- TestRedisDelayedRetryScheduler_IdleTicksResetOnDue
  - due 발견 시 idleTicks 가 0 reset + previous_idle_ticks 필드 노출
- safeBuffer helper — concurrent log write/read race 회피
Copilot AI review requested due to automatic review settings May 12, 2026 06:00
@juhy0987 juhy0987 added the refactor Code refactoring label May 12, 2026
@coderabbitai

coderabbitai Bot commented May 12, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@juhy0987 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 50 minutes and 33 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 65fd91cb-33c9-478f-8cd7-1e9bc4888df0

📥 Commits

Reviewing files that changed from the base of the PR and between 848ae33 and 87e0cfd.

📒 Files selected for processing (5)
  • cmd/issuetracker/main.go
  • internal/processor/fetcher/worker/retry_scheduler.go
  • pkg/config/config.go
  • test/internal/processor/fetcher/worker/retry_scheduler_test.go
  • test/pkg/config/config_test.go
📝 Walkthrough

Walkthrough

This PR implements configurable idle heartbeat log compression for the Redis delayed retry scheduler. It adds a HeartbeatEveryNIdleTicks field to reduce DEBUG logs during idle polling from every tick to periodic intervals (default 60), with environment variable support and test coverage validating both compressed and legacy per-tick behaviors.

Changes

Idle Heartbeat Compression

Layer / File(s) Summary
Config structure and idle tick tracking
internal/processor/fetcher/worker/retry_scheduler.go
RedisRetrySchedulerConfig gains HeartbeatEveryNIdleTicks field (default 60). RedisDelayedRetryScheduler adds idleTicks counter. Constructor validates and clamps negative values to default.
Scheduler logging and idle heartbeat emission
internal/processor/fetcher/worker/retry_scheduler.go
pollOnce now increments idleTicks when no due items exist and conditionally emits heartbeat logs based on the configured interval. Due item detection logs immediately and resets the counter.
Environment variable parsing and integration
cmd/issuetracker/main.go, .env.example
Main program imports strconv, parses RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS environment variable, and wires the validated config to the scheduler. .env.example documents the new variable and legacy option.
Test helpers and idle heartbeat validation
test/internal/processor/fetcher/worker/retry_scheduler_test.go
Test file imports are expanded. Helper functions safeBuffer, captureDebugLogger, and countLines enable safe log capture. Three new test cases validate compressed heartbeat emission, legacy per-tick behavior (when set to 0), and idle tick reset upon due processing.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A heartbeat once beat every second's chime,
Now rests and whispers every minute's time—
The rabbit hops through logs with gentler grace,
Less noise, more signal in the polling race! 🐰
Idle ticks count down 'til the next song plays. ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: compressing retry scheduler heartbeat logs from per-tick to periodic, matching the refactoring objective #370.
Linked Issues check ✅ Passed All coding requirements from issue #370 are implemented: HeartbeatEveryNIdleTicks field, idle tick counter with periodic heartbeat logic, message rename to 'retry pipeline idle heartbeat', previous_idle_ticks field in due logs, comprehensive unit tests, and environment variable/wiring in cmd.
Out of Scope Changes check ✅ Passed All changes directly support issue #370 objectives: config field additions, scheduler logic updates, test cases validating the new behavior, and environment variable setup are all in scope.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/#370/retry-heartbeat-compression

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

이 PR은 RedisDelayedRetryScheduler의 idle 상태 DEBUG 로그(heartbeat)를 매 tick 1줄에서 N tick마다 주기적으로 요약 출력하도록 변경해 운영 로그 노이즈를 줄이려는 리팩토링입니다.

Changes:

  • RedisRetrySchedulerConfig.HeartbeatEveryNIdleTicks 설정 추가 및 기본값(60) 부여, idle tick 누적 카운터(idleTicks) 도입
  • idle 로그 메시지를 "retry pipeline idle heartbeat"로 명확화하고, consecutive_idle_ticks / previous_idle_ticks 필드 추가
  • cmd/issuetracker에서 RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS 환경변수로 설정 override 및 .env.example 반영
  • 관련 단위 테스트 추가(압축 동작/legacy 동작/idle reset 검증)

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
internal/processor/fetcher/worker/retry_scheduler.go idle heartbeat 로그 압축 로직과 설정 필드/카운터 추가
cmd/issuetracker/main.go RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS env를 스케줄러 config에 와이어링
test/internal/processor/fetcher/worker/retry_scheduler_test.go heartbeat 압축/legacy/reset 동작을 검증하는 테스트 및 로그 캡처 헬퍼 추가
.env.example 신규 환경변수 문서화 및 기본값 추가

Comment thread internal/processor/fetcher/worker/retry_scheduler.go
Comment thread test/internal/processor/fetcher/worker/retry_scheduler_test.go Outdated
Comment thread test/internal/processor/fetcher/worker/retry_scheduler_test.go Outdated

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements heartbeat log compression for the Redis delayed retry scheduler to reduce log noise during idle periods. It introduces a new configuration setting, RETRY_HEARTBEAT_EVERY_N_IDLE_TICKS, which defaults to 60 ticks. The reviewer suggested centralizing the environment variable loading into the existing configuration package for consistency with the rest of the codebase.

Comment thread cmd/issuetracker/main.go Outdated
juhy0987 added 2 commits May 12, 2026 15:06
- NewRedisDelayedRetryScheduler docstring 갱신
  - 0 = legacy 의미로 보존 — 음수만 default 보정한다는 사실 명시 (Copilot 피드백)
- TestRedisDelayedRetryScheduler_IdleHeartbeat_CompressedEveryN
  - time.Sleep 추정 → Eventually 패턴으로 변경
  - heartbeat 횟수 ≤ peek 호출 횟수 / 2 (압축 비율 검증)
- TestRedisDelayedRetryScheduler_IdleHeartbeat_LegacyEveryTick
  - 동일하게 Eventually 도입 + heartbeat ≈ peek (1:1 ±1 delta) 검증
- TestRedisDelayedRetryScheduler_IdleTicksResetOnDue
  - previous_idle_ticks 정규식 [1-9] → [1-9][0-9]* (두 자릿수 이상 매칭)
- gemini-code-assist 지적: main.go 가 os.Getenv 직접 호출하는 패턴은
  나머지 cmd 가 config.LoadX 사용하는 일관성을 깸
- pkg/config 에 RetrySchedulerConfig + LoadRetryScheduler 추가
  - 음수 / 비숫자 / float 입력 명시 reject (LoadRedis 등과 동일 패턴)
  - 0 은 legacy 모드로 유효
- cmd/issuetracker: strconv import 제거 + config.LoadRetryScheduler() 사용
- TestLoadRetryScheduler_* 3건 신규 (default / override / invalid)
@juhy0987
juhy0987 merged commit ab01a91 into main May 12, 2026
8 checks passed
@juhy0987 juhy0987 self-assigned this May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor Code refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[REFACTOR] retry scheduler heartbeat 로그 압축 — 매 tick 로깅 → 주기적 요약

2 participants