Skip to content

support zapcore.BufferedWriteSyncer as an option. - #44

Merged
bb7133 merged 1 commit into
pingcap:masterfrom
bb7133:bb7133/buffered
Sep 17, 2025
Merged

support zapcore.BufferedWriteSyncer as an option.#44
bb7133 merged 1 commit into
pingcap:masterfrom
bb7133:bb7133/buffered

Conversation

@bb7133

@bb7133bb7133 commented Sep 16, 2025

Copy link
Copy Markdown
Member

This PR extends our logging library to support buffered log writing by wrapping the underlying WriteSyncer with zapcore.BufferedWriteSyncer.

Motivation

Previously, logs were written synchronously on every call, which can cause unnecessary I/O overhead under high log volume. With buffering enabled, log entries are accumulated in memory and only flushed to disk:

  • Periodically (on a configurable flush interval), or
  • When the buffer reaches a specified size.

This reduces the frequency of disk writes and improves performance, while still guaranteeing that logs are eventually persisted.

Implementation

  • Update InitLogger() to introduce BufferedWriteSyncer.
  • Configurable parameters:
    • Buffer size: maximum number of bytes held in memory before a flush is triggered.
    • Flush interval: maximum duration logs may remain in memory before being flushed.
  • We did not add a Close() method for now, because:
    • zap.Logger itself does not expose Close(), only Sync().
    • Our current use cases rely on application-wide or test-level lifecycle management, where calling Sync() at shutdown is sufficient.
    • Introducing Close() would require additional lifecycle handling (e.g. stopping background flush goroutines), which we may revisit if needed in the future.

Testing

  • Added unit tests that verify:
    • Logs are not written to disk before flush is triggered.
    • Flush occurs after the configured time interval.
    • Flush occurs immediately once the buffer exceeds the configured size.
    • Log rotation still works correctly when combined with buffered flush.
  • Benchmark on my laptop:
goos: darwin
goarch: arm64
pkg: github.com/pingcap/log
cpu: Apple M2 Pro
BenchmarkLogger/unbuffered-10 132064 9130 ns/op 368 B/op 5 allocs/op
BenchmarkLogger/buffered-4KB-10 182089 6272 ns/op 368 B/op 5 allocs/op
BenchmarkLogger/buffered-64KB-10 308062 3941 ns/op 369 B/op 5 allocs/op
BenchmarkLogger/buffered-256KB-10 291272 3868 ns/op 370 B/op 5 allocs/op
BenchmarkLogger/buffered-1MB-10 296786 3929 ns/op 372 B/op 5 allocs/op
PASS
ok github.com/pingcap/log	7.394s

@bb7133
bb7133 merged commit 19901e0 into pingcap:masterSep 17, 2025
2 checks passed
@bb7133
bb7133 deleted the bb7133/buffered branch September 17, 2025 02:11
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@bb7133@YangKeao@djshow832