fix(aws cloudwatch, aws s3): emit component_sent_bytes_total through Driver to ensure component labels by Clee2691 · Pull Request #285 · ViaQ/vector

Clee2691 · 2026-07-16T17:38:09Z

Description

This is a forward port of #284 that also includes the fix for S3 for v0.54.0-rh branch.

This is a bug for all AWS sinks.

/cc @vparfonov
/assign @jcantrill

Links

Related: #284
JIRA: https://redhat.atlassian.net/browse/LOG-9671

coderabbitai · 2026-07-16T17:38:20Z

📝 Walkthrough

Summary by CodeRabbit

New Features
- AWS sink metrics now include the resolved region alongside component labels.
- Added support for attaching custom labels to emitted byte-count metrics.
- Byte-count metrics now reflect actual request payload sizes across AWS S3, Kinesis, CloudWatch Logs, and SNS/SQS workflows.
Bug Fixes
- Improved reporting for CloudWatch Logs uploads that partially succeed before failing.
- Corrected CloudWatch Logs request size calculations, including batch overhead.
- Added validation for missing recorded events.

Walkthrough

AWS sinks now propagate resolved regions into request labels, emit component bytes-sent metrics, and avoid duplicate transport metrics. CloudWatch adds partial-upload byte accounting, while Kinesis and S3 responses report request byte sizes.

Changes

Shared metrics and AWS transport

Layer / File(s)	Summary
Shared labels and transport metrics `lib/vector-common/src/internal_event/bytes_sent.rs`, `lib/vector-stream/src/driver.rs`, `src/aws/mod.rs`, `lib/vector-common/src/event_test_util.rs`	`BytesSent` and `Driver` support extra labels, AWS clients can suppress transport metrics, and missing-event handling gains coverage.

CloudWatch partial upload metrics

Layer / File(s)	Summary
CloudWatch upload accounting and labeling `src/sinks/aws_cloudwatch_logs/*`	CloudWatch tracks batch and request bytes, reports partial upload failures, propagates the resolved region, and validates component metric labels.

Kinesis region and byte accounting

Layer / File(s)	Summary
Kinesis region propagation and responses `src/sinks/aws_kinesis/*`	Firehose and Streams use metric-free client construction, pass resolved regions into sinks, and report encoded request bytes.

S3 region and byte accounting

Layer / File(s)	Summary
S3 region propagation and request bytes `src/sinks/aws_s3/`, `src/sinks/s3_common/`	S3 service creation returns the resolved region, sinks attach it to HTTPS requests, responses report body bytes, and integration tests validate component labels.

SNS and SQS region labels

Layer / File(s)	Summary
SNS/SQS shared sink labeling `src/sinks/aws_s_s/*`	SNS and SQS propagate resolved regions into the shared sink, which emits HTTPS requests with region labels and uses encoded request sizes.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
participant Sink
participant Driver
participant AWSService
participant BytesSent
Sink->>Driver: configure HTTPS protocol and region label
Driver->>AWSService: send request
AWSService-->>Driver: return request byte size
Driver->>BytesSent: emit component bytes-sent metric

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.54% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is specific and matches the main change, though it names only some sinks while the fix spans more AWS sinks.
Description check	✅ Passed	The description is clearly related to the changeset and accurately frames the forward-port and AWS sink fix.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Clee2691 · 2026-07-17T15:30:32Z

/hold

Clee2691 · 2026-07-17T22:34:22Z

/hold cancel

Clee2691 · 2026-07-20T13:49:56Z

/hold

Clee2691 · 2026-07-21T18:39:00Z

/hold cancel

Clee2691 · 2026-07-22T14:41:25Z

/retest

Clee2691 · 2026-07-24T14:13:03Z

/hold

jcantrill · 2026-07-27T16:05:00Z

/approve
/lgtm

openshift-ci · 2026-07-27T16:05:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Clee2691, jcantrill

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jcantrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/sinks/aws_cloudwatch_logs/service.rs`:
- Around line 250-267: Prevent outer retries from replaying requests that
produce CloudwatchError::PutPartial, so the bytes_sent_handle emission in the
svc oneshot error path cannot be followed by full-request accounting on replay.
Update CloudwatchRetryLogic to classify PutPartial as non-retryable while
preserving existing retry behavior for other errors.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 69f88144-bf47-4663-a95f-ddc9b4f5e713

📥 Commits

Reviewing files that changed from the base of the PR and between ca95fca and 7e70e0b.

📒 Files selected for processing (31)

lib/vector-common/src/event_test_util.rs
lib/vector-common/src/internal_event/bytes_sent.rs
lib/vector-stream/src/driver.rs
src/aws/mod.rs
src/sinks/aws_cloudwatch_logs/config.rs
src/sinks/aws_cloudwatch_logs/integration_tests.rs
src/sinks/aws_cloudwatch_logs/request.rs
src/sinks/aws_cloudwatch_logs/request_builder.rs
src/sinks/aws_cloudwatch_logs/retry.rs
src/sinks/aws_cloudwatch_logs/service.rs
src/sinks/aws_cloudwatch_logs/sink.rs
src/sinks/aws_kinesis/config.rs
src/sinks/aws_kinesis/firehose/config.rs
src/sinks/aws_kinesis/firehose/integration_tests.rs
src/sinks/aws_kinesis/firehose/record.rs
src/sinks/aws_kinesis/service.rs
src/sinks/aws_kinesis/sink.rs
src/sinks/aws_kinesis/streams/config.rs
src/sinks/aws_kinesis/streams/integration_tests.rs
src/sinks/aws_kinesis/streams/record.rs
src/sinks/aws_s3/config.rs
src/sinks/aws_s3/integration_tests.rs
src/sinks/aws_s_s/service.rs
src/sinks/aws_s_s/sink.rs
src/sinks/aws_s_s/sns/config.rs
src/sinks/aws_s_s/sns/integration_tests.rs
src/sinks/aws_s_s/sqs/config.rs
src/sinks/aws_s_s/sqs/integration_tests.rs
src/sinks/s3_common/config.rs
src/sinks/s3_common/service.rs
src/sinks/s3_common/sink.rs

…Driver to ensure component labels

Clee2691 · 2026-07-27T20:55:56Z

@coderabbitai review

coderabbitai · 2026-07-27T20:56:01Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/vector-common/src/internal_event/bytes_sent.rs`:
- Around line 9-16: Update the CloudWatch bytes-sent instrumentation around
CloudwatchResponse::bytes_sent, the service-level BytesSent registration, and
Driver emission so each successful request records component_sent_bytes_total
exactly once, preferably through Driver with component labels. Remove or exclude
the overlapping service-side emission, preserve the intended labels, and add a
regression assertion preventing duplicate or unlabeled series.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 16b6dc9e-d0bc-43ae-9cad-c29ce41375c6

📥 Commits

Reviewing files that changed from the base of the PR and between 7e70e0b and dc3e599.

📒 Files selected for processing (30)

lib/vector-common/src/event_test_util.rs
lib/vector-common/src/internal_event/bytes_sent.rs
lib/vector-stream/src/driver.rs
src/aws/mod.rs
src/sinks/aws_cloudwatch_logs/config.rs
src/sinks/aws_cloudwatch_logs/integration_tests.rs
src/sinks/aws_cloudwatch_logs/request.rs
src/sinks/aws_cloudwatch_logs/request_builder.rs
src/sinks/aws_cloudwatch_logs/service.rs
src/sinks/aws_cloudwatch_logs/sink.rs
src/sinks/aws_kinesis/config.rs
src/sinks/aws_kinesis/firehose/config.rs
src/sinks/aws_kinesis/firehose/integration_tests.rs
src/sinks/aws_kinesis/firehose/record.rs
src/sinks/aws_kinesis/service.rs
src/sinks/aws_kinesis/sink.rs
src/sinks/aws_kinesis/streams/config.rs
src/sinks/aws_kinesis/streams/integration_tests.rs
src/sinks/aws_kinesis/streams/record.rs
src/sinks/aws_s3/config.rs
src/sinks/aws_s3/integration_tests.rs
src/sinks/aws_s_s/service.rs
src/sinks/aws_s_s/sink.rs
src/sinks/aws_s_s/sns/config.rs
src/sinks/aws_s_s/sns/integration_tests.rs
src/sinks/aws_s_s/sqs/config.rs
src/sinks/aws_s_s/sqs/integration_tests.rs
src/sinks/s3_common/config.rs
src/sinks/s3_common/service.rs
src/sinks/s3_common/sink.rs

🚧 Files skipped from review as they are similar to previous changes (25)

src/sinks/aws_s_s/sink.rs
src/sinks/aws_cloudwatch_logs/sink.rs
src/sinks/s3_common/sink.rs
src/sinks/aws_kinesis/sink.rs
src/sinks/aws_kinesis/firehose/integration_tests.rs
src/sinks/s3_common/config.rs
src/sinks/aws_kinesis/streams/record.rs
src/sinks/aws_s_s/service.rs
src/sinks/aws_kinesis/streams/integration_tests.rs
src/sinks/aws_s_s/sqs/config.rs
src/sinks/aws_kinesis/firehose/record.rs
src/sinks/aws_s_s/sqs/integration_tests.rs
lib/vector-stream/src/driver.rs
src/sinks/aws_s_s/sns/config.rs
src/sinks/s3_common/service.rs
src/sinks/aws_kinesis/service.rs
src/sinks/aws_s3/config.rs
lib/vector-common/src/event_test_util.rs
src/sinks/aws_kinesis/streams/config.rs
src/sinks/aws_cloudwatch_logs/config.rs
src/sinks/aws_s3/integration_tests.rs
src/sinks/aws_cloudwatch_logs/request.rs
src/sinks/aws_cloudwatch_logs/integration_tests.rs
src/aws/mod.rs
src/sinks/aws_cloudwatch_logs/service.rs

Clee2691 · 2026-07-28T03:56:36Z

/retest

jcantrill · 2026-07-28T14:27:26Z

/lgtm

Clee2691 · 2026-07-29T15:38:55Z

/hold cancel

openshift-ciBot requested review from jcantrill and vparfonov July 16, 2026 17:38

Clee2691 assigned jcantrill Jul 16, 2026

Clee2691 force-pushed the LOG-7893-0.54.0 branch from 3f4e627 to 61eb96bCompare July 16, 2026 22:04

openshift-ciBot added the do-not-merge/hold label Jul 17, 2026

openshift-ciBot removed the do-not-merge/hold label Jul 17, 2026

openshift-ciBot added the do-not-merge/hold label Jul 20, 2026

Clee2691 force-pushed the LOG-7893-0.54.0 branch 2 times, most recently from fdb9f35 to 0007855Compare July 20, 2026 19:44

openshift-ciBot removed the do-not-merge/hold label Jul 21, 2026

Clee2691 force-pushed the LOG-7893-0.54.0 branch from 0007855 to 83769a5Compare July 22, 2026 00:08

openshift-ciBot added the do-not-merge/hold label Jul 24, 2026

openshift-ciBot added the lgtm label Jul 27, 2026

openshift-ciBot added the approved label Jul 27, 2026

Clee2691 force-pushed the LOG-7893-0.54.0 branch from 83769a5 to 7e70e0bCompare July 27, 2026 18:20

openshift-ciBot removed the lgtm label Jul 27, 2026

coderabbitaiBot reviewed Jul 27, 2026
View reviewed changes

Comment threadsrc/sinks/aws_cloudwatch_logs/service.rs

fix(aws cloudwatch, aws s3): emit component_sent_bytes_total through …
dc3e599
…Driver to ensure component labels

Clee2691 force-pushed the LOG-7893-0.54.0 branch from 7e70e0b to dc3e599Compare July 27, 2026 20:55

coderabbitaiBot reviewed Jul 27, 2026
View reviewed changes

Comment threadlib/vector-common/src/internal_event/bytes_sent.rs

openshift-ciBot added the lgtm label Jul 28, 2026

openshift-ciBot removed the do-not-merge/hold label Jul 29, 2026

openshift-merge-botBot merged commit 8716349 into ViaQ:v0.54.0-rhJul 29, 2026
10 checks passed

Uh oh!

Conversation

Clee2691 commented Jul 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Links

Uh oh!

coderabbitaiBot commented Jul 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

Clee2691 commented Jul 17, 2026

Uh oh!

Clee2691 commented Jul 17, 2026

Uh oh!

Clee2691 commented Jul 20, 2026

Uh oh!

Clee2691 commented Jul 21, 2026

Uh oh!

Clee2691 commented Jul 22, 2026

Uh oh!

Clee2691 commented Jul 24, 2026

Uh oh!

jcantrill commented Jul 27, 2026

Uh oh!

openshift-ciBot commented Jul 27, 2026

Uh oh!

coderabbitaiBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Clee2691 commented Jul 27, 2026

Uh oh!

coderabbitaiBot commented Jul 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitaiBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Clee2691 commented Jul 28, 2026

Uh oh!

jcantrill commented Jul 28, 2026

Uh oh!

Clee2691 commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Clee2691 commented Jul 16, 2026•
edited
Loading

coderabbitaiBot commented Jul 16, 2026•
edited
Loading

coderabbitaiBot commented Jul 27, 2026•
edited
Loading