Enable metric collection for Async Instruments - Delta and Cumulative by lalitb · Pull Request #1334 · open-telemetry/opentelemetry-cpp

lalitb · 2022-04-18T19:55:56Z

Changes

AsyncMetricStorage invokes the instrumented observation callback and waits for it to be done. After the observation completes, it resets the active storage stash and builds metric data for the metric instrument.
Also created TemporalMetricStorage inspired from otel-java to store the history of metrics. This TemporalMetricStorage would be later re-used for synchronous storage too.

@ThomsonTan @esigo - Updated the description, so should be good to review now. Thanks

For significant contributions please make sure you have completed the following items:

CHANGELOG.md updated for non-trivial changes
Unit tests have been added
Changes in public API reviewed

codecov · 2022-04-18T20:03:37Z

Codecov Report

Merging #1334 (aafa07d) into main (13ac5c3) will decrease coverage by 0.26%.
The diff coverage is 56.89%.

@@            Coverage Diff             @@
##             main    #1334      +/-   ##
==========================================
- Coverage   85.96%   85.70%   -0.25%     
==========================================
  Files         149      151       +2     
  Lines        4250     4355     +105     
==========================================
+ Hits         3653     3732      +79     
- Misses        597      623      +26

Impacted Files	Coverage Δ
...lemetry/sdk/metrics/aggregation/drop_aggregation.h	`20.00% <0.00%> (ø)`
...ry/sdk/metrics/aggregation/histogram_aggregation.h	`86.67% <ø> (ø)`
...ry/sdk/metrics/aggregation/lastvalue_aggregation.h	`0.00% <ø> (ø)`
...elemetry/sdk/metrics/aggregation/sum_aggregation.h	`0.00% <ø> (ø)`
...k/src/metrics/aggregation/histogram_aggregation.cc	`83.79% <0.00%> (-7.39%)`	⬇️
...k/src/metrics/aggregation/lastvalue_aggregation.cc	`35.49% <0.00%> (-3.80%)`	⬇️
...etry/sdk/metrics/aggregation/default_aggregation.h	`33.97% <31.58%> (-1.33%)`	⬇️
sdk/src/metrics/aggregation/sum_aggregation.cc	`65.46% <50.00%> (-0.58%)`	⬇️
sdk/src/metrics/state/temporal_metric_storage.cc	`73.22% <73.22%> (ø)`
...telemetry/sdk/metrics/state/async_metric_storage.h	`83.34% <77.78%> (-9.52%)`	⬇️
... and 3 more

… async-impl

esigo

Thanks for the PR.
I had a few minor comments.

ThomsonTan · 2022-05-02T20:31:25Z

+    switch (aggregation_type)
+    {
+      case AggregationType::kDrop:
+        return std::unique_ptr<Aggregation>(new DropAggregation());


Probably make assignment of the new Aggregation in the switch and return it at the end of the function? return on each case could cause multiple copy of epilog being generated.

you mean multiple stack unwinding code being generated. I think modern compilers should be smart enough to handle that.

ThomsonTan · 2022-05-02T21:15:03Z


  /* Returns the result of merge of the existing aggregation with delta aggregation with same
   * boundaries */
-  virtual std::unique_ptr<Aggregation> Merge(const Aggregation &delta) const noexcept override;


Why is virtual removed?

It is not required, as we won't need to subclass HistogramAggregation and other aggregations further.

esigo

LGTM
Thanks for the PR :)

Co-authored-by: Ehsan Saei <71217171+esigo@users.noreply.github.com>

* install sdk config (#1273) * Bump actions/cache from 2 to 3 (#1277) * Add owent as an Approver (#1276) * add owent as reviewer * fix order * Disable benchmark action failure (#1284) * metrics exemplar round 1 (#1264) * [Metrics SDK] - fix spelling (AggregationTemporarily to AggregationTemporality) (#1288) * fix compilation error with protobuf 3.5 (#1289) * Fix span SetAttribute crash (#1283) * Synchronous Metric collection (Delta , Cumulative) (#1265) * Rename `http_client_curl` to `opentelemetry_http_client_curl` (#1301) Signed-off-by: owent <admin@owent.net> * Don't show coverage annotation for pull requests (#1304) * Implement periodic exporting metric reader (#1286) * Add `async-changes` branch to pull_request of github action (#1309) Signed-off-by: owent <admin@owent.net> * Add InstrumentationInfo and Resource to the metrics data to be exported. (#1299) * Excempt should be applied on issue instead of PR (#1316) * Bump codecov/codecov-action from 2.1.0 to 3 (#1318) * Move public definitions into `opentelemetry_api`. (#1314) Signed-off-by: owent <admin@owent.net> * Add building test without RTTI (#1294) * Remove implicitly deleted default constructor (#1267) Co-authored-by: Tom Tan <Tom.Tan@microsoft.com> Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * [ETW Exporter] - ETW provider handle cleanup (#1322) * Bump actions/stale from 4 to 5 (#1323) * ostream metrics example (#1312) * Prepare v1.3.0 release (#1324) * Update yield logic for ARM processor (#1325) * Fix for #1292 (#1326) * Implement Merge and Diff operation for Histogram Aggregation (#1303) * fix metrics compiler warnings (#1328) * Replace deprecated googletest API (#1327) * Remove redundant tail / in CMake install (#1329) * dependencies image as artifact (#1333) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * metrics histogram example (#1330) * Link `opentelemetry_ext` with `opentelemetry_api` (#1336) * ostream metrics cmake (#1344) * prometheus exporter (#1331) * remove exporter registration to meter provider (#1350) * Bump github/codeql-action from 1 to 2 (#1351) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add explicit type cast in baggage UrlDecode (#1353) * Fix scalar delete against array (#1356) * conditional include for codecvt header (#1355) * Add missing include guard (#1357) * Use latest TraceLoggingDynamic.h (#1354) * prometheus example (#1332) * Fix output time in metrics OStream exporter (#1346) * Simplify SDK Configuration: Use View with default aggregation if no matching View is configured (#1358) * Fix class member initialization order (#1360) * codecov ignore (#1364) * export opentelemetry_otlp_recordable (#1365) * Disable test on prometheus-cpp which not need (#1363) * Enable metric collection for Async Instruments - Delta and Cumulative (#1334) * fix baggage propagation for empty/invalid baggage context (#1367) * Fix empty tracestate header propagation (#1373) * Bump docker/setup-qemu-action from 1 to 2 (#1375) * Add noexcept/const qualifier at missing places for Trace API. (#1374) * fix noxcept * fix etw * Bump docker/build-push-action from 2 to 3 (#1377) * Bump docker/setup-buildx-action from 1 to 2 (#1376) * [Metrics SDK] Remove un-necessary files. (#1379) * reuse temporal metric storage for sync storage (#1369) * Prometheus exporter meters and instrument name (#1378) * Fix sharing resource in batched exported spans (#1386) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * fix: missing link to nlohmann_json (#1390) * Getting started document using ostream exporter (#1394) Co-authored-by: Reiley Yang <reyang@microsoft.com> * Connect async storage with async instruments (#1388) * get span_id from context when Logger::Log received invalid span_id (#1398) * Alpine image (#1382) * Upgrade proto to v0.17.0, update log data model (#1383) * Prepare v1.4.0 release (#1404) * Fix vcpkg package name in doc (#1392) * Document Getting Started with Prometheus and Grafana (#1396) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * fix OTEL_INTERNAL_LOG_INFO (#1407) * fix: WaitOnSocket select error when sockfd above FD_SETSIZE (#1410) * [BUILD] fix nlohmann_json's (third party) include dir (#1415) * [Metrics API/SDK] - Pass state to async callback function. (#1408) * Copy string_view passed to ETW exporter in PropertyVariant (#1425) * Fix ETW log exporter header inclusion (#1426) * [Metrics SDK] Only record non-negative / finite / Non-NAN histogram values (#1427) * validate histogram value * handle Nan * add changelog * divide by 0 error on windows * fix markdown lint * Fix global log handle symbols when using dlopen (#1420) * Add attributes/dimensions to metrics ostream exporter (#1400) * Log current timestamp instead of epoch time (#1434) * install sdk-config.h (#1419) * Fix GettingStarted documentation for Jaeger HTTP exporter (#1347) (#1439) * fix histogram (#1440) * Upgrade nlohmann_json to 3.10.5 (#1438) (#1441) * Fixed broken link to OpenTelemetry.io (#1445) (#1446) Co-authored-by: Ehsan Saei <71217171+esigo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> Co-authored-by: Tom Tan <Tom.Tan@microsoft.com> Co-authored-by: Ben Landrum <71329856+benlandrum@users.noreply.github.com> Co-authored-by: jmanjon <67091862+juandemanjon@users.noreply.github.com> Co-authored-by: Zsolt Bölöny <bolony.zsolt@gmail.com> Co-authored-by: Leo Di Donato <leodidonato@gmail.com> Co-authored-by: Reiley Yang <reyang@microsoft.com> Co-authored-by: Lalit Kumar Bhasin <labhas@microsoft.com> Co-authored-by: Hamed Mansouri <hamed0381@gmail.com> Co-authored-by: ztao <t@taozj.org> Co-authored-by: Flier Lu <flier@users.noreply.github.com> Co-authored-by: Marc Alff <marc.alff@free.fr> Co-authored-by: Marc Alff <marc.alff@oracle.com>

* install sdk config (#1273) * Bump actions/cache from 2 to 3 (#1277) * Add owent as an Approver (#1276) * add owent as reviewer * fix order * Disable benchmark action failure (#1284) * metrics exemplar round 1 (#1264) * [Metrics SDK] - fix spelling (AggregationTemporarily to AggregationTemporality) (#1288) * fix compilation error with protobuf 3.5 (#1289) * Fix span SetAttribute crash (#1283) * Synchronous Metric collection (Delta , Cumulative) (#1265) * Rename `http_client_curl` to `opentelemetry_http_client_curl` (#1301) Signed-off-by: owent <admin@owent.net> * Don't show coverage annotation for pull requests (#1304) * Implement periodic exporting metric reader (#1286) * Add `async-changes` branch to pull_request of github action (#1309) Signed-off-by: owent <admin@owent.net> * Add InstrumentationInfo and Resource to the metrics data to be exported. (#1299) * Excempt should be applied on issue instead of PR (#1316) * Bump codecov/codecov-action from 2.1.0 to 3 (#1318) * Move public definitions into `opentelemetry_api`. (#1314) Signed-off-by: owent <admin@owent.net> * Add building test without RTTI (#1294) * Remove implicitly deleted default constructor (#1267) Co-authored-by: Tom Tan <Tom.Tan@microsoft.com> Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * [ETW Exporter] - ETW provider handle cleanup (#1322) * Bump actions/stale from 4 to 5 (#1323) * ostream metrics example (#1312) * Prepare v1.3.0 release (#1324) * Update yield logic for ARM processor (#1325) * Fix for #1292 (#1326) * Implement Merge and Diff operation for Histogram Aggregation (#1303) * fix metrics compiler warnings (#1328) * Replace deprecated googletest API (#1327) * Remove redundant tail / in CMake install (#1329) * dependencies image as artifact (#1333) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * metrics histogram example (#1330) * Link `opentelemetry_ext` with `opentelemetry_api` (#1336) * ostream metrics cmake (#1344) * prometheus exporter (#1331) * remove exporter registration to meter provider (#1350) * Bump github/codeql-action from 1 to 2 (#1351) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add explicit type cast in baggage UrlDecode (#1353) * Fix scalar delete against array (#1356) * conditional include for codecvt header (#1355) * Add missing include guard (#1357) * Use latest TraceLoggingDynamic.h (#1354) * prometheus example (#1332) * Fix output time in metrics OStream exporter (#1346) * Simplify SDK Configuration: Use View with default aggregation if no matching View is configured (#1358) * Fix class member initialization order (#1360) * codecov ignore (#1364) * export opentelemetry_otlp_recordable (#1365) * Disable test on prometheus-cpp which not need (#1363) * Enable metric collection for Async Instruments - Delta and Cumulative (#1334) * fix baggage propagation for empty/invalid baggage context (#1367) * Fix empty tracestate header propagation (#1373) * Bump docker/setup-qemu-action from 1 to 2 (#1375) * Add noexcept/const qualifier at missing places for Trace API. (#1374) * fix noxcept * fix etw * Bump docker/build-push-action from 2 to 3 (#1377) * Bump docker/setup-buildx-action from 1 to 2 (#1376) * [Metrics SDK] Remove un-necessary files. (#1379) * reuse temporal metric storage for sync storage (#1369) * Prometheus exporter meters and instrument name (#1378) * Fix sharing resource in batched exported spans (#1386) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * fix: missing link to nlohmann_json (#1390) * Getting started document using ostream exporter (#1394) Co-authored-by: Reiley Yang <reyang@microsoft.com> * Connect async storage with async instruments (#1388) * get span_id from context when Logger::Log received invalid span_id (#1398) * Alpine image (#1382) * Upgrade proto to v0.17.0, update log data model (#1383) * Prepare v1.4.0 release (#1404) * Fix vcpkg package name in doc (#1392) * Document Getting Started with Prometheus and Grafana (#1396) Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> * fix OTEL_INTERNAL_LOG_INFO (#1407) * fix: WaitOnSocket select error when sockfd above FD_SETSIZE (#1410) * [BUILD] fix nlohmann_json's (third party) include dir (#1415) * [Metrics API/SDK] - Pass state to async callback function. (#1408) * Copy string_view passed to ETW exporter in PropertyVariant (#1425) * Fix ETW log exporter header inclusion (#1426) * [Metrics SDK] Only record non-negative / finite / Non-NAN histogram values (#1427) * validate histogram value * handle Nan * add changelog * divide by 0 error on windows * fix markdown lint * Fix global log handle symbols when using dlopen (#1420) * Add attributes/dimensions to metrics ostream exporter (#1400) * Log current timestamp instead of epoch time (#1434) * install sdk-config.h (#1419) * Fix GettingStarted documentation for Jaeger HTTP exporter (#1347) (#1439) * fix histogram (#1440) * Upgrade nlohmann_json to 3.10.5 (#1438) (#1441) * Fixed broken link to OpenTelemetry.io (#1445) (#1446) * Fix variables inizialization (#1430) * Remove `AsyncBatch*Processor`, implement [opentelemetry-specification#2452](open-telemetry/opentelemetry-specification#2452) Signed-off-by: WenTao Ou <admin@owent.net> * Remove invalid changelog Signed-off-by: WenTao Ou <admin@owent.net> * Fix compiling problems without `ENABLE_ASYNC_EXPORT` Signed-off-by: WenTao Ou <admin@owent.net> * Always return `ExportResult::kSuccess` Signed-off-by: WenTao Ou <admin@owent.net> Co-authored-by: Ehsan Saei <71217171+esigo@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Lalit Kumar Bhasin <lalit_fin@yahoo.com> Co-authored-by: Tom Tan <Tom.Tan@microsoft.com> Co-authored-by: Ben Landrum <71329856+benlandrum@users.noreply.github.com> Co-authored-by: jmanjon <67091862+juandemanjon@users.noreply.github.com> Co-authored-by: Zsolt Bölöny <bolony.zsolt@gmail.com> Co-authored-by: Leo Di Donato <leodidonato@gmail.com> Co-authored-by: Reiley Yang <reyang@microsoft.com> Co-authored-by: Lalit Kumar Bhasin <labhas@microsoft.com> Co-authored-by: Hamed Mansouri <hamed0381@gmail.com> Co-authored-by: ztao <t@taozj.org> Co-authored-by: Flier Lu <flier@users.noreply.github.com> Co-authored-by: Marc Alff <marc.alff@free.fr> Co-authored-by: Marc Alff <marc.alff@oracle.com> Co-authored-by: univisionsrl <31103417+univisionsrl@users.noreply.github.com>

initial commit -async storage

0b6f8fb

lalitb requested a review from a team April 18, 2022 19:55

Merge branch 'main' into async-impl

0767f90

lalitb marked this pull request as draft April 18, 2022 19:56

lalitb changed the title ~~Enable metric collection for Async Instruments - Delta and Cumulative~~ [WIP] Enable metric collection for Async Instruments - Delta and Cumulative Apr 18, 2022

lalitb added 2 commits April 19, 2022 20:13

add tests

9c4740d

Merge branch 'async-impl' of github.com:lalitb/opentelemetry-cpp into…

7198a0b

… async-impl

lalitb mentioned this pull request Apr 19, 2022

[Metrics API/SDK] Support registering multiple async callbacks to one single async instrument. #1338

Closed

lalitb changed the title ~~[WIP] Enable metric collection for Async Instruments - Delta and Cumulative~~ Enable metric collection for Async Instruments - Delta and Cumulative Apr 19, 2022

lalitb marked this pull request as ready for review April 19, 2022 15:44

lalitb added 2 commits April 19, 2022 23:50

fix format

b93d4dd

remove unwanted code

3a6f8eb

ThomsonTan assigned esigo Apr 25, 2022

esigo reviewed Apr 30, 2022

View reviewed changes

review comments

2dda99a

ThomsonTan reviewed May 2, 2022

View reviewed changes

fix review comments

77d344a

ThomsonTan reviewed May 2, 2022

View reviewed changes

Merge branch 'main' into async-impl

9dcaf1a

esigo reviewed May 3, 2022

View reviewed changes

Comment thread sdk/src/metrics/state/temporal_metric_storage.cc Outdated

Comment thread sdk/include/opentelemetry/sdk/metrics/state/temporal_metric_storage.h

esigo approved these changes May 3, 2022

View reviewed changes

Update sdk/src/metrics/state/temporal_metric_storage.cc

aafa07d

Co-authored-by: Ehsan Saei <71217171+esigo@users.noreply.github.com>

lalitb merged commit 9c734b3 into open-telemetry:main May 3, 2022

This was referenced May 4, 2022

Reuse TemporalMetricStorage for both SyncMetricStorage and AsyncMetricStorage #1369

Merged

[Metric SDK] Delta metric collection for Asynchronous Aggregation storage #1234

Closed

Navigation Menu

Uh oh!

Enable metric collection for Async Instruments - Delta and Cumulative - #1334

Enable metric collection for Async Instruments - Delta and Cumulative #1334
lalitb merged 10 commits into
open-telemetry:mainfrom
lalitb:async-impl

lalitb commented Apr 18, 2022 •

edited

Loading

Uh oh!

codecov Bot commented Apr 18, 2022 •

edited

Loading

Uh oh!

esigo left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ThomsonTan May 2, 2022

Uh oh!

lalitb May 2, 2022

Uh oh!

ThomsonTan May 2, 2022

Uh oh!

lalitb May 2, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

esigo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

lalitb commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

codecov Bot commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

esigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ThomsonTan May 2, 2022

Choose a reason for hiding this comment

Uh oh!

lalitb May 2, 2022

Choose a reason for hiding this comment

Uh oh!

ThomsonTan May 2, 2022

Choose a reason for hiding this comment

Uh oh!

lalitb May 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

esigo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lalitb commented Apr 18, 2022 •

edited

Loading

codecov Bot commented Apr 18, 2022 •

edited

Loading

lalitb May 2, 2022 •

edited

Loading