Skip to content

feat(telemetry): implement telemetry for aggregator - #1077

Merged
uri-99 merged 56 commits into
stagingfrom
1075-feattelemetry-implement-telemetry-for-aggregator
Oct 10, 2024
Merged

feat(telemetry): implement telemetry for aggregator#1077
uri-99 merged 56 commits into
stagingfrom
1075-feattelemetry-implement-telemetry-for-aggregator

Conversation

@JuArce

@JuArceJuArce commented Sep 25, 2024•
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

This PR supersedes #949

The aggregator has to communicate to the telemetry_api using HTTP endpoints
The telemetry_api will handle traces creation

closes#1075

Tasks

  • Docker compose with Otel Collector and Jaeger UI
  • Init traces on NewTask
  • Add span on Operator responses
  • Add span on Quorum Reached
  • Add span on Tasks errors (ie task expired)
  • Close traces after a period of time the quorum was reached to wait for most operators responses
  • Show operator not responding

How to Test

Tip

You will need a lot of terminals 😎

  1. Run anvil
make anvil_start_with_block_time
  1. Run aggregator
make aggregator_start
  1. Run batcher
make batcher_start_local
  1. Build operator
make build_operator
  1. Register operator 1
export OPERATOR_ADDRESS=0x70997970C51812dc3A010C7d01b50e0d17dc79C8
make operator_full_registration CONFIG_FILE=config-files/config-operator-1.yaml
  1. Register operator 2
export OPERATOR_ADDRESS=0x3C44CdDdB6a900fa2b585dd299e03d12FA4293BC
make operator_full_registration CONFIG_FILE=config-files/config-operator-2.yaml
  1. Register operator 3
export OPERATOR_ADDRESS=0x90F79bf6EB2c4f870365E785982E1f101E93b906
make operator_full_registration CONFIG_FILE=config-files/config-operator-3.yaml
  1. Run Telemetry
make telemetry_create_env
make telemetry_run_db
make open_telemetry_start
make telemetry_start
  1. Run Operator 1
./operator/build/aligned-operator start --config ./config-files/config-operator-1.yaml
  1. Run Operator 2
./operator/build/aligned-operator start --config ./config-files/config-operator-2.yaml
  1. Run Opreator 3
./operator/build/aligned-operator start --config ./config-files/config-operator-3.yaml
  1. Send proofs
make batcher_send_burst_groth16
  1. Go to Jaeger UI and find the new traces

You should see traces like the following one

imageimage

@JuArceJuArce self-assigned this Sep 25, 2024
@JuArceJuArce changed the title feat: create and finish traces for specified merkle rootfeat(telemetry): implement telemetry for aggregatorSep 25, 2024
This was referenced Sep 30, 2024
Closed
Closed
@JuArceJuArce linked an issue Sep 30, 2024 that may be closed by this pull request
rewrite operator_tracker in elixir, add telemetry to it #998
Closed
avilagaston9and others added 13 commits October 1, 2024 13:41
@avilagaston9
feat(telemetry): modify the aggregator to use the elixir Tracker (#1105)
c796740
@JulianVenturaJulian Ventura
feat: Add operators data fetch at startup (#1108)
54eb9fb
Co-authored-by: Julian Ventura <julian.ventura@lambdaclass.com>
@JuArce
Merge branch 'refs/heads/998-refactortracker-rewrite-operator_tracker…
2b6d382
…-in-elixir' into 1075-feattelemetry-implement-telemetry-for-aggregator
# Conflicts:
#	telemetry_api/config/dev.exs
#	telemetry_api/mix.exs
#	telemetry_api/mix.lock
@JuArce
Merge branch 'refs/heads/staging' into 998-refactortracker-rewrite-op…
b99b9bf
…erator_tracker-in-elixir
@JuArce
Merge branch 'refs/heads/998-refactortracker-rewrite-operator_tracker…
335a448
…-in-elixir' into 1075-feattelemetry-implement-telemetry-for-aggregator
@JuArce
chore: upload mix lock
49b9646
@JuArce
fix: parse operator id correctly
f60ebf1
fix: store address and operator_id in lowercase
@JuArce
Merge branch 'refs/heads/998-refactortracker-rewrite-operator_tracker…
948565c
…-in-elixir' into 1075-feattelemetry-implement-telemetry-for-aggregator
@JuArce
fix: encode address to lowercase
57e0776
@JuArce
nit
8da05e6
@JuArce
feat(telemetry): add operators details on OperatorResponse event (#1135)
6b71469
@avilagaston9
feat(telemetry): add event showing missing operators (#1145)
4b15e43
@JuArce
nit
fc00cc5
@JuArce
chore: disable dashboard
4a1578a
@JuArce
fix: telemetry dump db
aa43de0
@JuArce
Merge branch 'refs/heads/998-refactortracker-rewrite-operator_tracker…
6321a6a
…-in-elixir' into 1075-feattelemetry-implement-telemetry-for-aggregator
# Conflicts:
#	telemetry_api/lib/telemetry_api_web/controllers/operator_controller.ex
@JuArce
JuArce marked this pull request as ready for review October 4, 2024 14:27
Oppen
Oppen reviewed Oct 4, 2024
View reviewed changes
Comment threadaggregator/internal/pkg/telemetry.go Outdated
Comment on lines +34 to +35
client *http.Client
baseURL *url.URL

@OppenOppenOct 4, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these need to be pointers?

@avilagaston9avilagaston9Oct 9, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They didn't. Changed them!

Oppen
Oppen reviewed Oct 4, 2024
View reviewed changes
Comment threadtelemetry_api/lib/telemetry_api/periodic/operator_fetcher.ex Outdated
Base automatically changed from 998-refactortracker-rewrite-operator_tracker-in-elixir to stagingOctober 4, 2024 18:34
@avilagaston9avilagaston9 mentioned this pull request Oct 7, 2024
Merged
uri-99
uri-99 suggested changes Oct 8, 2024
View reviewed changes
Comment threadaggregator/internal/pkg/aggregator.go
Comment threadtelemetry_api/config/dev.exs
Comment threadtelemetry_api/lib/telemetry_api/operators.ex Outdated
Comment threadtelemetry_api/lib/telemetry_api/periodic/operator_fetcher.ex Outdated
Comment threadtelemetry_api/lib/telemetry_api/traces.ex Outdated
@uri-99
uri-99 merged commit 18f64e9 into stagingOct 10, 2024
@uri-99
uri-99 deleted the 1075-feattelemetry-implement-telemetry-for-aggregator branch October 10, 2024 16:12
@JulianVenturaJulianVentura mentioned this pull request Oct 14, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

4 more reviewers
@OppenOppenOppen left review comments
@avilagaston9avilagaston9avilagaston9 left review comments
@uri-99uri-99uri-99 approved these changes
@entropidelicentropidelicentropidelic approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@JulianVenturaJulianVentura

@JuArceJuArce

@avilagaston9avilagaston9

Labels

None yet

Projects

AlignedLayed Roadmap
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

rewrite operator_tracker in elixir, add telemetry to it

6 participants

@JuArce@Oppen@uri-99@entropidelic@avilagaston9@JulianVentura