Skip to content

[INF-122] Track metrics for dp4#3228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joaquincasares merged 7 commits into from
Jun 13, 2022
Merged

[INF-122] Track metrics for dp4 #3228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7147b28
ingest dp4 prometheus metrics
joaquincasares Jun 10, 2022
ab867dc
add deploy script
joaquincasares Jun 10, 2022
51d7454
update README with deploy guide
joaquincasares Jun 10, 2022
1775acf
don't auto-checkout master
joaquincasares Jun 10, 2022
74e7787
indent
joaquincasares Jun 10, 2022
914a0ff
update monitoring dir README (#3229)
SidSethi Jun 11, 2022
8b6e8d1
update README to differentiate between local and prod
joaquincasares Jun 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions monitoring/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,23 @@

### Launch Grafana
### Launch Prometheus & Grafana Locally

```bash
docker-compose up -d grafana prometheus
docker-compose build
docker-compose up -d prometheus grafana
./grafana/bin/create-data-sources.sh
./grafana/bin/upload-dashboards.sh
```

Visit `http://${REMOTE_DEV_HOST}:3000`.
Ports:

* Prometheus binds to :9090
* Grafana binds to :3000

### Deploy Production Changes

```bash
ssh prometheus-grafana-metrics
cd ~/audius-protocol/monitoring
git checkout master

@stereostevestereosteveJun 10, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should do a git pull origin master here?

@joaquincasaresjoaquincasaresJun 10, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

The git pull is already happening with the deploy script: https://github.com/AudiusProject/audius-protocol/pull/3228/files#diff-ae5a05e31eca7c4be5060db4c336f48c4de53189fc8e71c4358fd861f957161fR5.

That way it always gets changes, no matter what branch I'm deploying from.


./deploy.sh
```
9 changes: 9 additions & 0 deletions monitoring/deploy.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash

set -ex

git pull
docker-compose build

docker-compose down
docker-compose up -d
20 changes: 10 additions & 10 deletions monitoring/docker-compose.yml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,16 @@
version: '3'

services:
prometheus:
build: prometheus
user: 0:0
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "9090:9090"
volumes:
- ./data/prometheus:/prometheus

grafana:
build:
context: grafana
Expand All@@ -17,13 +27,3 @@ services:
restart: always
volumes:
- ./data/grafana:/var/lib/grafana

prometheus:
build: prometheus
user: 0:0
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "9090:9090"
volumes:
- ./data/prometheus:/prometheus
15 changes: 15 additions & 0 deletions monitoring/prometheus/generate-prom.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
// this script generates prometheus.yml dynamically
// it uses audiusLibs to add known service providers to prometheus.yml

const fs = require('fs');
const dotenv = require('dotenv');
Expand DownExpand Up@@ -86,6 +88,19 @@ scrape_configs:
service: 'audius'
component: 'discover-provider'
job: 'census'

# monitor canary nodes

- job_name: 'discoveryprovider4-audius-co'
scheme: https
metrics_path: '/prometheus_metrics'
static_configs:
- targets: ['discoveryprovider4.audius.co']
labels:
host: 'discoveryprovider4.audius.co'
environment: 'prod'
service: 'audius'
component: 'discover-provider'
`)

for (const env of ENVS) {
Expand Down