inventor

Prometheus HTTP SD implementation

Description

The Inventor is a Prometheus HTTP SD Server allows users to dynamically add or remove prometheus targets and labels and expose it to a Prometheus HTTP SD job.

Usage

Running the server:

cd src
go run main.go

Installing with Helm

helm repo add inventor https://code-tool.github.io/inventor/
helm install inventor inventor/inventor

secret.API_TOKEN has no default - helm install/helm template refuses to render until You set it explicitly, e.g. --set secret.API_TOKEN=.... secret.SD_TOKEN is optional and defaults to empty, matching the app's own behavior (see below) - only set it if you expose /discover, /group outside the cluster.

Registering new target:

curl -X PUT -H "x-api-token: secret" http://127.0.0.1:9101/target \
-d '{"static_config": {"targets": ["10.0.10.2:9100"], "labels": {"__meta_datacenter": "dc-01", "__meta_prometheus_job": "node"}, "target_group": "mygroup"}}'

Fetching a target by id (ids are deterministic - this is what the PUT above returns for targets: ["10.0.10.2:9100"]):

curl -X GET -H "x-api-token: secret" http://127.0.0.1:9101/target \
-d '{"id": "3ebd7d3f-feb7-5a81-aa0d-89c9d0a516ed"}'

Removing a target by id:

curl -X DELETE -H "x-api-token: secret" http://127.0.0.1:9101/target \
-d '{"id": "3ebd7d3f-feb7-5a81-aa0d-89c9d0a516ed"}'

More examples: tests/end-to-end

Prometheus SD config example

scrape_configs:
- job_name: http_sdhttp_sd_configs:
- url: http://127.0.0.1:9101/discover# if SD_TOKEN env variable is setheaders:
- "x-sd-token: REDACTED"

Prometheus SD config with groups example

scrape_configs:
- job_name: http_sd_mygrouphttp_sd_configs:
- url: http://127.0.0.1:9101/group?name=mygroup# if SD_TOKEN env variable is setheaders:
- "x-sd-token: REDACTED"

Configuration Environment Variables

• REDIS_ADDR: redis server address to store metrics and targets
• REDIS_PORT: redis server port
• REDIS_DBNO: redis server keyspace
• TTL_SECONDS: ttl for storing target, default is 6h (21600 seconds)
• API_TOKEN: API token for manipulating targets
• SD_TOKEN: Optional token for Prometheus HTTP SD, is empty by default and not validating (header x-sd-token)
• LISTEN_PORT: HTTP listen port, default is 9101
• APP_ENV: when unset, config is loaded from a .env file in the working directory; when set, all variables above must be present in the real environment

Custom discovered labels

• __meta_inventor_sd_module: contains element of modules: [], useful for relabeling to add __param_module

Data file with modules for exporter_exporter example:

{
"static_config": {
"targets": ["host.local:9999"],
"labels": {
"__inventor_sd_metrics_path": "/proxy",
"__inventor_sd_job": "inventor-exporter-proxy",
"datacenter": "the-dc"
},
"modules": ["node_exporter","ipmi_exporter"],
"target_group": "inventor-default"
}
}

Prometheus SD config with relabel_config example

scrape_configs:
- job_name: http_sd_mygrouphttp_sd_configs:
- url: http://127.0.0.1:9101/discover# if SD_TOKEN env variable is setheaders:
- "x-sd-token: REDACTED"relabel_configs:
# discovered label
- source_labels: [ __meta_inventor_sd_module ]target_label: __param_module# labels from data file
- source_labels: [ __inventor_sd_metrics_path ]target_label: __metrics_path__
- source_labels: [ __inventor_sd_job ]target_label: job

This produces scrape jobs with module parameter:

http://host.local:9999/proxy&module=node_exporter
http://host.local:9999/proxy&module=ipmi_exporter

API Methods

• GET /discover
  • Returning the list of targets in Prometheus HTTP SD format
• GET /group
  • Returning targets by group name /group?name=mygroup
• PUT /target
  • Adds the new target
• GET /target
  • Returning target by ID
• DELETE /target
  • Removing target by ID
• GET /metrics
  • Metrics in prometheus format
• GET /healthcheck
  • Health Check for kubernetes deployments

Build Docker image

docker build -t ghcr.io/code-tool/inventor/inventor:$(cat VERSION.txt) --build-arg BUILD_VERSION=$(cat VERSION.txt) -f docker/Dockerfile .

pulling image:

docker pull ghcr.io/code-tool/inventor/inventor:$(cat VERSION.txt)

License

Covered under the MIT license.