The official Python 2 and 3 client for Prometheus.
One: Install the client:
pip install prometheus_client
Two: Paste the following into a Python interpreter:
fromprometheus_clientimportstart_http_server, Summaryimportrandomimporttime# Create a metric to track time spent and requests made.REQUEST_TIME=Summary('request_processing_seconds', 'Time spent processing request')
# Decorate function with metric.@REQUEST_TIME.time()defprocess_request(t):
"""A dummy function that takes some time."""time.sleep(t)
if__name__=='__main__':
# Start up the server to expose the metrics.start_http_server(8000)
# Generate some requests.whileTrue:
process_request(random.random())Three: Visit http://localhost:8000/ to view the metrics.
From one easy to use decorator you get:
request_processing_seconds_count: Number of times this function was called.request_processing_seconds_sum: Total amount of time spent in this function.
Prometheus's rate function allows calculation of both requests per second,
and latency over time from this data.
In addition if you're on Linux the process metrics expose CPU, memory and
other information about the process for free!
pip install prometheus_client
This package can be found on PyPI.
Four types of metric are offered: Counter, Gauge, Summary and Histogram. See the documentation on metric types and instrumentation best practices on how to use them.
Counters go up, and reset when the process restarts.
fromprometheus_clientimportCounterc=Counter('my_failures_total', 'Description of counter')
c.inc() # Increment by 1c.inc(1.6) # Increment by given valueThere are utilities to count exceptions raised:
@c.count_exceptions()deff():
passwithc.count_exceptions():
pass# Count only one type of exceptionwithc.count_exceptions(ValueError):
passGauges can go up and down.
fromprometheus_clientimportGaugeg=Gauge('my_inprogress_requests', 'Description of gauge')
g.inc() # Increment by 1g.dec(10) # Decrement by given valueg.set(4.2) # Set to a given valueThere are utilities for common use cases:
g.set_to_current_time() # Set to current unixtime# Increment when entered, decrement when exited.@g.track_inprogress()deff():
passwithg.track_inprogress():
passA Gauge can also take its value from a callback:
d=Gauge('data_objects', 'Number of objects')
my_dict= {}
d.set_function(lambda: len(my_dict))Summaries track the size and number of events.
fromprometheus_clientimportSummarys=Summary('request_latency_seconds', 'Description of summary')
s.observe(4.7) # Observe 4.7 (seconds in this case)There are utilities for timing code:
@s.time()deff():
passwiths.time():
passThe Python client doesn't store or expose quantile information at this time.
Histograms track the size and number of events in buckets. This allows for aggregatable calculation of quantiles.
fromprometheus_clientimportHistogramh=Histogram('request_latency_seconds', 'Description of histogram')
h.observe(4.7) # Observe 4.7 (seconds in this case)The default buckets are intended to cover a typical web/rpc request from milliseconds to seconds.
They can be overridden by passing buckets keyword argument to Histogram.
There are utilities for timing code:
@h.time()deff():
passwithh.time():
passAll metrics can have labels, allowing grouping of related time series.
See the best practices on naming and labels.
Taking a counter as an example:
fromprometheus_clientimportCounterc=Counter('my_requests_total', 'HTTP Failures', ['method', 'endpoint'])
c.labels('get', '/').inc()
c.labels('post', '/submit').inc()Labels can also be passed as keyword-arguments:
fromprometheus_clientimportCounterc=Counter('my_requests_total', 'HTTP Failures', ['method', 'endpoint'])
c.labels(method='get', endpoint='/').inc()
c.labels(method='post', endpoint='/submit').inc()The Python client automatically exports metrics about process CPU usage, RAM,
file descriptors and start time. These all have the prefix process, and
are only currently available on Linux.
The namespace and pid constructor arguments allows for exporting metrics about other processes, for example:
ProcessCollector(namespace='mydaemon', pid=lambda: open('/var/run/daemon.pid').read())
There are several options for exporting metrics.
Metrics are usually exposed over HTTP, to be read by the Prometheus server.
The easiest way to do this is via start_http_server, which will start a HTTP
server in a daemon thread on the given port:
fromprometheus_clientimportstart_http_serverstart_http_server(8000)Visit http://localhost:8000/ to view the metrics.
To add Prometheus exposition to an existing HTTP server, see the MetricsHandler class
which provides a BaseHTTPRequestHandler. It also serves as a simple example of how
to write a custom endpoint.
To use prometheus with twisted, there is MetricsResource which exposes metrics as a twisted resource.
fromprometheus_client.twistedimportMetricsResourcefromtwisted.web.serverimportSitefromtwisted.web.resourceimportResourcefromtwisted.internetimportreactorroot=Resource()
root.putChild(b'metrics', MetricsResource())
factory=Site(root)
reactor.listenTCP(8000, factory)
reactor.run()To use Prometheus with WSGI, there is
make_wsgi_app which creates a WSGI application.
fromprometheus_clientimportmake_wsgi_appfromwsgiref.simple_serverimportmake_serverapp=make_wsgi_app()
httpd=make_server('', 8000, app)
httpd.serve_forever()Such an application can be useful when integrating Prometheus metrics with WSGI apps.
The method start_wsgi_server can be used to serve the metrics through the
WSGI reference implementation in a new thread.
fromprometheus_clientimportstart_wsgi_serverstart_wsgi_server(8000)The textfile collector allows machine-level statistics to be exported out via the Node exporter.
This is useful for monitoring cronjobs, or for writing cronjobs to expose metrics about a machine system that the Node exporter does not support or would not make sense to perform at every scrape (for example, anything involving subprocesses).
fromprometheus_clientimportCollectorRegistry, Gauge, write_to_textfileregistry=CollectorRegistry()
g=Gauge('raid_status', '1 if raid array is okay', registry=registry)
g.set(1)
write_to_textfile('/configured/textfile/path/raid.prom', registry)A separate registry is used, as the default registry may contain other metrics such as those from the Process Collector.
The Pushgateway allows ephemeral and batch jobs to expose their metrics to Prometheus.
fromprometheus_clientimportCollectorRegistry, Gauge, push_to_gatewayregistry=CollectorRegistry()
g=Gauge('job_last_success_unixtime', 'Last time a batch job successfully finished', registry=registry)
g.set_to_current_time()
push_to_gateway('localhost:9091', job='batchA', registry=registry)A separate registry is used, as the default registry may contain other metrics such as those from the Process Collector.
Pushgateway functions take a grouping key. push_to_gateway replaces metrics
with the same grouping key, pushadd_to_gateway only replaces metrics with the
same name and grouping key and delete_from_gateway deletes metrics with the
given job and grouping key. See the
Pushgateway documentation
for more information.
instance_ip_grouping_key returns a grouping key with the instance label set
to the host's IP address.
It is also possible to expose metrics to systems other than Prometheus. This allows you to take advantage of Prometheus instrumentation even if you are not quite ready to fully transition to Prometheus yet.
Metrics are pushed over TCP in the Graphite plaintext format.
fromprometheus_client.bridge.graphiteimportGraphiteBridgegb=GraphiteBridge(('graphite.your.org', 2003))
# Push once.gb.push()
# Push every 10 seconds in a daemon thread.gb.start(10.0)Sometimes it is not possible to directly instrument code, as it is not in your control. This requires you to proxy metrics from other systems.
To do so you need to create a custom collector, for example:
fromprometheus_client.coreimportGaugeMetricFamily, CounterMetricFamily, REGISTRYclassCustomCollector(object):
defcollect(self):
yieldGaugeMetricFamily('my_gauge', 'Help text', value=7)
c=CounterMetricFamily('my_counter_total', 'Help text', labels=['foo'])
c.add_metric(['bar'], 1.7)
c.add_metric(['baz'], 3.8)
yieldcREGISTRY.register(CustomCollector())SummaryMetricFamily and HistogramMetricFamily work similarly.
Experimental: This feature is new and has rough edges.
Prometheus client libaries presume a threaded model, where metrics are shared across workers. This doesn't work so well for languages such as Python where it's common to have processes rather than threads to handle large workloads.
To handle this the client library can be put in multiprocess mode. This comes with a number of limitations:
- Registries can not be used as normal, all instantiated metrics are exported
- Custom collectors do not work (e.g. cpu and memory metrics)
- The pushgateway cannot be used
- Gauges cannot use the
pidlabel - Gunicron's
preload_appfeature is not supported
There's several steps to getting this working:
One: Gunicorn deployment
The prometheus_multiproc_dir environment variable must be set to a directory
that the client library can use for metrics. This directory must be wiped
between Gunicorn runs (before startup is recommended).
Put the following in the config file:
defworker_exit(server, worker):
fromprometheus_clientimportmultiprocessmultiprocess.mark_process_dead(worker.pid)Two: Inside the application
fromprometheus_clientimportmultiprocessfromprometheus_clientimportgenerate_latest, CollectorRegistry, CONTENT_TYPE_LATEST, Gauge# Example gauge.IN_PROGRESS=Gauge("inprogress_requests", "help", multiprocess_mode='livesum')
# Expose metrics.@IN_PROGRESS.track_inprogress()defapp(environ, start_response):
registry=CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
data=generate_latest(registry)
status='200 OK'response_headers= [
('Content-type', CONTENT_TYPE_LATEST),
('Content-Length', str(len(data)))
]
start_response(status, response_headers)
returniter([data])Three: Instrumentation
Counters, Summarys and Histograms work as normal.
Gauges have several modes they can run in, which can be selected with the
multiprocess_mode parameter.
- 'all': Default. Return a timeseries per process alive or dead.
- 'liveall': Return a timeseries per process that is still alive.
- 'livesum': Return a single timeseries that is the sum of the values of alive processes.
- 'max': Return a single timeseries that is the maximum of the values of all processes, alive or dead.
- 'min': Return a single timeseries that is the minimum of the values of all processes, alive or dead.
The Python client supports parsing the Promeheus text format. This is intended for advanced use cases where you have servers exposing Prometheus metrics and need to get them into some other system.
fromprometheus_client.parserimporttext_string_to_metric_familiesforfamilyintext_string_to_metric_families("my_gauge 1.0\n"):
forsampleinfamily.samples:
print("Name: {0} Labels: {1} Value: {2}".format(*sample))