Kafka header counter for prometheus

This simple script developed in order to count a specefic header values from kafka messages. For example, we are going to count the messages fetched by individual user IDs by twitter crawler.

Configurations

A sample config file provided in config.example.yml. It's required to mount a config file inside the cotainer with the name config.yml at /app/config.yml or provide the config file absoulte path with environment variable CONFIG_ADDRESS.

config example

## Required configs #### just name the service for exported descriptionsservice: example_service## counter name for exported metricscounter_name: test_counter_name## kafka topics to fetch messages fromtopics:
- test_topic_a
- test_topic_b## consumer group name of this exporterconsumer: test_consumer_group## a list of kafka addresses with portaddress:
- kafka.example.com:9092## header name to fetch data from itheader_fields:
- test_header_field## Optional configs #### timeouts in mstimeout: 6000## offset reset policy, can be earliest or latestoffset_reset: earliest## exposed http server port for prometheus scraperport: 80

Type

RESTful API

Source

Apache kafka

Output

Prometheus

Tests

Not developed yet

Deployment Notes

Hardware Requirements

• Memory: 100 Mi - 128 Mi
• CPU: 0.7 - 1
• Storage: None