Collects, buffers, and outputs logs across multiple sources and destinations.
bulklog is written in go and requires little resource.
bulklog supports memory and redis buffering. bulklog also supports failover and can be set up for high availability.
bulklog tries to structure data as JSON since it has enough structure to be accessible while providing felxibility.
A collection is a set of declarative informations about how bulklog should process data.
bulklog outputs JSON docuemts to destinations such as Elasticsearch, MongoDB, etc...
docker.pkg.github.com/khezen/bulklog/bulklog
docker run -p 5017:5017 -v /etc/bulklog:/etc/bulklog docker.pkg.github.com/khezen/bulklog/bulklog:stablelatest2.0.1,2.0,2,stable
| key | Description | Default Value |
|---|---|---|
| CONFIG_PATH | path to the configuration folder | /etc/bulklog |
Deploy bulklog to a kubernetes cluster using Helm.
helm repo add khezen https://khezen.github.com/charts
helm install khezen/bulklog --name bulklogDefault config.yaml.
Peristence is disabled by default in which case data is buffered in memory. If enabled, it uses Redis(>= 2.4) to persist documents buffer. Learn how to tune Redis persistence for your requirements.
persistence:
enabled: trueredis:
endpoint: localhost:6379password: changeme #(optional)db: 0#(optional, default:0)idle_conn: 2#(optional, default: 0)max_conn: 10#(optional, defaut: no limit)provides declarative information about bulklog output.
output:
elasticsearch:
enabled: trueendpoint: localhost:9200scheme: http# aws_auth:# access_key_id: changeme# secret_access_key: changeme# region: eu-west-1# basic_auth:# username: elastic# password: changemefrom version 2.0.0 bulklog supports Elasticsearch 7.0.0 and above
examples:
collections:
- name: logsflush_period: 5 seconds # hours|minutes|seconds|millisecondsretention_period: 45 minutesschema: {}bulklog is schema free but we encourage you to provide some base structure since it might enbale output destination to process data more efficiently.
collections:
- name: logsflush_period: 5 seconds # hours|minutes|seconds|millisecondsretention_period: 45 minutesshards: 5replicas: 1schema:
source: type: stringmax_length: 64stream: type: stringlength: 6event: type: stringtime:
type: datetimedate_format: 2006-01-02T15:04:05.999999999Z07:00Even in the case above, bulklog remains schema free enabling log decoration with additional field.
- name:
{collection name} - flush_period:
{duration}- flush buffer to output every
{duration}
- flush buffer to output every
- retention_period:
{duration}- if an output is unavailable, retention_period set how long bulklog tries to output data to this output
- if the output is unavailable for too long, retention_period ensure that bulklog will not accumulate too much data and will be able to serve other outputs.
- shards: the number of shards to allocate this collection to
- replicas: the number of replicas to allocate this collection to
- schema:
{map of fields by field name}
Learn more about Elasticsearch sharding and replication.
- type:
{field type}- see supported types
- length:
{field exact length}(optional,string only) - max_length:
{field maximum length}(optional, string only) - date_format:
{date time formatting}(optional, datetime only)
POSTbulklog/v1/{collectionName} HTTP/1.1Content-Type: application/json
{
...
}
HTTP/1.1 200 OKexample:
POSTbulklog/v1/logs HTTP/1.1Content-Type: application/json
{
"source":"service1",
"stream": "stderr",
"event": "divizion by zero",
"time": "2018-11-15T14:12:12Z"
}
### push documents in batches```httpPOST /v1/{collectionName}/batch HTTP/1.1Content-Type: application/json
{...}
{...}
HTTP/1.1 200 OKexample:
POSTbulklog/v1/logs/batch HTTP/1.1Content-Type: application/json
{"source":"service1","stream": "stderr","event": "divizion by zero","time" : "2019-01-13T19:30:12"}
{"source":"service1","stream": "stdout","event": "successfully processed","time" : "2019-01-13T19:35:12"}
HTTP/1.1 200 OKGETbulklog/liveness HTTP/1.1HTTP/1.1 200 OKGETbulklog/readiness HTTP/1.1HTTP/1.1 200 OKbool :
TrueorFalseunint8 :
0to255uint16 :
0to65535uint32 :
0to4294967295unit64 :
0to18446744073709551615int8 :
-128to127int16 :
-32768to32767int32 :
-2147483648to2147483647int64 :
-9223372036854775808to9223372036854775807float32 :
-3.40282346638528859811704183484516925440e+38to3.40282346638528859811704183484516925440e+38float64 :
-1.797693134862315708145274237317043567981e+308to1.797693134862315708145274237317043567981e+308string : sequence of characters
- lenght: string exact length
- max_length: string maximum length
datetime :
1970-01-01T00:00:00.000000000Z(example)- bulklog doesn't check the date format. Most outputs accept any even if it defers from the configured one
- date_format: date format string
Mon Jan _2 15:04:05 2006Mon Jan _2 15:04:05 MST 2006Mon Jan 02 15:04:05 -0700 200602 Jan 06 15:04 MST02 Jan 06 15:04 -0700Monday, 02-Jan-06 15:04:05 MSTMon, 02 Jan 2006 15:04:05 MSTMon, 02 Jan 2006 15:04:05 -07002006-01-02T15:04:05Z07:002006-01-02T15:04:05.999999999Z07:00(default)3:04PMJan _2 15:04:05Jan _2 15:04:05.000Jan _2 15:04:05.000000Jan _2 15:04:05.0000000002006-01-02 15:04:05 MST2006-01-02 15:04:05.999999999 MST
object : inner document
If you have any problems or questions, please ask for help through a GitHub issue.
Help is always welcome! For example, documentation (like the text you are reading now) can always use improvement. There's always code that can be improved. If you ever see something you think should be fixed, you should own it. If you have no idea what to start on, you can browse the issues labeled with help wanted.
As a potential contributor, your changes and ideas are welcome at any hour of the day or night, weekdays, weekends, and holidays. Please do not ever hesitate to ask a question or send a pull request.