Query TXN2 data by Account, Model and index pattern. Save a Query and execute a saved Query.

Configuration

Configuration is inherited from txn2/micro. The following configuration is specific to query:

Flag	Environment Variable	Description
-esServer	ELASTIC_SERVER	Elasticsearch Server (default "http://elasticsearch:9200")
-provisionServer	PROVISION_SERVER	Provision Server (txn2/provision)
-authCache	AUTH_CACHE	Seconds to cache key (BasicAuth) authentication.

Routes

Method	Route Pattern	Description
POST	run/:account	Run a query to test it. This action does not save the query.
GET	exec/:account/:id	Executes a saved query.
GET	system/exec/:account/:id	Executes a saved system query.
POST	upsert/:account	Upsert (save) a query.
GET	get/:account/:id	Get a saved query.
POST	search/:account	Search for saved queries.

Local Development

The project includes a docker-compose.yml file with Elasticsearch, Logstash, Kibana, Cerebro, txn2/rxtx, txn2/rtbeat, txn2/provision and txn2/tm:

docker-compose up

Add test account with txn2/provision:

curl -X POST \
http://localhost:8070/account \
-H 'Content-Type: application/json' \
-d '{ "id": "test", "description": "This is a test account", "display_name": "Test Organization", "active": true, "access_keys": [ { "name": "test", "key": "PDWgYr3bQGNoLptBRDkLTGQcRmCMqLGRFpXoXJ8xMPsMLMg3LHvWpJgDu2v3LYBA", "description": "Generic access key 2", "active": true } ], "modules": [ "wx", "data_science" ]}'

Add an admin User with with txn2/provision given access to the test account:

curl -X POST \
http://localhost:8070/user \
-H 'Content-Type: application/json' \
-d '{	"id": "test",	"description": "Test User admin",	"display_name": "Test User",	"active": true,	"sysop": false,	"password": "eWidL7UtiWJABHgn8WA",	"sections_all": false,	"sections": ["api", "config", "data"],	"accounts": ["test"],	"admin_accounts": ["test"]}'

Get a user Token from txn2/provision with the User's id and password:

TOKEN=$(curl -s -X POST \ http://localhost:8070/authUser?raw=true \ -d '{ "id": "test", "password": "eWidL7UtiWJABHgn8WA" }')&&echo$TOKEN

Insert a sample Model into the test account using txn2/tm running on port 8085:

curl -X POST http://localhost:8085/model/test \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{ "machine_name": "some_metrics", "display_name": "Some Metrics", "description": "A sample model describing some metrics sent through rxtx", "fields": [ { "machine_name": "device_id", "display_name": "Device ID", "data_type": "keyword" }, { "machine_name": "random_number", "display_name": "Random Number", "data_type": "integer" }, { "machine_name": "another_number", "display_name": "Another Number", "data_type": "integer" } ]}'

Within Elasticsearch there is now a template _template/test-data-some_metrics for the account test describing txn2/rxtx/txn2/rtbeat inbound data matching index pattern test-data-some_metrics-*. Send some sample data to Elasticsearch through txn2/rxtx and wait for the batch interval (specified in the docker-compose) to complete:

curl -X POST \
http://localhost:8090/rx/test/some_metrics/sample-data \
-H 'Content-Type: application/json' \
-d "{\"device_id\": \"12345\",\"random_number\": \"${RANDOM}\",\"another_number\": 12345}"

The fields in the data sent to txn2/rxtx should match the fields described in the txn2/tmModel. Although the value for "random_number" is represented here as a string, the template mapping added with the Model instructs Elasticsearch to index it as an integer.

Example Queries

Run query from source. Configure it to use the services running from docker-compose above.

go run ./cmd/query.go --esServer=http://localhost:9200 --provisionServer=http://localhost:8070 --tokenKey="somesharedkey"

Run a Query:

curl -X POST \
http://localhost:8080/run/test \
-H "Authorization: Bearer $TOKEN" \
-d '{ "machine_name": "count_some_metrics", "display_name": "Get all records from the some_metrics index.", "description": "Return all matches", "model": "some_metrics", "idx_pattern": "-ts-*", "query": { "size": 0, "query": { "match_all": {} }	}}'

Upsert a Query:

(must have admin access to account)

curl -X POST \
http://localhost:8080/upsert/test \
-H "Authorization: Bearer $TOKEN" \
-d '{ "machine_name": "count_some_metrics", "display_name": "Get all records from the some_metrics index.", "description": "Return all matches", "model": "some_metrics", "idx_pattern": "-ts-*", "query": { "size": 0, "query": { "match_all": {} }	}}'

Search for queries:

curl -X POST \
http://localhost:8080/search/test \
-H "Authorization: Bearer $TOKEN" \
-d '{ "size": 10, "query": { "match_all": {} }}'

Get a Query:

curl -X GET \
http://localhost:8080/get/test/count_some_metrics \
-H "Authorization: Bearer $TOKEN"

Execute a Query:

curl -X GET \
http://localhost:8080/exec/test/count_some_metrics \
-H "Authorization: Bearer $TOKEN"

Execute a Query with BasicAuth using an AccountAccessKey:

curl -X GET \
http://localhost:8080/exec/test/count_some_metrics \
-H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB'

Query Templates

Query templates use Go's build in template system along with Sprig. The following example tests a query with /run and demonstrates a query string override of default values provided in the "parameters" section:

curl -X POST \
'http://localhost:8080/run/test?size=0&idx=2019.%2A' \
-H 'Authorization: Basic dGVzdDpQRFdnWXIzYlFHTm9McHRCUkRrTFRHUWNSbUNNcUxHUkZwWG9YSjh4TVBzTUxNZzNMSHZXcEpnRHUydjNMWUJB' \
-d '{ "machine_name": "get_some_metrics", "display_name": "Get records from the some_metrics index.", "description": "Get some_metrics", "model": "some_metrics", "idx_pattern": "-ts-{{ index . \"idx\" }}", "query_template": "{\"size\": {{ index . \"size\" }},\"query\": {\"match_all\": {}}}",	"parameters": [ { "machine_name": "size", "default_value": "1" }, { "machine_name": "idx", "default_value": "2019.*" }	]}'

Release Packaging

Build test release:

goreleaser --skip-publish --rm-dist --skip-validate

Build and release:

GITHUB_TOKEN=$GITHUB_TOKEN goreleaser --rm-dist