diff --git a/how-asserts-works/mapping-metrics-to-saafe-model/request-error-and-duration-red.md b/how-asserts-works/mapping-metrics-to-saafe-model/request-error-and-duration-red.md index 7ef2055..d619ac0 100644 --- a/how-asserts-works/mapping-metrics-to-saafe-model/request-error-and-duration-red.md +++ b/how-asserts-works/mapping-metrics-to-saafe-model/request-error-and-duration-red.md @@ -1,131 +1,177 @@ # Request, Error and Duration (RED) +**NOTE:** These instructions are meant for a quick PoC to demonstrate capability. They involve creating copy of existing metrics and hence would lead to additional metrics being created. This might be acceptable only if the scope of the data being looked at is small. + ### **asserts:request:total** In Asserts, the `asserts:request:total` records the total count of requests. -In **SpringBoot,** the request metrics for incoming requests are available through `http_server_requests_seconds` which is a [Histogram](https://prometheus.io/docs/concepts/metric\_types/#histogram). Similarly the request metrics for outgoing calls are available through `http_client_requests_second` which is also a Histogram. These metrics are mapped to `asserts:request:total` for incoming and outgoing requests. +In **SpringBoot,** the request metrics for incoming requests are available through `http_server_requests_seconds` which +is a [Histogram](https://prometheus.io/docs/concepts/metric\_types/#histogram). Similarly, the request metrics for +outgoing calls are available through `http_client_requests_second` which is also a Histogram. These metrics are mapped +to `asserts:request:total` for incoming and outgoing requests. ``` # Incoming requests - record: asserts:request:total - expr: http_server_requests_seconds_count + expr: |- + label_replace(http_server_requests_seconds_count, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_server_requests_seconds_count + asserts_source: spring_boot + asserts_metric_request: total asserts_request_type: inbound - asserts_source: spring_boot # Outgoing requests made through Spring classes like RestTemplate - record: asserts:request:total - expr: http_client_requests_seconds_count + expr: |- + label_replace(http_client_requests_seconds_count, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_client_requests_seconds_count - asserts_request_type: outbound - asserts_source: spring_boot + asserts_source: spring_boot + asserts_metric_request: total + asserts_request_type: outbound ``` -
Asserts Meta LabelDescription
asserts_nameUsed by Asserts to keep track of the source metric name from which an Asserts standard metric is recorded.
asserts_request_typeUsed by Asserts to categorize requests into different kinds. By default, for all supported http based frameworks, Asserts categorizes requests into inbound for incoming requests and outbound for outgoing http calls. But these can be arbitrary names to group apis e.g. timer_task.
asserts_request_contextUsed by Asserts to identify a unique request. For http requests, whether inbound, or outbound, this typically maps to the relative part of the request URI with the high cardinality parameters stripped off. For e.g. /track/order/{} with {} being a placeholder for an order id. Frameworks like Springboot actuator Prometheus metrics have labels like uri. A relabeling rule is used to map uri to asserts_request_context.
asserts_sourceUsed by Asserts to identify which framework/instrumentation captured the metric.
- -In the above Asserts meta labels, except for `asserts_request_context` the remaining labels are specified in the recording rule itself. The `asserts_request_context` is specified through a relabeling rule as follows - -``` -- source_labels: [asserts_name, uri] - regex: http_.*_requests_seconds_.*;(.+) # spring boot server and client - target_label: asserts_request_context - replacement: $1 -``` + + + + + + + + +
Asserts Meta LabelDescription
asserts_sourceUsed by Asserts to identify which framework/instrumentation captured the metric.
asserts_metric_requestUsed by Asserts to identify this as a request metric. Valid values are total when source metric is a counter and gauge when source metric is a gauge
asserts_request_typeUsed by Asserts to categorize requests into different kinds. By default, for all supported http based frameworks, Asserts categorizes requests into inbound for incoming requests and outbound for outgoing http calls. But these can be arbitrary names to group apis e.g. timer_task or query etc
asserts_request_contextUsed by Asserts to identify a unique request. For http requests, whether inbound, or outbound, this typically maps to the relative part of the request URI with the high cardinality parameters stripped off. For e.g. /track/order/{} with {} being a placeholder for an order id. Frameworks like Springboot actuator Prometheus metrics have labels like uri. The label_replace function is used to map uri to asserts_request_context.
-The label `uri` is available in both the server and client metrics. Notice how the `asserts_name` meta label is being used to target the relabeling only for the metrics of interest. Once these recording rules are [added](https://docs.asserts.ai/user-guide/assertion-management#howassertsworks-wip-additions) to Asserts, the following things will happen +Once these rules are added, the following things will happen * The **Request Rate** will be computed and shown in the Service KPI Dashboard. -* The **Request Rate** will be observed for anomalies, and the **RequestRateAnomaly** will be triggered when there are anomalies. +* The **Request Rate** will be observed for anomalies, and the **RequestRateAnomaly** will be triggered when there are + anomalies. + +NOTE: In the above example, the source metric is available as a counter. So it was mapped to asserts:request:total. If the source metric were a gauge, +then it should be mapped to asserts:request:gauge and set asserts_metric_request: gauge. ### **asserts:error:total** -In Asserts, the `asserts:error:total` metric records the total count of errors, broken down by different error types. Let's add this rule for SpringBoot `inbound` and `outbound` requests +In Asserts, the `asserts:error:total` metric records the total count of errors, broken down by different error types. +Let's add this rule for SpringBoot `inbound` and `outbound` requests ``` # Inbound request errors - record: asserts:client:error:total - expr: http_server_requests_seconds_count{status=~"4.."} + expr: | + label_replace(http_server_requests_seconds_count{status=~"4.."}, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_server_requests_seconds_count - asserts_request_type: inbound asserts_source: spring_boot + asserts_metric_error: client_total + asserts_request_type: inbound - record: asserts:error:total - expr: http_server_requests_seconds_count {status=~"5.."} + expr: | + label_replace(http_server_requests_seconds_count {status=~"5.."}, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_server_requests_seconds_count + asserts_source: spring_boot + asserts_metric_error: total asserts_request_type: inbound asserts_error_type: server_errors - asserts_source: spring_boot # Outbound request errors - record: asserts:error:total - expr: http_client_requests_seconds_count{status=~"4.."} + expr: | + label_replace(http_client_requests_seconds_count{status=~"4.."}, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_client_requests_seconds_count + asserts_source: spring_boot + asserts_metric_error: total asserts_request_type: outbound asserts_error_type: client_errors - asserts_source: spring_boot - record: asserts:error:total - expr: http_client_requests_seconds_count{status=~"5.."} + expr: | + label_replace(http_client_requests_seconds_count{status=~"5.."}, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_client_requests_seconds_count + asserts_source: spring_boot + asserts_metric_error: total asserts_request_type: outbound asserts_error_type: server_errors - asserts_source: spring_boot ``` -Notice how the `status` label from the Spring metric is used to classify errors into `client_errors` and `server_errors`. Once these rules are added, the following things will happen + + + + + + +
Asserts Meta LabelDescription
asserts_metric_errorUsed by Asserts to identify this as an error metric of type counter. Valid values are client_total and client_gauge
asserts_error_typeUsed by Asserts to categorize errors into different kinds. The commonly useful types are server_errors and client_errors. In this example, a condition on the status label has been used to define these types. Note that the client errors for inbound calls are mapped using a special type client_total. This is because, the inbound client errors tend to be noisy. Asserts will still observe them, but the signals captured surface only when there are anomalies. I.E. if there is a steady stream of client errors, there won't be a signal. However, if there is a sudden change in the rate of these errors, then an anomaly signal will be generated.
+ +Once these rules are added, the following things will happen * The **Error Ratio** will be computed for all request contexts and shown in the Service KPI Dashboard. * The **ErrorRatioBreach** will be triggered if the ratio breaches a certain threshold. * The **ErrorBuildup** (multi burn-multi window) will be triggered if the error budget breaches. -* The **Error Ratio** will be observed for anomalies and **ErrorRatioAnomaly** will be triggered when there are anomalies. +* The **Error Ratio** will be observed for anomalies and **ErrorRatioAnomaly** will be triggered when there are + anomalies. + +NOTE: In the above example, the source metric is available as a counter. So it was mapped to asserts:error:total. If the source metric were a gauge, +then it should be mapped to asserts:error:gauge and set asserts_metric_error: gauge or asserts_metric_error: client_gauge in the case of inbound client errors. ### **asserts:latency:total and asserts:latency:count** -Asserts computes the latency average using the `asserts:latency:total` and `asserts:latency:count` metrics. The former metric is the latency total time in seconds and the latter is the total number of requests. Let's add the recording rules for these two metrics from the respective Histogram metrics which have the `_sum` and `_count` metrics. +Asserts computes the latency average using the `asserts:latency:total` and `asserts:latency:count` metrics. The former +metric is the latency total time in seconds and the latter is the total number of requests. Let's add the recording +rules for these two metrics from the respective Histogram metrics which have the `_sum` and `_count` metrics. ``` # Inbound Latency - record: asserts:latency:total - expr: http_server_requests_seconds_sum + expr: | + label_replace(http_server_requests_seconds_sum, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_server_requests_seconds_sum + asserts_source: spring_boot + asserts_metric_latency: seconds_sum asserts_request_type: inbound - asserts_source: spring_boot - record: asserts:latency:count - expr: http_server_requests_seconds_count + expr: | + label_replace(http_server_requests_seconds_count, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_server_requests_seconds_count + asserts_source: spring_boot + asserts_metric_latency: count asserts_request_type: inbound - asserts_source: spring_boot # Outbound latency - record: asserts:latency:total - expr: http_client_requests_seconds_sum + expr: + label_replace(http_client_requests_seconds_sum, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_client_requests_seconds_sum + asserts_source: spring_boot + asserts_metric_latency: seconds_sum asserts_request_type: outbound - asserts_source: spring_boot - record: asserts:latency:count - expr: http_client_requests_seconds_count + expr: + label_replace(http_client_requests_seconds_count, "asserts_request_context", "$1", "uri", "(.+)") labels: - asserts_name: http_client_requests_seconds_count + asserts_source: spring_boot + asserts_metric_latency: count asserts_request_type: outbound - asserts_source: spring_boot ``` + + + + + +
Asserts Meta LabelDescription
asserts_metric_latencyUsed by Asserts to identify the numerator and denominator to compute the latency average along with the unit of the source latency metric. Valid values for latency, the numerator, are seconds_sum, milliseconds_sum and microseconds_sum. For the latency count, the denominator, the valid value is count
+ Once these rules are added. The following things happen * **Latency Average** is computed for all requests and shown in the Service KPI Dashboards -* The **Latency Average** will be observed for anomalies, and **LatencyAverageAnomaly** will be triggered when there are anomalies. +* The **Latency Average** will be observed for anomalies, and **LatencyAverageAnomaly** will be triggered when there are + anomalies. + +NOTE: In the above example, the source metric is available as a counter. So it was mapped to asserts:latency:total and asserts:latency:count. If the source metric were a gauge, +then it should be directly mapped to asserts:latency:average. While doing this, be mindful of the labels in the source metric. +When the source is a counter, Asserts does some aggregation internally and only the key labels are retained reducing the cardinality in the metrics it records. In the direct mapping this is not the case. + ### **asserts:latency:p99** @@ -135,33 +181,47 @@ Similarly, we can record the latency p99 for the requests as follows # Inbound requests latency P99 - record: asserts:latency:p99 expr: > - histogram_quantile ( - 0.99, - sum(rate(http_server_requests_seconds_bucket[5m]) > 0) - by (le, namespace, job, service, uri, asserts_request_context, asserts_env, asserts_site) + label_replace( + histogram_quantile ( + 0.99, + sum(rate(http_server_requests_seconds_bucket[5m]) > 0) by (le, namespace, job, service, workload, uri, asserts_env, asserts_site) + ) + , "asserts_request_context", "$1", "uri", "(.+)" ) labels: - asserts_name: http_server_requests_seconds_bucket + asserts_source: spring_boot asserts_entity_type: Service asserts_request_type: inbound - asserts_source: spring_boot # Outbound requests latency P99 - record: asserts:latency:p99 expr: > - histogram_quantile ( - 0.99, - sum(rate(http_client_requests_seconds_bucket[5m]) > 0) - by (le, namespace, job, service, uri, asserts_request_context, asserts_customer, asserts_env, asserts_site) + label_replace( + histogram_quantile ( + 0.99, + sum(rate(http_client_requests_seconds_bucket[5m]) > 0) by (le, namespace, job, service, workload, uri, asserts_env, asserts_site) + ) + , "asserts_request_context", "$1", "uri", "(.+)" ) labels: - asserts_name: http_client_requests_seconds_bucket + asserts_source: spring_boot asserts_entity_type: Service asserts_request_type: outbound - asserts_source: spring_boot ``` -Once this is recorded, Asserts will show this metric in the Service KPI Dashboard and also start observing for the clock minutes when the **Latency P99** exceeds a threshold. These minutes will be tracked through a total bad minutes counter. Based on the ratio of `bad minutes` to `total minutes` in a given time-window, the **LatencyP99ErrorBuildup,** a Multi-Burn, Multi-Window error-budget based alert will be triggered. + + + + + + + +
Asserts Meta LabelDescription
asserts_envUsed by Asserts to identify the environment. All discovered entities and observed metrics are automatically scoped to an environment.
asserts_siteUsed by Asserts to identify the region/site within an environment. For e.g. you could have a prod environment but multiple regions, for e.g. us-east-1, us-west-2 etc. This label is then used to capture the region information. Note that this depends on how environment information is encoded in the metrics. Sometimes, both the environment and the region information may be encoded in a single label value, in which case, the asserts_env will have that value and this label may not be present.
asserts_entity_typeUsed by Asserts to identify at what level is the metric being observed. The workload, service, job are special labels that Asserts uses to discover the Service. In this example, while aggregating, these labels are being retained. So this metric will be observed for the corresponding Service entity
+ +Once this is recorded, Asserts will show this metric in the Service KPI Dashboard and also start observing for the clock +minutes when the **Latency P99** exceeds a threshold. These minutes will be tracked through a total bad minutes counter. +Based on the ratio of `bad minutes` to `total minutes` in a given time-window, the **LatencyP99ErrorBuildup,** a +Multi-Burn, Multi-Window error-budget based alert will be triggered. ### **asserts:latency:service:p99** @@ -173,10 +233,9 @@ The Latency P99 for the entire service, regardless of different request contexts histogram_quantile ( 0.99, sum(rate(http_server_requests_seconds_bucket[5m]) > 0) - by (le, namespace, job, service, asserts_env, asserts_site) + by (le, namespace, job, service, workload, asserts_env, asserts_site) ) labels: - asserts_name: http_server_requests_seconds_bucket asserts_entity_type: Service asserts_request_type: inbound asserts_source: spring_boot