docs/node-mixin/alerts: use ratio for network alerts

[paulfantom]

Reduce noise in alerts depending on network traffic volume.

Related to openshift/cluster-monitoring-operator#937

/cc @SuperQ @simonpasquier

[simonpasquier]

LGTM
for context, this is the discussion that triggers the change: openshift/cluster-monitoring-operator#937 (comment)

[discordianfish]

Makes sense, LGTM

[beorn7]

Hmmm, I'm not sure of this. The discussion linked above seems to revolve around an actual problem (i.e. while those few errors are harmless in practice at the moment, they are definitely not OK to happen and point towards a problem of the switch in question). I can imagine the alert is designed to detect this kind of error, and therefore deliberately not based on a ratio.

I'm not a network expert, though. I don't know for sure if on modern high-bandwidth network hardware, it's acceptable to have a certain low percentage of errors. If that's the case, we should definitely base the alerts on a ratio. If not, i.e. if any error showing up is essentially a reason to assume something is broken, I'd prefer the absolute thresholds.

[paulfantom]

Considering that most network QoS settings are based on the rate and most networking nomenclature circulates around rates like PER (packet error rate) or BER (bit error rate). It seemed to me like a good idea to also base an alert on a rate.

Digging deeper, RX and TX errors are cumulative and can be increased when one of the following happens:

• too-long-frame error
• ring-buffer overflow error
• CRC error
• frame alignment error
• fifo overrun
• missed packets

Those can happen for various reasons: firewall misconfiguration, NIC dying, microwave near an access point, etc. In those cases, it is not important that a small number of errors happened, but that there is a constant error ratio increase. A small number of errors can be handled without much delay and without disruption to running services (especially in case of TCP as it will cause packets to be resent). On top of that, some network mediums will inherently have a fairly stable high error number due to an underlying physical medium (basically every over-the-air network).

[beorn7]

Thanks @paulfantom . Makes a lot of sense, too. @tomwilkie originally authored this. @tomwilkie : did you have anything in mind that we missed in the discussion so far?

[paulfantom]

Any updates on this?

[beorn7]

I guess @tomwilkie had enough time to intervene. Silence == consent. Approved!

[beorn7]

Merging as @discordianfish has approved this, too.