[RFE] Export certain network routes (e.g. default)

[squeed]

(I intend to implement this, and I have a functioning patch for Linux already, but I'd like to discuss exactly how this should look before filing the PR)

Goal:

Write alerts that detect, for example, that a node's uplink network interface is nearing saturation.

Challenge:

Modern Linux boxes, especially with containers, have lots of network interfaces, and you can't count on the uplink being called eth0 (see systemd.net-naming-scheme).

It would be useful for the node exporter to indicate which network interface has the default route. This is a pretty reasonable proxy for the "uplink" interface.

Proposal:

The simplest solution is to add an additional label (per family) to the node_network_info metric, so it looks something like

node_network_info{address="01:01:01:01:01:01",broadcast="ff:ff:ff:ff:ff:ff",default_route_v4="1",default_route_v6="1",device="eth0",duplex="full",ifalias="",operstate="up"}

Alternatively, we could allow for more flexibility. I could see wanting to expose which interface has a route to a specific IP address. For example, I could see wanting know which interface is the SAN interface. So, I'd configure the node-exporter with the destination 1.2.3.4 as "san" and get the following metrics:

node_network_has_route{device="enp1s0",route="default"} 1
node_network_has_route{device="enp1s0",route="san"} 0
node_network_has_route{device="enp6s1",route="default"} 0
node_network_has_route{device="enp6s1",route="san"} 1

[SuperQ]

A couple comments.

• For node_network_info, let's go with true and false for the default route labels.
• How would this work with multiple route tables?
• How is this data gathered? netlink? Does it require root?

[squeed]

* How would this work with multiple route tables?

Good question! It should "just work". My current implementation is equivalent to executing ip route get 0.0.0.1, asking the kernel to evaluate all the routing rules and compute the source interface. While this is a bit of a hack, and someone could theoretically have a route to 0.0.0.1, this is less fragile than dumping all the route tables and trying to evaluate ourselves.

* How is this data gathered? netlink? Does it require root?

Netlink for linux, yes. The BSDs have a similar mechanism. Neither requires root. I still have to figure out how to stub this out for testing.

[squeed]

Annoyingly, node_network_info is linux-only. That might be nice to expose more broadly.

[discordianfish]

While it's a borderline case of whether this is really a monitoring metric, I see how it's useful. Especially for using this metric to decide which interface to use in graphs etc. So yeah seems reasonable.

[roidelapluie]

This has been added to #1811

[squeed]

woot woot! Thanks for the cleanup.