Error output when using `--collector.cpu.info.flags-include` on` node-exporter: v1.2.0` or later

[yosshi825]

Host operating system: output of uname -a

Linux XXXX 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.2.0 (branch: HEAD, revision: 12968948aec1e2b216a2ecefc45cf3a50671aecb)
  build user:       root@6b17174de526
  build date:       20210715-16:35:54
  go version:       go1.16.6
  platform:         linux/amd64

node-exporter: v1.2.0 or later

node_exporter command line flags

--collector.cpu.info.flags-include=^(aes|avx.?|constant_tsc)$$

I tried the following docker-compose on my mac and found the same error.

uname -a
Darwin XXX Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64

version: '3.9'
services:
  exporter:
    container_name: node-exporter
    image: prom/node-exporter:v1.2.0
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.cpu.info.flags-include=^(aes|avx.?|constant_tsc)$$'
    ports:
      - 9100:9100
    restart: always

I get an error when using --collector.cpu.info.flags-include

Are you running node_exporter in Docker?

yes

What did you do that produced an error?

1. Access / metrics
curl localhost:9100/metrics

2. The following error will be output every time you call /metrics

level=error ts=2021-09-25T09:43:50.476Z caller=stdlib.go:105 msg="error gathering metrics: 20 error(s) occurred:\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"constant_tsc\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"aes\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx2\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"constant_tsc\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"aes\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx2\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"constant_tsc\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"aes\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx2\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"constant_tsc\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"aes\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx2\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"constant_tsc\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"aes\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"node_cpu_flag_info\" { label:<name:\"flag\" value:\"avx2\" > gauge:<value:1 > } was collected before with the same name and label values"

It seems to occur in v1.2.0 or later

What did you expect to see?

I want to get the following metrics without node-exporter outputting an error

# HELP node_cpu_flag_info The `flags` field of CPU information from /proc/cpuinfo.
# TYPE node_cpu_flag_info gauge
node_cpu_flag_info{flag="aes"} 1
node_cpu_flag_info{flag="avx"} 1
node_cpu_flag_info{flag="avx2"} 1
node_cpu_flag_info{flag="constant_tsc"} 1

What did you see instead?

Until node-exporter: v1.1.2, no error occurred even with the same settings.
You can check it in the docker-compose file below.

version: '3.9'
services:
  exporter:
    container_name: node-exporter
    image: prom/node-exporter:v1.1.2
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      # - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
      - '--collector.cpu.info.flags-include=^(aes|avx.?|constant_tsc)$$'
    ports:
      - 9100:9100
    restart: always

[SuperQ]

Yes, it looks like that flag is broken for multiple CPUs. ☹️

[binjip978]

This happens because system is multicore, and /proc/cpuinfo with procfs will report flags for all cpu, but node exporter will try to emit only one metric for all of them with flag label.

Error can be fixed by adding additional label tuple (ProcessorID, CoreID), but for most modern Intel/AMD CPU this will be quite repeatable. But for new ARM systems this may have some sense.

# HELP node_cpu_flag_info The `flags` field of CPU information from /proc/cpuinfo.                      
# TYPE node_cpu_flag_info gauge                                                                         
node_cpu_flag_info{coreID="0",flag="aes",processorID="0"} 1                                             
node_cpu_flag_info{coreID="0",flag="avx",processorID="0"} 1                                             
node_cpu_flag_info{coreID="0",flag="avx2",processorID="0"} 1                                            
node_cpu_flag_info{coreID="0",flag="constant_tsc",processorID="0"} 1                                    
node_cpu_flag_info{coreID="1",flag="aes",processorID="0"} 1                                             
node_cpu_flag_info{coreID="1",flag="avx",processorID="0"} 1                                             
node_cpu_flag_info{coreID="1",flag="avx2",processorID="0"} 1  

[SuperQ]

I wonder if it's worth adding a label, or simply deduplicating. I guess in theory there could be systems with flags only on some CPUs, but it seems unlikely.

[binjip978]

Alternatively to deduplication we may not run updateFieldInfo function in a loop and pretend that all cores are the same and they use the same set of flags. And emit all metrics for bugs and flags only once. But help message should be changed if we decide to avoid adding additional labels, because we are just assuming that all cores are the same.

[yosshi825]

It was fixed in node-exporter: v1.3.0.