Monitors & collects system resource utilization on Linux for Docker containers and Kubernetes pods with fine granularity and low overhead, emitting resource utilization logs in CSVY (csv + yaml) format. Originally, developed in Rust as a custom tool to help detect and analyze millibottlenecks in containerized online systems, rAdvisor runs by polling the target provider (either the local Docker daemon or the Kubernetes API server) every 1 second to get a list of active, running containers/pods. From this list, rAdvisor runs a collection thread every 50ms to get resource utilization data for each active target using Linux
cgroups, outputting the resultant logs in/var/log/radvisor/stats.
Note: filenames correspond to the ID/UID of the container/pod, with the collector initialization timestamp appended at the end.
---
Version: 1.4.0Provider: dockerMetadata:
Created: "2022-01-09T13:25:04Z"Command: 'bash -c ''sleep 2s; apt-get update; sleep 2s; DEBIAN_FRONTEND=noninteractive apt-get install -y stress wget; sleep 2s; dd if=/dev/zero of=/tmp/file1 bs=512M count=1 oflag=direct; sleep 2s; stress --cpu 8 --io 4 --vm 4 --vm-bytes 1024M --timeout 10s; sleep 2s; wget "http://ipv4.download.thinkbroadband.com/10MB.zip"; sleep 2s'''Id: 7762ff15c99a2d238f4d26c22b5eda5b97ebc03bd0a711693104dcb6f71fe411Image: ubuntuLabels: {}Names:
- /silly_elionPorts: []Status: Up Less than a secondSizeRw: ~SizeRootFs: ~PerfTable: # ...System:
OsType: LinuxOsRelease: 4.15.0Distribution: # ...MemoryTotal: 65870408SwapTotal: 3145724Hostname: node-0.sandbox.infosphere.emulab.netCpuCount: 32CpuOnlineCount: 4CpuSpeed: 1279CollectorType: cgroup_v2CollectorMetadata:
Cgroup: system.slice/docker-7762ff15c99a2d238f4d26c22b5eda5b97ebc03bd0a711693104dcb6f71fe411.scopeCgroupDriver: systemdPolledAt: 1641734740142271945InitializedAt: 1641734740157676566
---
read,pids.current,pids.max,cpu.stat/usage_usec,cpu.stat/system_usec,cpu.stat/user_usec,cpu.stat/nr_periods,cpu.stat/nr_throttled,cpu.stat/throttled_usec,memory.current,memory.high,memory.max,memory.stat/anon,memory.stat/file,memory.stat/kernel_stack,memory.stat/pagetables,memory.stat/percpu,memory.stat/sock,memory.stat/shmem,memory.stat/file_mapped,memory.stat/file_dirty,memory.stat/file_writeback,memory.stat/swapcached,memory.stat/inactive_anon,memory.stat/active_anon,memory.stat/inactive_file,memory.stat/active_file,memory.stat/unevictable,memory.stat/pgfault,memory.stat/pgmajfault,io.stat/rbytes,io.stat/wbytes,io.stat/rios,io.stat/wios,io.stat/dbytes,io.stat/dios1641734705052508079,1,28989,58688,40630,18057,0,0,0,5558272,max,max,405504,3514368,49152,0,0,0,0,2838528,0,0,0,270336,0,1486848,2027520,0,1650,0,3891200,0,58,0,0,0# ...More information about what each column represents can be found in the docs pages (for information about the columns ouputted when CollectorType: cgroup_v1, see this page instead).
---
Version: 1.4.0Provider: kubernetesMetadata:
Uid: 9f0b1893-15e7-442a-966a-b0d19a35fc1cName: kube-proxy-hsplgCreatedAt: "2020-03-29T04:32:35Z"Labels:
controller-revision-hash: c8bb659c5k8s-app: kube-proxypod-template-generation: "1"Namespace: kube-systemNodeName: node-0.sandbox.infosphere.emulab.netHostIp: 130.127.133.26Phase: RunningQosClass: BestEffortStartedAt: "2020-03-29T04:32:36Z"PerfTable: # ...System:
OsType: LinuxOsRelease: 4.15.0Distribution: # ...MemoryTotal: 65870408SwapTotal: 3145724Hostname: node-0.sandbox.infosphere.emulab.netCpuCount: 32CpuOnlineCount: 32CpuSpeed: 1198PolledAt: 1585470948008442929CollectorType: cgroup_v1CollectorMetadata:
Cgroup: /kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f0b1893_15e7_442a_966a_b0d19a35fc1c.sliceCgroupDriver: systemdInitializedAt: 1585470948030565581
---
read,pids.current,pids.max,cpu.usage.total,cpu.usage.system,cpu.usage.user,cpu.usage.percpu,cpu.stat.user,cpu.stat.system,cpu.throttling.periods,cpu.throttling.throttled.count,cpu.throttling.throttled.time,memory.usage.current,memory.usage.max,memory.limit.hard,memory.limit.soft,memory.failcnt,memory.hierarchical_limit.memory,memory.hierarchical_limit.memoryswap,memory.cache,memory.rss.all,memory.rss.huge,memory.mapped,memory.swap,memory.paged.in,memory.paged.out,memory.fault.total,memory.fault.major,memory.anon.inactive,memory.anon.active,memory.file.inactive,memory.file.active,memory.unevictable,blkio.time,blkio.sectors,blkio.service.bytes.read,blkio.service.bytes.write,blkio.service.bytes.sync,blkio.service.bytes.async,blkio.service.ios.read,blkio.service.ios.write,blkio.service.ios.sync,blkio.service.ios.async,blkio.service.time.read,blkio.service.time.write,blkio.service.time.sync,blkio.service.time.async,blkio.queued.read,blkio.queued.write,blkio.queued.sync,blkio.queued.async,blkio.wait.read,blkio.wait.write,blkio.wait.sync,blkio.wait.async,blkio.merged.read,blkio.merged.write,blkio.merged.sync,blkio.merged.async,blkio.throttle.service.bytes.read,blkio.throttle.service.bytes.write,blkio.throttle.service.bytes.sync,blkio.throttle.service.bytes.async,blkio.throttle.service.ios.read,blkio.throttle.service.ios.write,blkio.throttle.service.ios.sync,blkio.throttle.service.ios.async,blkio.bfq.service.bytes.read,blkio.bfq.service.bytes.write,blkio.bfq.service.bytes.sync,blkio.bfq.service.bytes.async,blkio.bfq.service.ios.read,blkio.bfq.service.ios.write,blkio.bfq.service.ios.sync,blkio.bfq.service.ios.async
1602390175053135973,18,4915,45675783181,0,45675783181,9719044209 12310201631 11027849186 12618688155 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0,2925,1668,0,0,0,2802823168,3667771392,9223372036854771712,9223372036854771712,0,9223372036854771712,,35323904,2754781184,0,28672,,6837817,6156636,6854722,0,0,2754711552,7380992,27942912,0,2273087370,1306336,0,668844032,662777856,6066176,0,331937,331753,184,0,68057100860,68011971780,45129080,0,0,0,0,0,222907407415,222860666999,46740416,0,32,0,32,0,668844032,662777856,6066176,0,331937,331753,184,,,,,,,,# ...Many of the specific details of collection can be controlled via the command line interface. At the moment, this includes collection/polling intervals and output directory. To view information on the available CLI options, run radvisor help:
$ radvisor helpradvisor 1.4.0Joseph Azevedo <joseph.az@gatech.edu>, Bhanu Garg <bgarg6@gatech.edu>Monitors container resource utilization with high granularity and low overheadUSAGE: radvisor [FLAGS] [OPTIONS] <SUBCOMMAND>FLAGS: -h, --help Prints help information -q, --quiet Whether to run in quiet mode (minimal output) -v, --verbose Whether to run in verbose mode (maximum output) -V, --version Prints version informationOPTIONS: -c, --color <color-mode> Color display mode for stdout/stderr output [default: auto]SUBCOMMANDS: help Prints this message or the help of the given subcommand(s) run Runs a collection thread that writes resource statistics to output CSV files$ radvisor run <provider>The main subcommand of rAdvisor is run, which additionally requires the target provider (Docker or Kubernetes) to use to discover collection targets. For example, to run rAdvisor and collect resource utilization statistics on Docker containers each 40ms, the following command would be used:
$ radvisor run docker -i 40msInitializing Docker API providerBeginning statistics collectionIdentified cgroupfs as cgroup driverAt the moment, rAdvisor only supports Linux (due to its heavy reliance on cgroups), though there is a tracking issue for extending its functionality to work with Window's own first-party containerization API, HCS: radvisor/issues/#3.
To build rAdvisor using Docker, run the following command (needs to have docker installed and running, and likely needs to be run as root):
$ sudo make
For the Docker build method, the Rust stable image is used (rust) to run a Docker container with the necessary toolchains pre-installed.
To build rAdvisor from source, Rust stable is used. We recommend using rustup to install the Rust toolchain.
Now, in the cloned repository root, run make compile to generate a release-grade binary at ./target/release/radvisor. This build process may take up to ten minutes.
$ make compilecargo build --release --bins \--target x86_64-unknown-linux-gnu Compiling libc v0.2.68 Compiling autocfg v1.0.0 Compiling cfg-if v0.1.10 ... Compiling shiplift v0.6.0 Compiling radvisor v1.4.0 (/home/jazev/dev/radvisor) Finished release [optimized] target(s) in 4m 52s
$ ./radvisor --versionradvisor 1.4.0This project is licensed under the GNU General Public License v3.0.
Additionally, this software was developed as a part of research work at the Georgia Institute of Technology, and may be subject to license restrictions.