os.cpu_count() should return a count assigned to a container or that the process is restricted to

[keirlawson]

BPO	36054
Nosy	@vstinner, @giampaolo, @tiran, @jab, @methane, @matrixise, @corona10, @Zheaoli, @mcnelsonphd, @NargiT, @Redeyed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-02-20.16:56:22.384>
labels = ['interpreter-core', '3.8', '3.7', 'library', 'performance']
title = 'On Linux, os.count() should read cgroup cpu.shares and cpu.cfs (CPU count inside docker container)'
updated_at = <Date 2021-10-11.14:25:34.762>
user = 'https://github.com/keirlawson'

bugs.python.org fields:

activity = <Date 2021-10-11.14:25:34.762>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2019-02-20.16:56:22.384>
creator = 'keirlawson'
dependencies = []
files = []
hgrepos = []
issue_num = 36054
keywords = []
message_count = 24.0
messages = ['336117', '336126', '336146', '336148', '336149', '338724', '338778', '338780', '338783', '339401', '339404', '339429', '339439', '353690', '364894', '364898', '364901', '364902', '364916', '365071', '365075', '366811', '403622', '403654']
nosy_count = 12.0
nosy_names = ['vstinner', 'giampaolo.rodola', 'christian.heimes', 'jab', 'methane', 'matrixise', 'corona10', 'Manjusaka', 'mcnelsonphd', 'galen', 'nargit', 'RedEyed']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue36054'
versions = ['Python 3.7', 'Python 3.8']

[keirlawson]

There appears to be no way to detect the number of CPUs allotted to a Python program within a docker container. With the following script:

import os

print("os.cpu_count(): " + str(os.cpu_count()))
print("len(os.sched_getaffinity(0)): " + str(len(os.sched_getaffinity(0))))

when run in a container (from an Ubuntu 18.04 host) I get:

docker run -v "$PWD":/src/ -w /src/ --cpus=1 python:3.7 python detect_cpus.py
os.cpu_count(): 4
len(os.sched_getaffinity(0)): 4

Recent vesions of Java are able to correctly detect the CPU allocation:

docker run -it --cpus 1 openjdk:10-jdk
Feb 20, 2019 4:20:29 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
| Welcome to JShell -- Version 10.0.2
| For an introduction type: /help intro

jshell> Runtime.getRuntime().availableProcessors()
$1 ==> 1

[matrixise]

I would like to work on this issue.

[matrixise]

so, I have also tested with the last docker image of golang.

docker run --rm --cpus 1 -it golang /bin/bash

here is the code of golang:

package main

import "fmt"
import "runtime"

func main() {
cores := runtime.NumCPU()
fmt.Printf("This machine has %d CPU cores.\n", cores)
}

Here is the output

./demo
This machine has 4 CPU cores.

When I try with grep on /proc/cpuinfo
I get this result

grep processor /proc/cpuinfo -c
4

I will test with openjdk because it's related to Java and see if I can get the result of 1

[matrixise]

ok, I didn't see your test with openjdk:10, sorry

[keirlawson]

I believe this is related to this ticket: https://bugs.python.org/issue26692

Looking at Java's implementation it seems like they are checking if cgroups are enabled via /proc/self/cgroup and then if it is parsing the cgroup information out of the filesystem.

[matrixise]

I am really sorry but I thought to work on this issue but it's not the case. Feel free to submit a PR.

[Zheaoli]

I think that I may work on a PR for this issue. Is there anybody has worked on it ？

[matrixise]

Hi Manjusaka,

Could you explain your solution, because I have read the code of
openjdk, (C++) and I am going to be honnest, it was not complex but not
really clear.

Also, if you need my help for the review or for the construction of your
PR, I can help you.

Have a nice day,

Stéphane

[Zheaoli]

Hi Stéphane

Thanks a lot!

In my opinion, I would like to make an independent library that name is cgroups. For ease of use and compatibility, I think it's better than combining code with the os module.

Thanks for you working!

Manjusaka

[Zheaoli]

Hi Stéphane:

I have checked the JVM implantation about container improvements. I confirm that maybe we need a new Libary for container environment. I don't think that combine it into the os module is a good idea. I will make a PR during this week.