This is a Kubernetes device plugin that registers virtual device resources (e.g. gpu-memory, gpu-cores) -- the resources HAMi tracks but the kubelet would normally ignore -- onto a node, without requiring real hardware. It lets you test the HAMi scheduler (scheduling policies, quotas, webhooks) on machines that have no GPU/NPU.
After deployment these resources show up under node.status.allocatable and node.status.capacity.
The mock plugin does not detect hardware. It does the following every ~30s:
node annotation: hami.io/node-<vendor>-register = [ {devmem, devcore, ...}, ... ] (1)
node capacity: <count resource> (e.g. nvidia.com/gpu) > 0 (2) <- health gate
| mock reads
v
registers into allocatable: <vendor>/...mem , <vendor>/...cores , ...
On a real cluster, (1) and (2) are produced by the real device plugin. In a mock-only (no hardware) environment you provide them yourself:
- (1) the
node-<vendor>-registerannotation describing the fake cards --kubectl annotate. - (2) the count extended resource (e.g.
nvidia.com/gpu) -- patched onto the nodestatus.
There is no auto-labeller in this repo, so (1) and (2) are manual today. Forgetting them is the usual cause of
device xxx is unhealthy/no allocation-- see issues #14 / #16.
- Kubernetes >= v1.18
- The
hami-scheduler-deviceConfigMap (the device config). If HAMi is installed it already exists; otherwise create it from device-configmap.yaml.
make deploy # = kubectl apply -f k8s-mock-rbac.yaml && kubectl apply -f k8s-mock-plugin.yaml# or manually:
kubectl apply -f k8s-mock-rbac.yaml
kubectl apply -f k8s-mock-plugin.yamlThe mock derives the registered resources from the annotation, not from the count resource. This trips people up, so to be explicit:
- Number of fake cards = the number of entries in the annotation array (not the count value).
- Registered
...-memory= sum ofdevmemover all entries. - Registered
...-cores/...-core= sum ofdevcoreover all entries. - The count extended resource is only a health gate: its value just needs to be
> 0. It does not affect the registered memory/cores. By convention it is set tocards x splits-per-card(e.g. Ascend2 x VDeviceCount(4) = 8), but1would work equally well for the memory/cores to appear.
Annotation entry fields:
| field | meaning |
|---|---|
id | unique device UUID (any string) |
devmem | per-card memory in MB -- summed into ...-memory |
devcore | per-card cores. NVIDIA/Hygon: summed into ...-cores (NVIDIA: percentage, 100 = a whole card). Ascend: ignored -- huawei.com/<chip>-core is percentage-based, registered as 100 per card. |
count | per-card split count (informational for the mock) |
type | device model string |
health | must be true to be counted |
index | card index 0,1,2,... (0 may be omitted) |
numa, mode | optional |
Worked example (Ascend, below): the annotation has 2 entries, each
devmem=32768. So...-memory = 2x32768 = 65536and...-core = 2x100 = 200(Ascend...-coreis percentage-based: a whole card is always 100). The count resource=8is a separate health-gate value and is unrelated to these numbers.
To mock one card you always need the three pieces: the vendor config block (in the hami-scheduler-device ConfigMap, gives the resource names), the count extended resource (passes the health gate), and the node-register annotation (describes the fake cards). Replace <node> below with your target node. Resource names follow the ConfigMap (HAMi defaults shown). See Understanding the values for how the numbers are derived.
- config block:
nvidia:| annotation:hami.io/node-nvidia-register(JSON) | count:nvidia.com/gpu - mock registers:
nvidia.com/gpumem,nvidia.com/gpucores,nvidia.com/gpumem-percentage
# (2) count resource: splitCount=10 -> one card advertises 10
kubectl patch node <node> --subresource=status --type=json \
-p '[{"op":"add","path":"/status/capacity/nvidia.com~1gpu","value":"10"}]'# (1) annotation: 1 x A100-80G (devmem=81920 MB, devcore=100)
kubectl annotate node <node> \
'hami.io/node-nvidia-register=[{"id":"GPU-MOCK-0","count":10,"devmem":81920,"devcore":100,"type":"NVIDIA-A100-SXM4-80GB","health":true,"numa":0,"mode":"hami-core"}]'# verify (~30s later)
kubectl get node <node> -o json | jq '.status.allocatable|with_entries(select(.key|test("nvidia.com")))'# expect: nvidia.com/gpu=10, nvidia.com/gpumem=81920, nvidia.com/gpucores=100, nvidia.com/gpumem-percentage=100- config block:
vnpus.configs(entry for910B4) | annotation:hami.io/node-register-Ascend910B4(JSON) | count:huawei.com/Ascend910B4 - mock registers:
huawei.com/Ascend910B4-memory, and -- only when the node runs inhami-vnpu-coresoft mode (see below) --huawei.com/Ascend910B4-core
# (2) count resource: 2 cards x VDeviceCount(4) = 8
kubectl patch node <node> --subresource=status --type=json \
-p '[{"op":"add","path":"/status/capacity/huawei.com~1Ascend910B4","value":"8"}]'# (1) annotation: 2 x 910B4 (devmem=32768 MB, devcore=20), matching the real ascend-device-plugin report
kubectl annotate node <node> \
'hami.io/node-register-Ascend910B4=[{"id":"MOCK-0","count":4,"devmem":32768,"devcore":20,"type":"Ascend910B4","health":true},{"id":"MOCK-1","index":1,"count":4,"devmem":32768,"devcore":20,"type":"Ascend910B4","health":true}]'# (1b) soft mode: mark the node as hami-vnpu-core so the -core resource is reported.# Either set vnpus.hamiVnpuCore: true globally in the ConfigMap, or annotate this node# (the node annotation overrides the global setting, same precedence as the real plugin):
kubectl annotate node <node> hami-vnpu-core=true
# verify
kubectl get node <node> -o json | jq '.status.allocatable|with_entries(select(.key|test("Ascend910B4")))'# expect: huawei.com/Ascend910B4-memory=65536, huawei.com/Ascend910B4-core=200 (2 cards x 100)The Ascend -core resource is percentage-based: each physical card contributes 100 (a whole card), independent of the annotation's devcore. HAMi caps a core request at 100 and, in hami-vnpu-core soft mode, treats a card's total core as 100.
When is -core reported? Only on a node that runs in hami-vnpu-core (soft-partition) mode, mirroring the real ascend-device-plugin and the HAMi scheduler. Mode is decided by: the node's hami-vnpu-core annotation if present, otherwise the global vnpus.hamiVnpuCore in the ConfigMap (true/false, default false). On a hard/template node (hami-vnpu-core off) the HAMi scheduler filters out any pod that requests -core (ModeNotFit), so the mock does not register -core there. There is no ascend-device-plugin in a mock environment, so nothing writes the hami-vnpu-core node annotation automatically -- set it yourself (like the node-register annotation and the count resource).
- config block:
hygon:| annotation:hami.io/node-dcu-register(CSV, not JSON) | count:hygon.com/dcunum - mock registers:
hygon.com/dcumem, and -- when the config setsresourceCoreName--hygon.com/dcucores(sum of per-carddevcore)
# (2) count resource: 2 DCUs
kubectl patch node <node> --subresource=status --type=json \
-p '[{"op":"add","path":"/status/capacity/hygon.com~1dcunum","value":"2"}]'# (1) annotation: CSV form is "id,count,devmem,devcore,type,numa,health,index,mode:" per card
kubectl annotate node <node> \
'hami.io/node-dcu-register=DCU-MOCK-0,2,16384,100,Z100L,0,true,0,hami-core:DCU-MOCK-1,2,16384,100,Z100L,1,true,1,hami-core:'# verify
kubectl get node <node> -o json | jq '.status.allocatable|with_entries(select(.key|test("hygon.com")))'# expect: hygon.com/dcumem=32768, hygon.com/dcucores=200 (2 cards x devcore 100)The Ascend vnpus config has two layouts and the plugin accepts both:
# New (HAMi >= v2.9.0): nested objectvnpus:
hamiVnpuCore: true # node runs in soft mode -> the -core resource is reportedconfigs:
- chipName: 910B4resourceCoreName: huawei.com/Ascend910B4-core # name of the -core resource...# Legacy (older / downstream): a bare listvnpus:
- chipName: 910B4...The new nested format is tried first; if that fails it falls back to the legacy flat list; if neither matches an error is returned. This lets clusters that have not upgraded their ConfigMap keep working.
| Devices | Mocking Resources |
|---|---|
| Nvidia GPU | nvidia.com/gpumem, nvidia.com/gpumem-percentage, nvidia.com/gpucores |
| Hygon DCU | hygon.com/dcumem, hygon.com/dcucores (when resourceCoreName is set) |
| Ascend | huawei.com/Ascend{chip}-memory, huawei.com/Ascend{chip}-core (when resourceCoreName is set and the node is in hami-vnpu-core mode) |
Note: If the counted memory is too large (e.g. > 120GB) it may display as 0. Set memoryFactor in the hami-scheduler-device ConfigMap (default 1).
make build # build the binary into bin/
make test# run unit tests
make docker-build # build the image (override: IMG=myrepo/mock:tag)
make help# list all targets