Skip to content

fix(nvidia): honest memory, C2C link and fan reporting on SoCs - #558

Open
Archmonger wants to merge 1 commit into
Syllo:masterfrom
Archmonger:fix/gb10-uma-and-na-metrics
Open

Archmonger wants to merge 1 commit into
Syllo:masterfrom
Archmonger:fix/gb10-uma-and-na-metrics

Conversation

@Archmonger

Copy link
Copy Markdown
Contributor

Supersedes #523 (which I also opened, closed as superseded by #542 — its two ideas are now upstream; this picks up what was left) and addresses the parts of #426 that remain after 3.4.0.

1. Memory accounting on unified-memory SoCs (the "incorrect values" part)

On GB10 the memory pool is shared with the host, but used was reconstructed from the process list while total came from MemTotal. Those are different counters, so the header advertised headroom the machine does not have. Measured on a DGX Spark with ~106 GiB allocated to GPU processes:

total used actually allocatable
before 121.69 GiB 106.0 GiB 5.8 GiB
after 121.69 GiB 115.8 GiB 5.8 GiB

This is #426's last complaint: @raziel2001au reported that 3.3.0 was the last release showing correct headroom and that current releases give "the illusion that we have a lot of VRAM free".

The device has one pool, so used now means one thing across the whole header: memory not available to a new allocation, host allocations included. total = MemTotal, free = MemAvailable, used = total - free, and mem_util_rate follows, so the graph is consistent with the bar. Per-process GPU allocations are still listed separately and unchanged.

This changes the semantics documented in the README from #542 (used = visible process allocations). I think the whole-pool reading is the defensible one — total is the system pool, so a used counting only GPU processes leaves total - used meaningless, and on these machines host processes genuinely consume the memory the GPU would otherwise use — but flagging it since it is your call which definition ships. The generic process-list fallback in gpuinfo_fix_dynamic_info_from_process_info is untouched and still serves backends that report only a total.

2. PCIe GEN 1@ 1x was fabricated

GB10 is attached over NVLink-C2C, but NVML answers the PCIe queries with placeholders that return NVML_SUCCESS: Max Gen 1 / Max Width 16x, Current Gen 1 / Width 1x. The dynamic path already suppressed these via nvmlDeviceGetC2cModeInfoV (#542), but the static max gen/width were still exported as valid, landing in snapshots and in pcie_max_bandwidth_kbs.

The link count is read once from NVML_FI_DEV_C2C_LINK_COUNT (field 170, = 8 on GB10), with the existing C2C-mode boolean as a fallback for drivers that expose only that. When present, the header shows C2C LINK 8x in place of the PCIe link, styled like it replaced, and the placeholder max gen/width are left unknown. NVML exposes no byte-throughput counter for C2C, so RX/TX are not shown rather than shown as N/A. A device with no C2C links keeps full PCIe reporting.

3. FAN N/A

The driver exposes no fan on this platform: nvmlDeviceGetFanSpeed NOT_SUPPORTED, _v2 INVALID_ARGUMENT, GetFanSpeedRPM fails, GetNumFans returns 0, and the box has no fan or PWM sensor in hwmon/sensors//sys/class/pwm (all its thermal zones are acpitz). nvidia-smi prints Fan: N/A too.

A memory-shared SoC has no add-in board, so it is now classified as integrated and the header shows CPU-FAN — the existing path for integrated devices, which have the same situation — instead of claiming a missing sensor.

Remaining N/A

Memory clocks, power limit (and therefore effective load), ECC, and PCIe throughput stay unavailable. I probed all three sources before deciding to leave them: NVML (nvmlDeviceGetFieldValues answers only util/energy/replay/C2C/throttle fields; no memory clock, no power limit, no ECC), NVAPI (GetAllClockFrequencies has no memory domain, tach and cooler queries fail), and sysfs (no devfreq, no relevant clk). None of them can be estimated, so they remain N/A.

Tests

nvidiaTests gains c2c_link_count (link count suppresses the placeholder PCIe link, zero keeps it) and updates uma_version/uma_idle for the new memory contract. ctest: 29/29 locally, 13/13 on aarch64. Verified against the patched build and an AppImage on two DGX Spark machines under load.

Header on GB10 under load (before → after)
before: Device 0 [NVIDIA GB10] PCIe GEN 1@ 1x RX: N/A TX: N/A ... MEM N/A MHz ... FAN N/A ... MEM[| 106.0Gi/121.69Gi]
after:  Device 0 [NVIDIA GB10] C2C LINK 8x GPU 2184MHz MEM N/A MHz TEMP 67C CPU-FAN POW 45 W GPU[95%] MEM[| 115.831Gi/121.686Gi]

Closes #426

On GB10 (DGX Spark) class devices the memory pool is shared with the
host, but the header paired a process-sum "used" with a MemTotal
"total", advertising headroom the machine does not have: host
processes consume the same memory and no longer leave it for the GPU.
Report the pool as a whole instead — total from MemTotal, free from
MemAvailable, used as their difference — so total, free and the used
amount stay consistent and the bar shows the memory a new allocation
can actually claim. This restores the usable-headroom reading that
3.3.0 gave, at the correct total.

The generic process-list fallback no longer zeroes the used amount of
shared-memory devices; it still reconstructs used memory for backends
that only report a total.

GB10 attaches to the CPU over NVLink-C2C, yet NVML answers the PCIe
queries with placeholders ("Max Gen 1 / Max Width 16x", "Current Gen 1
/ Width 1x"), and the driver exposes no fan: GetFanSpeed and
GetFanSpeedRPM fail, GetNumFans is zero, and the platform has no fan
sensor. Read the C2C link count (NVML_FI_DEV_C2C_LINK_COUNT) once when
probing static info: show it as "C2C LINK <n>x" in place of the PCIe link,
suppress the placeholder max generation and width, and classify
memory-shared devices as integrated so the header shows "CPU-FAN"
instead of a GPU fan speed it cannot have. A device with no C2C links
keeps full PCIe reporting.

Remaining N/A fields (memory clocks, power limit, ECC, PCIe
throughput) are unsupported by the driver on this platform — NVML,
NVAPI and sysfs were all checked — and stay unknown rather than
estimated.
@Archmonger
Archmonger force-pushed the fix/gb10-uma-and-na-metrics branch from 2515e4e to 1048c2e Compare November 15, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug/Support] NVIDIA GB10 (Grace Blackwell) Reporting Issues (PCIe, Fan, Mem)

1 participant