Repository navigation
fix(nvidia): honest memory, C2C link and fan reporting on SoCs - #558
Open
Archmonger wants to merge 1 commit into
Open
Archmonger wants to merge 1 commit into
Archmonger wants to merge 1 commit into
Conversation
Archmonger
force-pushed
the
fix/gb10-uma-and-na-metrics
branch
from
November 15, 2026 08:16
81201cf to
2515e4e
Compare
On GB10 (DGX Spark) class devices the memory pool is shared with the
host, but the header paired a process-sum "used" with a MemTotal
"total", advertising headroom the machine does not have: host
processes consume the same memory and no longer leave it for the GPU.
Report the pool as a whole instead — total from MemTotal, free from
MemAvailable, used as their difference — so total, free and the used
amount stay consistent and the bar shows the memory a new allocation
can actually claim. This restores the usable-headroom reading that
3.3.0 gave, at the correct total.
The generic process-list fallback no longer zeroes the used amount of
shared-memory devices; it still reconstructs used memory for backends
that only report a total.
GB10 attaches to the CPU over NVLink-C2C, yet NVML answers the PCIe
queries with placeholders ("Max Gen 1 / Max Width 16x", "Current Gen 1
/ Width 1x"), and the driver exposes no fan: GetFanSpeed and
GetFanSpeedRPM fail, GetNumFans is zero, and the platform has no fan
sensor. Read the C2C link count (NVML_FI_DEV_C2C_LINK_COUNT) once when
probing static info: show it as "C2C LINK <n>x" in place of the PCIe link,
suppress the placeholder max generation and width, and classify
memory-shared devices as integrated so the header shows "CPU-FAN"
instead of a GPU fan speed it cannot have. A device with no C2C links
keeps full PCIe reporting.
Remaining N/A fields (memory clocks, power limit, ECC, PCIe
throughput) are unsupported by the driver on this platform — NVML,
NVAPI and sysfs were all checked — and stay unknown rather than
estimated.
Archmonger
force-pushed
the
fix/gb10-uma-and-na-metrics
branch
from
November 15, 2026 08:20
2515e4e to
1048c2e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #523 (which I also opened, closed as superseded by #542 — its two ideas are now upstream; this picks up what was left) and addresses the parts of #426 that remain after 3.4.0.
1. Memory accounting on unified-memory SoCs (the "incorrect values" part)
On GB10 the memory pool is shared with the host, but
usedwas reconstructed from the process list whiletotalcame fromMemTotal. Those are different counters, so the header advertised headroom the machine does not have. Measured on a DGX Spark with ~106 GiB allocated to GPU processes:This is #426's last complaint: @raziel2001au reported that 3.3.0 was the last release showing correct headroom and that current releases give "the illusion that we have a lot of VRAM free".
The device has one pool, so
usednow means one thing across the whole header: memory not available to a new allocation, host allocations included.total = MemTotal,free = MemAvailable,used = total - free, andmem_util_ratefollows, so the graph is consistent with the bar. Per-process GPU allocations are still listed separately and unchanged.This changes the semantics documented in the README from #542 (
used= visible process allocations). I think the whole-pool reading is the defensible one —totalis the system pool, so ausedcounting only GPU processes leavestotal - usedmeaningless, and on these machines host processes genuinely consume the memory the GPU would otherwise use — but flagging it since it is your call which definition ships. The generic process-list fallback ingpuinfo_fix_dynamic_info_from_process_infois untouched and still serves backends that report only a total.2.
PCIe GEN 1@ 1xwas fabricatedGB10 is attached over NVLink-C2C, but NVML answers the PCIe queries with placeholders that return
NVML_SUCCESS: Max Gen 1 / Max Width 16x, Current Gen 1 / Width 1x. The dynamic path already suppressed these vianvmlDeviceGetC2cModeInfoV(#542), but the static max gen/width were still exported as valid, landing in snapshots and inpcie_max_bandwidth_kbs.The link count is read once from
NVML_FI_DEV_C2C_LINK_COUNT(field 170, = 8 on GB10), with the existing C2C-mode boolean as a fallback for drivers that expose only that. When present, the header showsC2C LINK 8xin place of the PCIe link, styled like it replaced, and the placeholder max gen/width are left unknown. NVML exposes no byte-throughput counter for C2C, so RX/TX are not shown rather than shown asN/A. A device with no C2C links keeps full PCIe reporting.3.
FAN N/AThe driver exposes no fan on this platform:
nvmlDeviceGetFanSpeedNOT_SUPPORTED,_v2INVALID_ARGUMENT,GetFanSpeedRPMfails,GetNumFansreturns 0, and the box has no fan or PWM sensor in hwmon/sensors//sys/class/pwm(all its thermal zones areacpitz).nvidia-smiprintsFan: N/Atoo.A memory-shared SoC has no add-in board, so it is now classified as integrated and the header shows
CPU-FAN— the existing path for integrated devices, which have the same situation — instead of claiming a missing sensor.Remaining
N/AMemory clocks, power limit (and therefore effective load), ECC, and PCIe throughput stay unavailable. I probed all three sources before deciding to leave them: NVML (
nvmlDeviceGetFieldValuesanswers only util/energy/replay/C2C/throttle fields; no memory clock, no power limit, no ECC), NVAPI (GetAllClockFrequencieshas no memory domain, tach and cooler queries fail), and sysfs (no devfreq, no relevant clk). None of them can be estimated, so they remainN/A.Tests
nvidiaTestsgainsc2c_link_count(link count suppresses the placeholder PCIe link, zero keeps it) and updatesuma_version/uma_idlefor the new memory contract.ctest: 29/29 locally, 13/13 on aarch64. Verified against the patched build and an AppImage on two DGX Spark machines under load.Header on GB10 under load (before → after)
Closes #426