Skip to content

[Bug/Support] NVIDIA GB10 (Grace Blackwell) Reporting Issues (PCIe, Fan, Mem) #426

Open

Description

@chromedot

Generated with my good buddy Antigravity + Gemini 3 Pro High

Description
I am running nvtop on the new NVIDIA GB10 (Grace Blackwell) platform (Dell Pro Max Workstation Gb10 = Nvidia DGX Spark). While it correctly detects the GPU name and utilization, several metrics are reported incorrectly or as N/A, likely due to the SoC nature of the architecture (NVLink-C2C instead of traditional PCIe).

System Information
Hardware: NVIDIA GB10 (Grace Blackwell Superchip)
Architecture: ARM64 (aarch64)
OS: Ubuntu 24.04.3 LTS
NVIDIA Driver: 580.95.05
CUDA Version: 13.0
Kernel: Linux 6.8.0-49-generic (aarch64)

Observed Issues
PCIe Misreporting:
Reported as: PCIe GEN 1@ 1x
Context: This is an SoC (System on Chip) where the GPU is connected via NVLink-C2C to the Grace CPU, not a standard PCIe slot. It seems nvtop is falling back to a default/placeholder value.
Memory Usage (Header):
Reported as: MEM[ N/A] in the bar gauge.
Context: nvidia-smi also reports "Not Supported" for memory usage on this platform (Unified Memory architecture). However, the process list at the bottom of nvtop does correctly show memory usage (e.g., 63931MiB for my python process). It would be great if the header gauge could aggregate the process memory or use the same source as the process list.
Fan Speed:
Reported as: FAN N/A%
Context: Likely managed by the chassis controller or SoC thermal subsystem, not exposed via standard NVML fan APIs for discrete cards.
Memory Clock:
Reported as: MEM N/A MHz
Working Metrics
GPU Utilization: Correctly shows load (e.g., 96%).
Power Usage: Correctly shows wattage (e.g., 58 W).
Process List: Correctly identifies processes and their per-process memory usage.
Additional Context
The GB10 uses the Unified Memory architecture (LPDDR5X) shared between CPU and GPU. The standard NVML queries for "FB Memory" might need to be adjusted or replaced with queries for the unified memory pool on this architecture.

Output of nvidia-smi for reference:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A |
| N/A 32C P8 3W / N/A | Not Supported | 0% Default |
+-----------------------------------------+------------------------+----------------------+
I have attached a screenshot of nvtop running under load to demonstrate the issue.

Screenshot

Image

Activity

  1. mhillebrand commented on Jan 7, 2026

    @mhillebrand

    For the RTX 6000 Pro, it's showing my 256GB of RAM, not my 96 GB of GPU VRAM. Also the full name of my GPU is NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, which is so long that I have to increase the width of my terminal window by 50% to see the erroneous VRAM stats. Ugh.

    Image
  2. avirtuos commented on Mar 5, 2026

    @avirtuos

    Same issue for me. My DGX Spark is running the lastest updates from nvidia and still no accurate vram usage in nvtop

  3. raziel2001au commented on Mar 16, 2026

    @raziel2001au

    They haven't updated the version of nvtop available via apt on the DGX Spark, but DGX Spark support has been broken since nvtop 3.3.1, see the issue I raised here: #449

    If you don't mind running an older version of nvtop, which seems to be the only option at the moment, you can use the one I compiled here: https://forums.developer.nvidia.com/t/nvtop-with-dgx-spark-unified-memory-support

  4. robolamp commented on Mar 30, 2026

    @robolamp

    Same issue for me, thanks @raziel2001au for instruction

  5. Archmonger commented on Oct 4, 2026

    @Archmonger
    Contributor

    AI generated comment (by a locally hosted LLM), but was human-verified.

    Two DGX Spark nodes here (driver 580.178.04, aarch64). Root-caused both header fields; PR with a fix at #523.

    Memory N/A — the driver has no versioned memory ABI, so nvmlDeviceGetMemoryInfo_v2 answers NVML_ERROR_FUNCTION_NOT_FOUND (13). The UMA probe only treats NOT_SUPPORTED as "shares memory with the host", so this falls through to else if and leaves memory_shared_with_host false; the dedicated-FB path then fails too and memory renders N/A while the process list already accounts ~100 GiB for the same GPU. Treating 13 as "not probed" makes the v1 query run, which answers NOT_SUPPORTED and selects the existing /proc/meminfo path → MEM[100.241Gi/121.686Gi] (MemTotal + per-process NVML).

    PCIe GEN 1@ 1x — gating on nvmlDeviceGetBusType isn't enough: GB10 reports NVML_BUS_TYPE_PCIE (2) despite the on-die C2C attach, so the link queries return NVML_SUCCESS with placeholders (Current 1 vs Device Max 5 / Host Max 5; Width Current 1x vs Max 16x). It isn't a real degraded link — the root port has no maintained link state:

    000f:00:00.0 PCI bridge  LnkCap: Speed 32GT/s, Width x16   LnkSta: Speed unknown, Width x0
    000f:01:00.0 VGA         LnkCap: Speed 2.5GT/s, Width x16  LnkSta: 2.5GT/s, Width x1 (downgraded)
    dmesg: 0.000 Gb/s available PCIe bandwidth, limited by Unknown x0 link at 000f:00:00.0
    

    nvmlDeviceGetC2cModeInfoV already exists in nvml.h and returns isC2cEnabled = 1 here, so it's the direct signal. Suppressed the link on that (a failed probe is not read as a C2C attach, so discrete GPUs keep their readout — checked against a real RTX 2000 Ada, where the call returns NOT_SUPPORTED).

    RX/TX stay N/A and that looks unfixable client-side: nvmlDeviceGetPcieThroughput and every candidate field value (138/139, 252-254 NVML_FI_DEV_C2C_LINK_ERROR_*, 14, 1011) return NOT_SUPPORTED, matching nvidia-smi dmon printing -.

  6. Syllo commented on Nov 14, 2026

    @Syllo
    Owner

    Several GB10 fixes have landed since this report and are included in the latest 3.4.0 release:

    Could you retest with 3.4.0 (or current master) and let us know whether the memory and PCIe problems are resolved? If the fan reading is still incorrect, please include the relevant sensors/sysfs output so we can keep this open specifically for that.

  7. Archmonger commented on Nov 14, 2026

    @Archmonger
    Contributor

    Using the 3.4.0 appimage, there are still a few N/A values shown

    image
    mark@spark1:~$ sensors
    mlx5-pci-0100
    Adapter: PCI adapter
    asic:         +54.0°C  (crit = +105.0°C, highest = +77.0°C)
    
    mt7925_phy0-pci-90100
    Adapter: PCI adapter
    temp1:        +43.0°C  
    
    acpitz_6-acpi-0
    Adapter: ACPI interface
    temp1:        +49.8°C  
    
    mlx5-pci-20100
    Adapter: PCI adapter
    asic:         +53.0°C  (crit = +105.0°C, highest = +77.0°C)
    
    acpitz_4-acpi-0
    Adapter: ACPI interface
    temp1:        +49.8°C  
    
    acpitz_2-acpi-0
    Adapter: ACPI interface
    temp1:        +49.7°C  
    
    acpitz_0-acpi-0
    Adapter: ACPI interface
    temp1:        +49.8°C  
    
    mlx5-pci-0101
    Adapter: PCI adapter
    asic:         +53.0°C  (crit = +105.0°C, highest = +77.0°C)
    
    nvme-pci-40100
    Adapter: PCI adapter
    Composite:    +44.9°C  (low  = -273.1°C, high = +82.8°C)
                           (crit = +84.8°C)
    Sensor 1:     +43.9°C  (low  = -273.1°C, high = +65261.8°C)
    Sensor 2:     +44.9°C  (low  = -273.1°C, high = +65261.8°C)
    
    mlx5-pci-20101
    Adapter: PCI adapter
    asic:         +53.0°C  (crit = +105.0°C, highest = +77.0°C)
    
    acpitz_5-acpi-0
    Adapter: ACPI interface
    temp1:        +49.8°C  
    
    acpitz_3-acpi-0
    Adapter: ACPI interface
    temp1:        +49.8°C  
    
    acpitz_1-acpi-0
    Adapter: ACPI interface
    temp1:        +48.8°C  
    
    
  8. raziel2001au commented on Nov 15, 2026

    @raziel2001au

    I haven't tried it, but just looking at the total GPU memory from that screenshot, it also seems wrong... 121.6GB is the maximum theoretical VRAM assuming that nothing has been allocated to CPU processes. The actual total VRAM should be smaller than that, since if you allocated over 4GB to CPU processes, which is what that screenshot shows, then it should display around 117GB as the GPU's total VRAM. Keep in mind there is a single memory pool, if memory is allocated by CPU processes, it should reduce the total VRAM the system has access to, which I'm not seeing in this screenshot. From what I've seen, version 3.3.0 is the one and only release that has worked correctly on the DGX Spark, I'll try 3.4.0 out myself later, but on the surface this doesn't look right.

  9. raziel2001au commented on Nov 15, 2026

    @raziel2001au

    Okay, I compiled 3.4.0 myself and can confirm that it's not correct, below I've got 3.3.0 running on the left, and 3.4.0 running on the right so you can see the differences clearly.

    As you will note, the total VRAM is displayed correctly in 3.3.0, but not in 3.4.0 on the right:
    Screenshot 2026-11-15 at 3 06 06 pm

    This creates the illusion that we have a lot of VRAM free, when we really don't, as almost all of the VRAM displayed in the total has already been allocated to CPU processes. So the view on the left (3.3.0) correctly shows that we've used 112GB out of 114GB of VRAM so we're close to hitting the ceiling. Looking at 3.4.0 on the right, it incorrectly indicates that we've only used 112GB out of 121.6GB of VRAM, but we don't actually have 121.6GB of VRAM, as some of that memory has been lost to CPU processes.

  10. Archmonger commented on Nov 15, 2026

    @Archmonger
    Contributor

    I've made a follow-up PR to fix all possible GB10 issues in #558

    There are some firmware-level limitations that have no viable solution, and thus remained N/A: Memory clocks, power limit (and therefore effective load), ECC, and PCIe throughput.

    image
  11. raziel2001au commented on Nov 15, 2026

    @raziel2001au

    In your screenshot, the total VRAM is still wrong.

    EDIT: Okay, I can see what you did... you increased the 'used' amount instead of decreasing the amount of VRAM available. The issue with that approach is that it hides issues in software. For example, if a program allocates more in system ram than it should, then you can't see that in nvtop. The way it should work is that when you allocate memory on the CPU, it should decrease the total amount of VRAM available. Refer to my screenshot on the left in: #426 (comment)

  12. Archmonger commented on Nov 15, 2026

    @Archmonger
    Contributor

    I took a different approach compared to your suggestion to VRAM calculation. Due to the unified memory architecture, I've effectively emulated the equivalent memory readout from btop/nvidia spark dashboard. More specifically, the top-most memory readout is the combined CPU+GPU memory usage, which is what an end-user would actually care about to help them determine how much more "VRAM" they have remaining. Hence why Nvidia and BTOP show metrics this way.

    In my opinion, the UX would feel awkward for the "max memory" to be dynamically changing around alongside "used memory".

    Will defer the final decision to the repo maintainer.

  13. raziel2001au commented on Nov 15, 2026

    @raziel2001au

    That's not useful though as you can't see how much of your RAM is actually available as VRAM. So if all you can see is that in total you've used X amount of memory, then you can't tell if there's something in system ram that shouldn't be there. It completely hides how much actual VRAM you have access to. I've used this in the past to identify issues in software, as you can quickly tell there's a problem when your available VRAM drops to 70GB for example, as it suggests a model was loaded and then not unloaded after uploading it to VRAM, so you basically have multiple copies of the same model sitting in RAM. With the current approach, you lose that insight completely. The approach used for 3.3.0 feels correct, as it shows the actual VRAM available once taking other things in memory into account.

  14. Archmonger commented on Nov 15, 2026

    @Archmonger
    Contributor

    Easy to change the printout, but will let @Syllo make the decision on how he wants this information represented.

  15. raziel2001au commented on Nov 15, 2026

    @raziel2001au

    Good to hear it's an easy change. Let's see what @Syllo says.

    From my perspective, the way it was done in 3.3.0 is the correct way to do it, since nvtop is supposed to monitor the GPU, and more specifically: the VRAM of that GPU. This gives insights you don't get from tools like btop, which just monitors total system memory, which is different from monitoring the VRAM of the GPU, which is where I feel nvtop should be doing something different from btop, which is a different tool with a different purpose. nvtop should monitor GPU resources, and if you've allocated memory in a CPU process, it ceases to be a GPU resource in a unified memory system, nvtop should reflect that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions