[dot1x]

This adds a backend for Intel Gaudi (Habana Labs) HPUs. It is built on Linux by default through the new GAUDI_SUPPORT option.

The backend loads libhlml.so at runtime. HLML is the management library behind hl-smi and is installed with the habanalabs driver packages. Because the library is loaded at runtime, building nvtop does not need the Habana SDK headers. On machines without the habanalabs module, the backend bails out before calling dlopen.

 Device 0 [Gaudi2 HL-225] PCIe GEN 4@16x RX: 0.000 KiB/s TX: 6.000 KiB/s Device 1 [Gaudi2 HL-225] PCIe GEN 4@16x RX: 0.000 KiB/s TX: 0.000 KiB/s
 HPU 1650MHz HBM N/A MHz  TEMP  37°C   FAN N/A   POW  76 / 550 W         HPU 1650MHz HBM N/A MHz  TEMP  37°C   FAN N/A   POW  68 / 550 W
 HPU[                    0%(eff 0%)] MEM[|||||||||||||87.980Gi/96.000Gi] HPU[                    0%(eff 0%)] MEM[|||||||||||||87.980Gi/96.000Gi]
...
    PID USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
 399951 test   3  Compute   0%  89371MiB  91%   100%   3801MiB VLLM::Worker_TP0_EP0
 400320 test   1  Compute   0%  89324MiB  91%   100%   3813MiB VLLM::Worker_TP2_EP2

What is reported

• Static:
  • device name (Gaudi2 HL-225)
  • architecture
  • memory type (HBM, or DDR4 for Goya)
  • max PCIe link
  • shutdown/slowdown temperatures
• Dynamic:
  • AIP utilization
  • device memory
  • SoC clock and its max
  • chip temperature
  • power draw and limit
  • current PCIe link (from sysfs, since HLML returns "not supported" on Gaudi2)
  • PCIe rx/tx throughput
• Processes:
  • each process using a device
  • its device memory
  • its utilization

Gaudi devices are labeled HPU (gpu_processing_unit_hpu), like the existing PPU label for T-Head.

Design notes

• Background sampling. Several HLML calls go through the device firmware. hlml_device_get_pcie_throughput takes about 25 ms per call and per direction, and the utilization and power calls take 3 to 6 ms each. On an 8-card machine, querying in the UI loop added more than 0.5 s per refresh.
  • The dynamic info is therefore sampled by a background thread, and refresh_dynamic_info only copies the latest sample.
  • Sampling is driven by one-shot refresh requests, so the thread is idle when nvtop is not refreshing. A request made while a device is already being sampled schedules another round instead of being lost.
  • The first refresh samples synchronously, so the first frame and -s already have data. After that, only the sampler thread calls HLML, so the backend does not rely on libhlml being thread safe. If the thread cannot be created, the backend falls back to synchronous sampling.
• Temperature. hlml_device_get_temperature leaks memory on every call: about 260 bytes, from sensors_get_label inside libhlml, found with ASan. With 8 cards refreshing every second, that is about 180 MB a day.
  • HLML computes the chip temperature as the hottest hwmon channel mapped by its hardcoded per-ASIC ON_AIP table. The driver's /etc/sensors.d/hl_sensors.conf labels those channels On Chip ... and covers Gaudi 1, 2 and 3.
  • The backend parses that file once for the board's chip block (honoring ignore), then reads those hwmon channels directly. The number of selected channels is checked against HLML's per-ASIC map (Gaudi2: 8, Gaudi3 OAM/FPGA: 16, Gaudi3 HL-338: 12) so a partial or user-modified config falls back to HLML instead of reporting the maximum of the wrong set.
  • HLML is only used as a fallback when the config is missing or does not match.
• Processes. HLML cannot enumerate processes, so the backend does the same as hl-smi: it scans /proc/*/fd for the /dev/accel/accelN compute node.
  • Nodes are matched by st_rdev rather than by path, so devices remapped inside containers are attributed correctly.
  • The control node accel_controlD*, which monitoring tools open, is ignored.
  • /proc is scanned once per refresh, not once per device.
  • Gaudi allows a single compute context per device. The process memory is the value the framework publishes in /dev/shm/mem_usage_accelN. The process keeps that file open, so it is read through /proc/<pid>/fd, which also works across mount namespaces. If the framework does not publish a value, the process memory is N/A; the device-wide used memory is not attributed to the process because it includes the 768 MiB the device reserves.
  • The process utilization is the device utilization when a single process can be identified as the compute context: the only process holding the compute node, or the single process publishing the framework's memory when several processes inherited it after a fork.
• libhlml ABI. The output structures are transcribed from the 1.24 SDK because nvtop must build without the Habana headers. libhlml writes the size of its own structure into the caller's object, with no size negotiation, so the utilization and memory output structures have a 512-byte reserved tail. This absorbs ABI growth from a newer library instead of overwriting the stack. The transcribed prefixes are checked with compile-time offset and size assertions.

Shared code

Besides the CMake option, the README section and the new test, the shared code gains the gpu_processing_unit_hpu enum value (following the PPU precedent) and its name mapping. Device enumeration also rejects a second call instead of indexing the previous device array, matching the PPU backend.

Testing

Tested on 8x Gaudi2 HL-225, habanalabs driver 1.24.1, Ubuntu 24.04 (kernel 6.17), while vLLM was serving on all cards:

• Compared with hl-smi at the same time, nvtop shows the same per-process memory to the MiB, the same device memory, and the same power and temperature within the noise between two reads.
• The -s / -l snapshots include device_architecture: Gaudi2 and memory_type: HBM.
• About 550 refreshes under ThreadSanitizer (nvtop -l -d 1 for 60 s): no reports.
• About 550 refreshes under AddressSanitizer (same run): no errors. The only leak report is a fixed 26.8 KB allocated by hlml_init/hlml_shutdown inside libhlml, and it is the same after 15 s and after 60 s.
• nvtop uses about 2% CPU in total (main thread plus sampler) with the default 1 s refresh.
• No new warnings with -Wall -Wextra. The new files are formatted with the repository's .clang-format.

The new gaudiTests target runs in CI and covers the libhlml ABI growth padding, double enumeration, device-name composition, AIP sensor count validation, lm-sensors config parsing, PCI chip address conversion, /proc scanning with a fake tree (including container remapping), process memory/utilization attribution, and the sampler request helpers. It passes under AddressSanitizer/UBSan and ThreadSanitizer.

Known differences with hl-smi

• The power limit is the HLML enforced limit of the main 54 V supply, 550 W on HL-225. hl-smi's summary adds the 50 W of the auxiliary 12 V supply, which HLML does not expose.
• Goya is enumerated by HLML and reported with DDR4 memory, but it has not been validated on hardware.