Skip to content

testing.md

Latest commit

 

History

History
227 lines (184 loc) · 15.3 KB

testing.md

File metadata and controls

227 lines (184 loc) · 15.3 KB
Raw

Testing

Use this page for test principles and quick commands. The focused guides cover configuration, execution and authoring in detail.

Detailed guides

Guide Contents
Environments and hardware Dependency profiles, hardware markers and platform selection
CLI and logging reference Compile warm-up, command options, log levels and CLI conventions
Parallel execution Resource reuse, runtime isolation, device allocation and filtering
Test suites and unit tests Suite roles, C++/Python execution, build coverage and CTest NPU resources
Writing scene tests Scene template, sibling synchronization, output validation and manual cases
Adding a C++ unit test Directory contracts, target declarations, support archives and checks
Scene-test device-memory fixtures Persistent test fixtures, round state and result readback

Design Principles

Three axioms govern hardware classification across all test categories:

  1. sim = no hardware: a2a3sim, a5sim, and no --platform are all equivalent — no hardware required.
  2. a2a3 / a5 are distinct platforms: Tests may support one, both, or neither.
  3. requires_hardware has two levels:
    • requires_hardware (no argument) — needs any hardware (a2a3 or a5)
    • requires_hardware("a2a3") — needs specifically a2a3

These principles apply uniformly to ut-py (pytest markers), ut-cpp (ctest labels), and st (@scene_test(platforms=[...])).

Assertion principle: no wall-clock proxies

All CI correctness tests (ut-py/ut-cpp, simulation and onboard hardware) must establish the tested state and event ordering explicitly. Synchronize through handshakes, barriers, acknowledgements or observable state transitions, then assert the functional result. See the always-loaded CI correctness rule.

Do not use wall-clock magnitude, sleeps, calibrated spin counts or relative execution speed as a proxy for concurrency, ordering or blocking. For example, hold a predecessor at an observable gate while checking successor preparation, then release it; do not assume preparation finishes inside an artificial delay. Such assumptions change with machine load, clock frequency and host scheduling. A fault test must confirm that its injected event occurred in the intended state. Waiting longer for background interference, or relying on repeated CI runs to eventually encounter it, does not establish fault coverage; keep such probabilistic stress experiments in explicitly requested diagnostics.

Timeouts provide a finite failure backstop for event waits. Their expiry or non-expiry does not prove an unrelated correctness property. When wall-time is the subject under test (a timeout firing, a startup deadline being respected, a configured budget), asserting it is correct. Budget generously for CI variance, and check the mechanism's observable effect rather than relying only on a tight measured bound. Explicit performance benchmarks are separate from correctness tests. See #1496.

Quick Reference

# Python unit tests, no hardware (sim or github-hosted)
pytest tests/ut

# Python unit tests, a2a3 hardware
pytest tests/ut --platform a2a3

# C++ unit tests, no hardware
cmake -B tests/ut/cpp/build -S tests/ut/cpp && cmake --build tests/ut/cpp/build
ctest --test-dir tests/ut/cpp/build -LE requires_hardware --output-on-failure

# C++ unit tests, a2a3 hardware (only hw + a2a3-specific tests)
ctest --test-dir tests/ut/cpp/build -L "^requires_hardware(_a2a3)?$" --output-on-failure

# Scene tests (pytest, @scene_test classes)
pytest examples tests/st --platform a2a3sim       # A2/A3 simulation
pytest examples tests/st --platform a5sim         # A5 simulation
pytest examples tests/st -m "not sdma and not sdma_fault" --platform a2a3 --exclude-level 4          # hardware
pytest examples tests/st -m "not sdma and not sdma_fault" --platform a2a3 --exclude-level 4 --device 4-7  # hardware with device pool

# Compile the selected hardware batch without creating a Worker or using an NPU
python -m simpler_setup.tools.scene_test_compile examples tests/st \
    -m "not sdma and not sdma_fault" --platform a2a3 --exclude-level 4 --require-pto-isa --compile-workers 8

# SDMA demos and the fault-injection case use separate device tasks, as in CI.
# The fault case is selected by @pytest.mark.sdma_fault (issue #1425).
pytest examples tests/st -m "sdma and not sdma_fault" --platform a2a3 --device 4-5
pytest examples tests/st -m sdma_fault --platform a2a3 --device 4

# A5 runs the non-network1 corpus, including SDMA tests, on both host architectures
pytest examples tests/st --platform a5 --exclude-level 4 --device 0-7

# Single scene test (standalone)
python examples/a2a3/tensormap_and_ringbuffer/vector_example/test_vector_example.py -p a2a3sim --manual include

# Benchmark mode (100 rounds, skip golden comparison)
python examples/a2a3/tensormap_and_ringbuffer/vector_example/test_vector_example.py \
    -p a2a3 -d 0 --rounds 100 --skip-golden

# Profiling (single-round only; diagnostics are disabled with --rounds > 1)
python examples/a2a3/tensormap_and_ringbuffer/vector_example/test_vector_example.py \
    -p a2a3 --enable-chip-swimlane

# Args dump
python tests/st/a2a3/tensormap_and_ringbuffer/alternating_matmul_add/test_alternating_matmul_add.py \
    -p a2a3 -d 11 --dump-args

Test Organization

Three test categories:

Category Abbrev Location Runner Description
System tests st examples/, tests/st/ pytest (@scene_test) or standalone python test_*.py Full end-to-end cases (compile + run + validate)
Python unit tests ut-py tests/ut/ pytest Unit tests for nanobind-exposed and Python modules
C++ unit tests ut-cpp tests/ut/cpp/ ctest (GoogleTest) Unit tests for pure C++ modules

Choosing ut-py vs ut-cpp

If a module is exposed via nanobind (used by both C++ and Python), test in ut-py (tests/ut/). If a module is pure C++ with no Python binding, test in ut-cpp (tests/ut/cpp/).

Test Directory Structure

tests/
  conftest.py          # pytest configuration (markers, fixtures, parametrization)
  ut/                  # Python unit tests (ut-py)
    test_task_interface.py
    test_runtime_builder.py
    cpp/               # C++ unit tests (ut-cpp, GoogleTest)
  st/                  # Scene tests
    setup/             # Compilation toolchain (KernelCompiler, RuntimeBuilder, etc.)
    conftest.py        # ST-specific sys.path setup
    test_worker_api.py # L3 distributed worker tests
    a2a3/              # @scene_test classes organized by {arch}/{runtime}/{name}/
    a5/

examples/              # Small examples (sim + onboard)
  a2a3/
    tensormap_and_ringbuffer/
      vector_example/
        test_vector_example.py   # @scene_test class + __main__ entry
        kernels/
          orchestration/*.cpp
          aic/*.cpp               # optional
          aiv/*.cpp               # optional
  a5/...

conftest.py            # Root: --platform/--device options, ST fixtures

Related guides

Sanitizer builds (ASAN / UBSan / TSAN)

Opt-in -fsanitize instrumentation of host-compiled code via --sanitizer (build with SIMPLER_SANITIZER=..., run with the matching runtime preloaded). The design, load-bearing invariants, usage, and presets now live in their own page — see sanitizers.md. The nightly CI job is in ci.md; the scoping rationale (macOS / TSAN / LSan) is in investigations/2026-06-sanitizer-scope.md.

CI Pipeline

See ci.md for the full CI pipeline documentation, including the job matrix, runner constraints, and marker scheme.

Previous section links

Old links into this overview remain available here; follow the corresponding link for the full section.

Previous section Current location
Test environments Read section
Scene Test CLI Options Read section
Lock-free kernel warm-up Read section
pytest Read section
Standalone (test_*.py) Read section
Option Reference Read section
Log levels Read section
Integer layout (Python-aligned) Read section
Where each tier ends up Read section
Setting it Read section
CLI Design Principles Read section
Parallel Test Execution and Resource Reuse Read section
Quick examples Read section
Fail-fast (-x / --exitfirst) Read section
Device-count constraints Read section
--device vs --max-parallel (two separate knobs) Read section
Mixed L2 + L3 in one file Read section
Profiling under parallelism Read section
Dispatcher skip conditions (normal pytest runs) Read section
Hardware Classification Read section
Helper function Read section
ut-py: @pytest.mark.requires_hardware[(platform)] Read section
ut-cpp: ctest labels Read section
st: @scene_test(platforms=[...]) Read section
Test Types Read section
C++ Unit Tests (tests/ut/cpp/) Read section
How many times a case is built Read section
Where the stand-ins come from Read section
Python Unit Tests (tests/ut/) Read section
Examples (examples/{arch}/) Read section
Device Scene Tests (tests/st/{arch}/) Read section
Writing New Tests Read section
New C++ Unit Test Read section
C++ hardware tests needing NPU devices Read section
New Scene Test Read section
Kernels That Wait for Siblings (Mix, sync_start) Read section
Output Validation Read section
Case Selection and Manual Cases Read section
Sharing an Example Between examples/ and tests/st/ Read section
Runtime Isolation Constraint (Onboard) Read section
Root Cause Read section
Impact Read section
Mitigation Read section
Device Allocation (Orchestrator + xdist) Read section
Resource phase — device bin-packing Read section
L2 phase — xdist fanout per device Read section
L2 phase — standalone fanout Read section
Sim platforms Read section
Per-Case Device Filtering Read section
When separate directories are still needed Read section
Explicit L2 child memory Read section