Skip to content

refactor(infra)!: unify CUDA backend configuration - #20645

Merged
tqchen merged 14 commits into
apache:mainfrom
spectrometerHBH:refactor-cuda-compile-config
Oct 11, 2026
Merged

tqchen merged 14 commits into
apache:mainfrom
spectrometerHBH:refactor-cuda-compile-config

Conversation

@spectrometerHBH

@spectrometerHBH spectrometerHBH commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

CUDA compiler settings need to be consistent across build defaults, target presets, and individual device entries without maintaining a Python field for every native compiler flag. Introduce a lightweight per-backend mapping accepted directly by tvm.compile(..., backend_config=...) and T.device_entry(..., backend_config=...).

from tvm.backend.cuda import BackendConfig

cuda_config = BackendConfig(
    arch="sm_100a",
    compiler="nvrtc",
    nvrtc=["--use_fast_math", "--ftz=false"],
)
executable = tvm.compile(func, backend_config={"cuda": cuda_config})
  • CUDA exposes six optional keys: arch, compiler, target_format, nvcc, nvrtc, and ptxas. Native argv lists are forwarded to the selected toolchain. BackendConfig is a TypedDict whose constructor returns a regular dictionary.
  • Resolve backend defaults, Target/tag defaults, compile overrides, and entry overrides before architecture-sensitive lowering. Lists replace inherited lists; an empty list clears them. Snapshot inputs and use canonical configuration cache keys.
  • Compile entries with different architectures or options in separate existing CUDA modules. Specialize shared private device helpers before tile dispatch and dtype legalization so each helper follows its caller's configuration.
  • Keep CUDA runtime metadata to one opaque backend configuration string appended after the existing serialized module prefix. Remove the versioned metadata helper and persisted compiler logs. Accept legacy payloads, avoid invoking a compiler for source-only fallback modules, and replay saved source independently of an unrelated current Target.
  • Export cuda_host as ordinary C++ with embedded independently compiled device binaries. Exported libraries require TVM FFI and CUDA, with no device compiler required at runtime.
  • Replace CompileConfig, compiler-policy environment variables, and decorator-level architecture settings. Obsolete compiler-policy environment variables are ignored. Update build, Relax, external-kernel, and IKET consumers and document native compiler options separately from the runtime schema.

The caller migration is in mlc-ai/TIRx-kernels#8. The earlier CUDA launch configuration PR #20639 is merged.

Validation on the current implementation:

  • TVM C++ build and changed-file pre-commit checks passed.
  • Full TIRx suite: 2,691 passed / 290 skipped on seven B200 GPUs (160 seconds). Workload import gate (274 workloads / 93 kernels) and strict registry import gate passed.
  • CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
  • Backend configuration tests: 45 passed, covering native flags and FTZ behavior, Target/tag defaults, mixed NVCC/NVRTC entries, helper specialization, concurrent compilation, explicit offline architecture, source fallback, legacy payloads, fresh-process artifact replay, and standalone C++ host export.
  • TIRx-kernels unit tests: 225 passed, including configuration forwarding, CPU preparation without CUDA initialization, mutable-input snapshots, and cache keys across Target defaults.

Comment thread src/backend/cuda/module_metadata.h Outdated
ffi::String text(code->data(), code->size());
auto result = compile(text, config.value()).cast<ffi::Array<ffi::Any>>();
source->Set("cuda", text);
source->Set("cuda.compile_config", result[2].cast<ffi::String>());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source remember the code and backend_config as string

ffi::String fmt_;
// function information table.
ffi::Map<ffi::String, FunctionInfo> fmap_;
// In-memory source map for InspectSource — never serialized.

@tqchen tqchen Oct 10, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ffi::String backend_config_; // serialized json string of backend config;

@spectrometerHBH spectrometerHBH changed the title refactor(lower-tirx)!: unify per-entry CUDA compilation configuration refactor(infra)!: unify CUDA backend configuration Oct 11, 2026
@tqchen
tqchen merged commit 0f861ee into apache:main Oct 11, 2026
8 checks passed
tqchen pushed a commit that referenced this pull request Oct 11, 2026
CUDA backend configuration already travels through Target attributes.
Remove the parallel `backend_config` keyword from `tvm.compile`,
`tvm.tirx.build`, `tvm.relax.build`, and `IketProfiler.compile`;
compilation now resolves its defaults from the supplied Target.

```python
import tvm
from tvm.backend.cuda import BackendConfig

target = tvm.target.Target({
    "kind": "cuda",
    "arch": "sm_100a",
    "backend_config": {
        "cuda": BackendConfig(compiler="nvrtc", nvrtc=["--use_fast_math", "--ftz=false"])
    },
})
executable = tvm.compile(func, target=target)
```

Each `device_entry(backend_config=...)` keeps its independent overrides,
including architecture and toolchain settings. The existing per-entry
resolution and module grouping continue to apply. Update callers, source
helpers, and documentation to use Target defaults followed by entry
overrides.

Follow-up to merged #20645.

Validation:

- Built successfully on the merged #20645 baseline (`0f861ee568`) in 67
seconds.
- Full TIRx suite: 2,689 passed / 290 skipped on seven B200 GPUs, 165
seconds.
- Backend configuration, launch, and IKET regressions: 131 passed / 2
skipped.
- CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
- Updated TIRx-kernels callers: 94 focused tests passed; all eight MQA
registry correctness cases passed. Both workload and strict registry
import gates passed (274 workloads / 93 kernels).
- Changed-file and staged pre-commit checks passed.

The latest upstream commit (`27576ce2d1`) fails the local NCCL-enabled
C++ build in `cuda_ipc_memory.cc` after its object-handle changes. The
checks above therefore use the merged #20645 baseline; this follow-up
changes Python APIs, tests, and documentation only.
tqchen pushed a commit that referenced this pull request Oct 16, 2026
CUDA compiler settings need to be consistent across build defaults,
target presets, and individual device entries without maintaining a
Python field for every native compiler flag. Introduce a lightweight
per-backend mapping accepted directly by `tvm.compile(...,
backend_config=...)` and `T.device_entry(..., backend_config=...)`.

```python
from tvm.backend.cuda import BackendConfig

cuda_config = BackendConfig(
    arch="sm_100a",
    compiler="nvrtc",
    nvrtc=["--use_fast_math", "--ftz=false"],
)
executable = tvm.compile(func, backend_config={"cuda": cuda_config})
```

- CUDA exposes six optional keys: `arch`, `compiler`, `target_format`,
`nvcc`, `nvrtc`, and `ptxas`. Native argv lists are forwarded to the
selected toolchain. `BackendConfig` is a TypedDict whose constructor
returns a regular dictionary.
- Resolve backend defaults, Target/tag defaults, compile overrides, and
entry overrides before architecture-sensitive lowering. Lists replace
inherited lists; an empty list clears them. Snapshot inputs and use
canonical configuration cache keys.
- Compile entries with different architectures or options in separate
existing CUDA modules. Specialize shared private device helpers before
tile dispatch and dtype legalization so each helper follows its caller's
configuration.
- Keep CUDA runtime metadata to one opaque backend configuration string
appended after the existing serialized module prefix. Remove the
versioned metadata helper and persisted compiler logs. Accept legacy
payloads, avoid invoking a compiler for source-only fallback modules,
and replay saved source independently of an unrelated current Target.
- Export `cuda_host` as ordinary C++ with embedded independently
compiled device binaries. Exported libraries require TVM FFI and CUDA,
with no device compiler required at runtime.
- Replace `CompileConfig`, compiler-policy environment variables, and
decorator-level architecture settings. Obsolete compiler-policy
environment variables are ignored. Update build, Relax, external-kernel,
and IKET consumers and document native compiler options separately from
the runtime schema.

The caller migration is in
[mlc-ai/TIRx-kernels#8](mlc-ai/TIRx-kernels#8).
The earlier CUDA launch configuration PR #20639 is merged.

Validation on the current implementation:

- TVM C++ build and changed-file pre-commit checks passed.
- Full TIRx suite: 2,691 passed / 290 skipped on seven B200 GPUs (160
seconds). Workload import gate (274 workloads / 93 kernels) and strict
registry import gate passed.
- CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
- Backend configuration tests: 45 passed, covering native flags and FTZ
behavior, Target/tag defaults, mixed NVCC/NVRTC entries, helper
specialization, concurrent compilation, explicit offline architecture,
source fallback, legacy payloads, fresh-process artifact replay, and
standalone C++ host export.
- TIRx-kernels unit tests: 225 passed, including configuration
forwarding, CPU preparation without CUDA initialization, mutable-input
snapshots, and cache keys across Target defaults.
tqchen pushed a commit that referenced this pull request Oct 16, 2026
CUDA backend configuration already travels through Target attributes.
Remove the parallel `backend_config` keyword from `tvm.compile`,
`tvm.tirx.build`, `tvm.relax.build`, and `IketProfiler.compile`;
compilation now resolves its defaults from the supplied Target.

```python
import tvm
from tvm.backend.cuda import BackendConfig

target = tvm.target.Target({
    "kind": "cuda",
    "arch": "sm_100a",
    "backend_config": {
        "cuda": BackendConfig(compiler="nvrtc", nvrtc=["--use_fast_math", "--ftz=false"])
    },
})
executable = tvm.compile(func, target=target)
```

Each `device_entry(backend_config=...)` keeps its independent overrides,
including architecture and toolchain settings. The existing per-entry
resolution and module grouping continue to apply. Update callers, source
helpers, and documentation to use Target defaults followed by entry
overrides.

Follow-up to merged #20645.

Validation:

- Built successfully on the merged #20645 baseline (`0f861ee568`) in 67
seconds.
- Full TIRx suite: 2,689 passed / 290 skipped on seven B200 GPUs, 165
seconds.
- Backend configuration, launch, and IKET regressions: 131 passed / 2
skipped.
- CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
- Updated TIRx-kernels callers: 94 focused tests passed; all eight MQA
registry correctness cases passed. Both workload and strict registry
import gates passed (274 workloads / 93 kernels).
- Changed-file and staged pre-commit checks passed.

The latest upstream commit (`27576ce2d1`) fails the local NCCL-enabled
C++ build in `cuda_ipc_memory.cc` after its object-handle changes. The
checks above therefore use the merged #20645 baseline; this follow-up
changes Python APIs, tests, and documentation only.
tqchen pushed a commit that referenced this pull request Oct 18, 2026
CUDA compiler settings need to be consistent across build defaults,
target presets, and individual device entries without maintaining a
Python field for every native compiler flag. Introduce a lightweight
per-backend mapping accepted directly by `tvm.compile(...,
backend_config=...)` and `T.device_entry(..., backend_config=...)`.

```python
from tvm.backend.cuda import BackendConfig

cuda_config = BackendConfig(
    arch="sm_100a",
    compiler="nvrtc",
    nvrtc=["--use_fast_math", "--ftz=false"],
)
executable = tvm.compile(func, backend_config={"cuda": cuda_config})
```

- CUDA exposes six optional keys: `arch`, `compiler`, `target_format`,
`nvcc`, `nvrtc`, and `ptxas`. Native argv lists are forwarded to the
selected toolchain. `BackendConfig` is a TypedDict whose constructor
returns a regular dictionary.
- Resolve backend defaults, Target/tag defaults, compile overrides, and
entry overrides before architecture-sensitive lowering. Lists replace
inherited lists; an empty list clears them. Snapshot inputs and use
canonical configuration cache keys.
- Compile entries with different architectures or options in separate
existing CUDA modules. Specialize shared private device helpers before
tile dispatch and dtype legalization so each helper follows its caller's
configuration.
- Keep CUDA runtime metadata to one opaque backend configuration string
appended after the existing serialized module prefix. Remove the
versioned metadata helper and persisted compiler logs. Accept legacy
payloads, avoid invoking a compiler for source-only fallback modules,
and replay saved source independently of an unrelated current Target.
- Export `cuda_host` as ordinary C++ with embedded independently
compiled device binaries. Exported libraries require TVM FFI and CUDA,
with no device compiler required at runtime.
- Replace `CompileConfig`, compiler-policy environment variables, and
decorator-level architecture settings. Obsolete compiler-policy
environment variables are ignored. Update build, Relax, external-kernel,
and IKET consumers and document native compiler options separately from
the runtime schema.

The caller migration is in
[mlc-ai/TIRx-kernels#8](mlc-ai/TIRx-kernels#8).
The earlier CUDA launch configuration PR #20639 is merged.

Validation on the current implementation:

- TVM C++ build and changed-file pre-commit checks passed.
- Full TIRx suite: 2,691 passed / 290 skipped on seven B200 GPUs (160
seconds). Workload import gate (274 workloads / 93 kernels) and strict
registry import gate passed.
- CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
- Backend configuration tests: 45 passed, covering native flags and FTZ
behavior, Target/tag defaults, mixed NVCC/NVRTC entries, helper
specialization, concurrent compilation, explicit offline architecture,
source fallback, legacy payloads, fresh-process artifact replay, and
standalone C++ host export.
- TIRx-kernels unit tests: 225 passed, including configuration
forwarding, CPU preparation without CUDA initialization, mutable-input
snapshots, and cache keys across Target defaults.
tqchen pushed a commit that referenced this pull request Oct 18, 2026
CUDA backend configuration already travels through Target attributes.
Remove the parallel `backend_config` keyword from `tvm.compile`,
`tvm.tirx.build`, `tvm.relax.build`, and `IketProfiler.compile`;
compilation now resolves its defaults from the supplied Target.

```python
import tvm
from tvm.backend.cuda import BackendConfig

target = tvm.target.Target({
    "kind": "cuda",
    "arch": "sm_100a",
    "backend_config": {
        "cuda": BackendConfig(compiler="nvrtc", nvrtc=["--use_fast_math", "--ftz=false"])
    },
})
executable = tvm.compile(func, target=target)
```

Each `device_entry(backend_config=...)` keeps its independent overrides,
including architecture and toolchain settings. The existing per-entry
resolution and module grouping continue to apply. Update callers, source
helpers, and documentation to use Target defaults followed by entry
overrides.

Follow-up to merged #20645.

Validation:

- Built successfully on the merged #20645 baseline (`0f861ee568`) in 67
seconds.
- Full TIRx suite: 2,689 passed / 290 skipped on seven B200 GPUs, 165
seconds.
- Backend configuration, launch, and IKET regressions: 131 passed / 2
skipped.
- CUDA codegen and Relax VM build regressions: 415 passed / 4 skipped.
- Updated TIRx-kernels callers: 94 focused tests passed; all eight MQA
registry correctness cases passed. Both workload and strict registry
import gates passed (274 workloads / 93 kernels).
- Changed-file and staged pre-commit checks passed.

The latest upstream commit (`27576ce2d1`) fails the local NCCL-enabled
C++ build in `cuda_ipc_memory.cc` after its object-handle changes. The
checks above therefore use the merged #20645 baseline; this follow-up
changes Python APIs, tests, and documentation only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants