OpenVMM: intermittent abort while stopping the VM in MSHV test_ttrpc_interface vmm-test #393

Summary

ttrpc::test_ttrpc_interface intermittently fails on Linux/MSHV virtual-machine CI runners because the OpenVMM process ends while it stops the VM worker after a guest power-off. The controller never logs vm stopped, and no OpenVMM panic text is captured.

It has failed this way twice:

• 2026-09-19 in Enable backend-specific OpenVMM VMM tests #132's CI, where OpenVMM exited with wait status 134 (SIGABRT, core dumped).
• 2026-10-05 on a dev push, where the ttrpc connection closed during TeardownVm. That failed Required status check and skipped the development release and the performance-baseline update.

#135 recorded the first failure as intermittent on the virtual runner. #156 then enabled the test on MSHV after ten consecutive bare-metal passes, without an OpenVMM change, so the abort was never root-caused.

Current incident

• Workflow: CI, run 37337789585, attempt 1
• Event: push to dev at 9db38045663bc4b187ecce769fdd6e65dd12b271, the merge of harness: monitor measure_once's console to its end after every teardown #379
• OpenVMM source: nanvix/openvmm@974f356521829b96798ee6ac30cc5d1f0e1bb87b
• Failed job/step: OpenVMM vmm-tests / Linux / MSHV / Run OpenVMM tests on MSHV, which runs python3 scripts/nvx.py test-openvmm --backend mshv. Required status check failed only because of it.
• Runner: azure-mshv-1, machine esaurez-nvx-ci-linux-vmss000000, labels self-hosted, linux, mshv, virtual-machine
• Result: 57 tests run: 56 passed, 1 failed, 40 skipped; the failing test took 5.751 s. The KVM lane passed the same test (58/58), and the WHP lane passed.
• Diagnostics: artifact openvmm-vmm-tests-Linux-mshv-37337789585-1, which expires on 2026-10-12
• Classification: intermittent OpenVMM failure, not a cancellation or supersession. Since the last green dev run (e2fbe8ba1), NVX changed only benchmark harness code and tests, docs, and performance data. The workflow, scripts/nvx_tools/ci.py, and the openvmm gitlink are unchanged.

The test panicked at the first iteration's TeardownVm call, after WaitVm returned and PropertiesVm reported HALTED:

thread 'main' panicked at vmm_tests/vmm_tests/tests/tests/ttrpc.rs:944:26:
called `Result::unwrap()` on an `Err` value: Status { code: 14, message: "channel closed", details: [] }

The OpenVMM stderr ends after the consomme warning, with no vm stopped line and no panic text:

[    stderr]    4.559202463s  INFO openvmm_entry::vm_controller:  guest halted reason=PowerOff
[    stderr]    4.559238563s  INFO openvmm_entry::ttrpc:  guest halted (via controller) reason=PowerOff
[     petri] 2026-10-05T16:48:48.632075Z DEBUG mesh_rpc::client: response stream_id=35
[     petri] 2026-10-05T16:48:48.632281Z DEBUG mesh_rpc::client: response stream_id=41
[    stderr]    4.559639460s  INFO openvmm_entry::vm_controller:  quitting
[    stderr]    4.559680259s  INFO openvmm_entry::vm_controller:  controller RPC channel closed, stopping VM
[     petri] 2026-10-05T16:48:48.632882Z  INFO petri::tracing: disconnected from linux com1: EOF
[    stderr]    4.564110926s  WARN net_consomme:  Consomme port request channel failure err=channel closed
[     petri] 2026-10-05T16:48:49.476359Z  INFO petri::tracing: disconnected from virtio console: EOF
[     petri] 2026-10-05T16:48:49.734153Z ERROR petri::tracing: test failed error=test panicked: called `Result::unwrap()` on an `Err` value: Status { code: 14, message: "channel closed", details: [] }

The mesh RPC client returns Unavailable / channel closed when the connection closes while a request is still pending (support/mesh/mesh_rpc/src/client.rs:274). In this test the ttrpc server, the VM controller, and the VM worker all run in the single OpenVMM process (VmmMesh::new(.., true)). The connection closing about 1.1 s into the stop therefore means the process ended mid-stop. The test's OpenvmmChild guard did not log killing openvmm, which was still running, which is consistent with the process having already exited.

Earlier incident (2026-09-19)

• Enable backend-specific OpenVMM VMM tests #132's CI, run 35474253275, attempt 1, job OpenVMM tests / Linux / MSHV
• NVX 438991cebfa26e98b8822b65ee8babcaa0b38dde, OpenVMM 0bc357bbcf3a654b63dfb51f1103c5751bf3d31f
• Runner azure-azlinux-2, machine esaurez-nvx-ci-linux-vmss000001

The test failed at its final exit-status check:

thread 'main' panicked at vmm_tests/vmm_tests/tests/tests/ttrpc.rs:1184:5:
openvmm exited abnormally: ExitStatus(unix_wait_status(134))

The third VM, which powers off and then receives Quit, shows the same sequence:

[    stderr]   12.931858036s  INFO openvmm_entry::vm_controller:  guest halted reason=PowerOff
[    stderr]   12.932418038s  INFO openvmm_entry::vm_controller:  quitting
[    stderr]   12.932446538s  INFO openvmm_entry::vm_controller:  controller RPC channel closed, stopping VM
[     petri] 2026-09-19T22:55:28.678064Z  INFO petri::tracing: disconnected from linux com1: EOF
[    stderr]   12.936356854s  WARN net_consomme:  Consomme port request channel failure err=channel closed
[     petri] 2026-09-19T22:55:29.777814Z  INFO petri::tracing: disconnected from virtio console: EOF
[     petri] 2026-09-19T22:55:29.817670Z  INFO tests::ttrpc: openvmm exited exit_status=ExitStatus(unix_wait_status(134))

The first two VMs in that run logged vm stopped 0.62 s and 0.44 s after stopping VM. In both failures, the process ended about 1.1 s after stopping VM, shortly after the virtio console disconnected.

Analysis

• Both failures end OpenVMM during the VM-worker stop that follows a guest power-off. That is the point where VmController::run (openvmm/openvmm_entry/src/vm_controller.rs) would log vm stopped.
• The release profile sets panic = 'abort', so a Rust panic in the stop path would produce exactly SIGABRT (wait status 134). An explicit or libc abort is also possible. The exit status of today's failure was not observed, because the test failed on the closed RPC connection first.
• Any panic text was probably lost. In neither failure did petri log disconnected from openvmm stderr, so the test ended before petri finished reading OpenVMM's stderr. In the passing test_ttrpc_uefi_boot from the same job, petri logs that line before openvmm exited.
• It is not specific to one runner VM. The failures ran on vmss000001 and vmss000000. Today's VM is the one ci: every Linux/MSHV snapshot-generation stability rejection ran on azure-mshv-1 #383 flags for timing stalls, but the earlier abort ran on a different VM.
• It is not a recent OpenVMM regression. The two OpenVMM pins are on different histories, since 0bc357b is not an ancestor of 974f356.
• It is rare. Since ci: enable MSHV TTRPC interface test #156 enabled the test on MSHV on 2026-09-21, today's run is the first failure observed. MSHV vmm-tests jobs passed 158 times between 2026-09-29 and 2026-10-05. No KVM or WHP failure of this test appears in the CI runs checked between 2026-09-29 and 2026-10-05.

Next steps

• Retrieve the core or kernel log for the abort. The 2026-09-19 wait status (134) includes the core-dump flag, so the runner VMs may keep cores for OpenVMM aborts. On esaurez-nvx-ci-linux-vmss000000, check coredumpctl list openvmm and journalctl -k around 2026-10-05T16:48:49Z.
• Reproduce by looping ttrpc::test_ttrpc_interface on a Linux/MSHV virtual-machine host. ci: enable MSHV TTRPC interface test #156's ten bare-metal runs did not reproduce it.
• Make the test report the failure. When the RPC connection closes unexpectedly, the test should wait for the child and report its exit status. It should also drain OpenVMM's stderr before failing, so any panic text is kept.
• Trace the VM-worker teardown after a guest power-off on MSHV, including device and partition teardown in the VM worker and virt_mshv.

Acceptance criteria

• The abort's root cause is fixed in OpenVMM or the MSHV integration, not hidden by retries, exclusions, or runtime skips.
• The exact test passes repeatedly on the MSHV virtual-machine runner class.
• A future abort in this test reports OpenVMM's exit status and its final stderr.

Related

• openvmmCI: enable remaining backend-native VMM tests #135 records this test as failing intermittently on the MSHV virtual runner. ci: enable MSHV TTRPC interface test #156 enabled the test on MSHV, and OpenVMM VMM test: enable test_ttrpc_interface #81 is the original enablement issue.
• OpenVMM CI: fix MSHV PCIe save/restore pulse hang #184 and mshv: hang on partition stop in reboot test openvmm#3464 track MSHV hangs during restore and partition stop. Those are hangs, not aborts.
• ci: every Linux/MSHV snapshot-generation stability rejection ran on azure-mshv-1 #383 tracks timing stalls on the same runner VM as today's failure.