Skip to content

BLD: Build with CYTHON_USE_TYPE_SPECS=1 to fix segfaults with NumPy main - #515

Closed
bashtage wants to merge 1 commit into
mainfrom
fix/cython-type-specs-numpy-heap-bitgenerator
Closed

bashtage wants to merge 1 commit into
mainfrom
fix/cython-type-specs-numpy-heap-bitgenerator

Conversation

@bashtage

@bashtage bashtage commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

The python312_prerelease (pip-pre) Azure job has segfaulted (exit code 139) since the 2026-11-07 NumPy nightly. The same randomgen commit (51c3967d) passed on 2026-11-05 and crashed on 2026-11-08, so the cause is the NumPy nightly, not randomgen.

Cause. numpy/numpy#32332 (PyPy 3.12 support) added -DCYTHON_USE_TYPE_SPECS=1 to numpy/random/meson.build (see also cython/cython#7947). numpy.random.bit_generator.BitGenerator is now a heap type. The dealloc of a heap-type base decrefs Py_TYPE(o), but randomgen's subclasses are static types that never took that reference. Every bit generator that is created and destroyed therefore removes one reference from its class. Once the class is freed the interpreter segfaults, and the crash lands in a different test on each run because of pytest-randomly.

Change. Add -DCYTHON_USE_TYPE_SPECS=1 to cython_c_args so randomgen's extension types are heap types too. tests/meson.build uses the same variable.

Verification (local, WSL Ubuntu, Python 3.12, Cython 3.3.1)

  • On numpy 2.6.0.dev0+git20261107.80049a0, the CI test command segfaulted 6 out of 6 runs before the change. With it, 3 out of 3 runs passed: 8764 passed, 148 skipped.
  • sys.getrefcount(cls) changes by 0 over construct/destroy cycles for all 31 bit generator classes. Without the change it drops by 1 per instance for all 31 on the nightly.
  • Compatibility matrix: wheels built on numpy 2.0.2, with and without the change, run on numpy 1.26.4, 2.0.2, 2.2.6, 2.4.6, 2.5.4 and the 2026-11-07 nightly.
    • Without the change, the wheel is fine on every release but loses a type reference on all 31 classes on the nightly.
    • With the change, there is no refcount drift on any version, and the full suite passes on 2.0.2 through the nightly (8793 passed, 117 skipped).
    • On numpy 1.26.4 the changed wheel also has no drift, but 23 pickle tests fail. The same 23 fail without the change, so the change does not cause them. This wheel was built on 2.0.2 and run on 1.26.4, and I did not investigate the cause. The python310_last_numpy_1 job builds on 1.26.4 itself, so CI covers that combination.

Caveats

  • This only fixes newly built wheels. Wheels already released, which are built against static-type NumPy, hit the same crash on NumPy main, so this also needs a NumPy-side fix.
  • Windows and macOS builds (MSVC, clang) were not tested locally. Meson should translate the -D flag, as numpy's own build relies on.
  • The prerelease test step has continueOnError: true, so a crash there shows as "succeeded with issues" instead of failing the job. Check that step's result, not only the job's overall status.

🤖 Generated with Claude Code

…tGenerator

NumPy main (numpy/numpy#32332) now builds numpy.random with
-DCYTHON_USE_TYPE_SPECS=1, so numpy.random.bit_generator.BitGenerator is a
heap type. The dealloc of a heap-type base decrefs the instance's type. The
static-type subclasses randomgen builds never took that reference, so every
bit generator that is created and destroyed removes one reference from its
class. Once the class is freed the interpreter segfaults, which is what the
pip-pre Azure job has been hitting since the 2026-11-07 NumPy nightly.

Build randomgen's extensions as heap types as well. This is correct for both
the heap-type BitGenerator of NumPy main and the static-type BitGenerator of
NumPy <= 2.5.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.80%. Comparing base (b29ad24) to head (888a66f).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #515   +/-   ##
=======================================
  Coverage   94.80%   94.80%           
=======================================
  Files          56       56           
  Lines        9922     9922           
  Branches      215      215           
=======================================
  Hits         9407     9407           
  Misses        495      495           
  Partials       20       20           
Flag Coverage Δ
adder 94.74% <ø> (ø)
subtractor 94.74% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bashtage bashtage closed this Nov 11, 2026
@bashtage
bashtage deleted the fix/cython-type-specs-numpy-heap-bitgenerator branch November 11, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant