Skip to content

gh-159137: Keep surviving objects in order in the garbage collector #159152

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pablogsal wants to merge 7 commits into
base: main
Choose a base branch
from
+158 −113
Open

gh-159137: Keep surviving objects in order in the garbage collector #159152

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d7de116
gh-159137: Keep surviving objects in order in the garbage collector
pablogsal Oct 10, 2026
3a73e71
Drop the NEWS entry and the doc changes
pablogsal Oct 10, 2026
51dc0ca
Simplify
pablogsal Oct 10, 2026
40e9364
Update the internal docs
pablogsal Oct 10, 2026
514d0f4
Move runs of unreachable objects at once
pablogsal Oct 10, 2026
5618614
Add a NEWS entry
pablogsal Oct 10, 2026
6c08848
Look for the unreachable objects from both ends of the list
pablogsal Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 29 additions & 32 deletions InternalDocs/garbage_collector.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,8 +260,10 @@ ended having `gc_ref == 0` but is referenced still by the `link_1` object that
is reachable from the outside. To obtain the set of objects that are really
unreachable, the garbage collector re-scans the container objects using the
`tp_traverse` slot; this time with a different traverse function that marks objects with
`gc_ref == 0` as "tentatively unreachable" and then moves them to the
tentatively unreachable list. The following image depicts the state of the lists in a
`gc_ref == 0` as "tentatively unreachable". The images show these objects in a
list of their own; the implementation sets the `NEXT_MASK_UNREACHABLE` flag on them
and leaves them where they are until the scan ends. The following image depicts the
state of the lists in a
moment when the GC processed the `link_3` and `link_4` objects but has not
processed `link_1` and `link_2` yet.

Expand All @@ -275,56 +277,51 @@ already in what will become the reachable list):

When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses
its references using the `tp_traverse` slot to find all the objects that are
reachable from it, moving them to the end of the list of reachable objects (where
they started originally) and setting its `gc_ref` field to 1. This is what happens
to `link_2` and `link_3` below as they are reachable from `link_1`. From the
reachable from it. Those that the scan has not processed yet get their `gc_ref`
field set to 1, so that the GC knows they are reachable when it gets to them. This is
what happens to `link_2` below as it is reachable from `link_1`. From the
state in the previous image and after examining the objects referred to by `link_1`
the GC knows that `link_3` is reachable after all, so it is moved back to the
original list and its `gc_ref` field is set to 1 so that if the GC visits it again,
it will know that it's reachable. To avoid visiting an object twice, the GC marks all
the GC knows that `link_3` is reachable after all, so it is no longer tentatively
unreachable. To avoid visiting an object twice, the GC marks all
objects that have already been visited once (by unsetting the `PREV_MASK_COLLECTING`
flag) so that if an object that has already been processed is referenced by some other
object, the GC does not process it twice.

![gc-image5](images/python-cyclic-gc-5-new-page.png)

Notice that an object that was marked as "tentatively unreachable" and was later
moved back to the reachable list will be visited again by the garbage collector
as now all the references that the object has need to be processed as well. This
process is really a breadth first search over the object graph. Once all the objects
are scanned, the GC knows that all container objects in the tentatively unreachable
list are really unreachable and can thus be garbage collected.
found to be reachable has to be traversed as well, as now all the references that
the object has need to be processed. The scan has already passed it, so the GC puts
it in a small stack and traverses the objects in the stack before it moves on. Once
all the objects are scanned, the GC knows that all container objects that are still
tentatively unreachable are really unreachable and can thus be garbage collected.
They are then moved to the unreachable list.

Pragmatically, it's important to note that no recursion is required by any of this,
and neither does it in any other way require additional memory proportional to the
number of objects, number of pointers, or the lengths of pointer chains. Apart from
`O(1)` storage for internal C needs, the objects themselves contain all the storage
the GC algorithms require.
the GC algorithms require. The stack has a fixed size: when it is full, an object
found to be reachable is moved to the end of the list instead, where the scan gets
to it again.

Why moving unreachable objects is better
----------------------------------------

It sounds logical to move the unreachable objects under the premise that most objects
are usually reachable, until you think about it: the reason it pays isn't actually
obvious.
Why reachable objects are not moved
-----------------------------------

Suppose we create objects A, B, C in that order. They appear in the young generation
in the same order. If B points to A, and C to B, and C is reachable from outside,
then the adjusted refcounts after the first step of the algorithm runs will be 0, 0,
and 1 respectively because the only reachable object from the outside is C.

When the next step of the algorithm finds A, A is moved to the unreachable list. The
same for B when it's first encountered. Then C is traversed, B is moved *back* to
the reachable list. B is eventually traversed, and then A is moved back to the reachable
list.

So instead of not moving at all, the reachable objects B and A are each moved twice.
Why is this a win? A straightforward algorithm to move the reachable objects instead
would move A, B, and C once each. The key is that this dance leaves the objects in
order C, B, A - it's reversed from the original order. On all *subsequent* scans,
none of them will move. Since most objects aren't in cycles, this can save an
unbounded number of moves across an unbounded number of later collections. The only
time the cost can be higher is the first time the chain is scanned.
When the next step of the algorithm finds A, A is marked as tentatively unreachable.
The same for B when it's first encountered. Then C is traversed, B is found to be
reachable and is traversed from the stack, and then A is. The list is still A, B, C.

Objects are added to the list as they are allocated, so the order of the list is
close to the order of the objects in memory, and a walk of the list goes through
memory mostly in address order. The collector walks the list several times in every
collection. Keeping the order keeps those walks friendly to the CPU cache, in this
collection and in all the later ones.

Destroying unreachable objects
==============================
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Speed up the garbage collector on large heaps by keeping the objects that
survive a collection in the order in which they were allocated.
:func:`gc.get_objects` returns them in that order.
207 changes: 126 additions & 81 deletions Python/gc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,8 @@ typedef struct _gc_runtime_state GCState;

// Lowest bit of _gc_next is used for UNREACHABLE flag.
//
// This flag represents the object is in unreachable list in move_unreachable()
// move_unreachable() sets this flag on the objects it assumes unreachable.
// They stay in the list being scanned until the scan ends.
//
// Although this flag is used only in move_unreachable(), move_unreachable()
// doesn't clear this flag to skip unnecessary iteration.
Expand Down Expand Up @@ -192,14 +193,13 @@ _gc_next takes these values:
NEXT_MASK_UNREACHABLE flag described below.

NEXT_MASK_UNREACHABLE
move_unreachable() then moves objects not reachable (whether directly or
indirectly) from outside the generation into an "unreachable" set and
set this flag.
move_unreachable() sets this flag on objects not reachable (whether
directly or indirectly) from outside the generation, and moves them into
an "unreachable" set when its scan ends.

Objects that are found to be reachable have gc_refs set to 1.
When this flag is set for the reachable object, the object must be in
"unreachable" set.
The flag is unset and the object is moved back to "reachable" set.
When this flag is set for the reachable object, the scan has already
passed it. The flag is unset and the object is traversed in place.

move_legacy_finalizers() will remove this flag from "unreachable" set.
*/
Expand Down Expand Up @@ -500,11 +500,19 @@ subtract_refs(PyGC_Head *containers)
}
}

/* State shared by move_unreachable() and visit_reachable(). */
struct reachable_state {
PyGC_Head *young;
Py_ssize_t unreachable; /* objects with NEXT_MASK_UNREACHABLE */
Py_ssize_t top;
PyGC_Head *stack[1024]; /* reachable objects to traverse */
};

/* A traversal callback for move_unreachable. */
static int
visit_reachable(PyObject *op, void *arg)
{
PyGC_Head *reachable = arg;
struct reachable_state *state = arg;
OBJECT_STAT_INC(object_visits);
if (!_PyObject_IS_GC(op)) {
return 0;
Expand All @@ -527,23 +535,30 @@ visit_reachable(PyObject *op, void *arg)
if (gc->_gc_next & NEXT_MASK_UNREACHABLE) {
/* This had gc_refs = 0 when move_unreachable got
* to it, but turns out it's reachable after all.
* Move it back to move_unreachable's 'young' list,
* and move_unreachable will eventually get to it
* again.
* Leave it where it is, so that the list keeps its
* order, and let move_unreachable traverse it.
*/
// Manually unlink gc from unreachable list because the list functions
// don't work right in the presence of NEXT_MASK_UNREACHABLE flags.
PyGC_Head *prev = GC_PREV(gc);
PyGC_Head *next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE);
_PyObject_ASSERT(FROM_GC(prev),
prev->_gc_next & NEXT_MASK_UNREACHABLE);
_PyObject_ASSERT(FROM_GC(next),
next->_gc_next & NEXT_MASK_UNREACHABLE);
prev->_gc_next = gc->_gc_next; // copy NEXT_MASK_UNREACHABLE
_PyGCHead_SET_PREV(next, prev);

gc_list_append(gc, reachable);
gc_set_refs(gc, 1);
gc->_gc_next &= ~NEXT_MASK_UNREACHABLE;
state->unreachable--;
if (state->top < (Py_ssize_t)Py_ARRAY_LENGTH(state->stack)) {
gc_clear_collecting(gc);
state->stack[state->top++] = gc;
}
else {
/* No room. Move it to the end of 'young' and
* move_unreachable will eventually get to it again.
*/
// Manually unlink gc because the list functions don't work
// right in the presence of NEXT_MASK_UNREACHABLE flags.
PyGC_Head *prev = GC_PREV(gc);
PyGC_Head *next = GC_NEXT(gc);
prev->_gc_next = (prev->_gc_next & NEXT_MASK_UNREACHABLE)
| (uintptr_t)next;
_PyGCHead_SET_PREV(next, prev);

gc_list_append(gc, state->young);
gc_set_refs(gc, 1);
}
}
else if (gc_refs == 0) {
/* This is in move_unreachable's 'young' list, but
Expand Down Expand Up @@ -574,21 +589,28 @@ visit_reachable(PyObject *op, void *arg)
* doubly linked list after this function.
* But _gc_next in unreachable list has NEXT_MASK_UNREACHABLE flag.
* So we can not gc_list_* functions for unreachable until we remove the flag.
*
* The objects left in young keep their order. It is the order in which they
* were allocated, so walking the list visits memory mostly in address order.
*/
static void
move_unreachable(PyGC_Head *young, PyGC_Head *unreachable)
{
struct reachable_state state;
state.young = young;
state.unreachable = 0;
state.top = 0;
// previous elem in the young list, used for restore gc_prev.
PyGC_Head *prev = young;
PyGC_Head *gc = GC_NEXT(young);

/* Invariants: all objects "to the left" of us in young are reachable
* (directly or indirectly) from outside the young list as it was at entry.
/* Invariants: all objects "to the left" of us in young without
* NEXT_MASK_UNREACHABLE are reachable (directly or indirectly) from
* outside the young list as it was at entry.
*
* All other objects from the original young "to the left" of us are in
* unreachable now, and have NEXT_MASK_UNREACHABLE. All objects to the
* left of us in 'young' now have been scanned, and no objects here
* or to the right have been scanned yet.
* All objects to the left of us in 'young' now have been scanned,
* their _gc_prev is a pointer again, and no objects here or to the
* right have been scanned yet.
*/

while (gc != young) {
Expand All @@ -605,46 +627,91 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable)
traverseproc traverse = Py_TYPE(op)->tp_traverse;
_PyObject_ASSERT_WITH_MSG(op, gc_get_refs(gc) > 0,
"refcount is too small");
// NOTE: visit_reachable may change gc->_gc_next when
// young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before!
(void) traverse(op,
visit_reachable,
(void *)young);
// relink gc_prev to prev element.
_PyGCHead_SET_PREV(gc, prev);
// gc is not COLLECTING state after here.
gc_clear_collecting(gc);
prev = gc;
// NOTE: visit_reachable may change gc->_gc_next when
// young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before!
(void) traverse(op,
visit_reachable,
(void *)&state);
while (state.top > 0) {
op = FROM_GC(state.stack[--state.top]);
(void) Py_TYPE(op)->tp_traverse(op,
visit_reachable,
(void *)&state);
}
}
else {
/* This *may* be unreachable. To make progress,
* assume it is. gc isn't directly reachable from
* any object we've already traversed, but may be
* reachable from an object we haven't gotten to yet.
* visit_reachable will eventually move gc back into
* young if that's so, and we'll see it again.
* visit_reachable will eventually clear the flag
* if that's so.
*/
// Move gc to unreachable.
// No need to gc->next->prev = prev because it is single linked.
prev->_gc_next = gc->_gc_next;

// We can't use gc_list_append() here because we use
// NEXT_MASK_UNREACHABLE here.
PyGC_Head *last = GC_PREV(unreachable);
// NOTE: Since all objects in unreachable set has
// NEXT_MASK_UNREACHABLE flag, we set it unconditionally.
// But this may pollute the unreachable list head's 'next' pointer
// too. That's semantically senseless but expedient here - the
// damage is repaired when this function ends.
last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc);
_PyGCHead_SET_PREV(gc, last);
gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable);
unreachable->_gc_prev = (uintptr_t)gc;
_PyGCHead_SET_PREV(gc, prev);
gc->_gc_next |= NEXT_MASK_UNREACHABLE;
state.unreachable++;
}
gc = (PyGC_Head*)prev->_gc_next;
prev = gc;
gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE);
}
// young->_gc_prev must be last element remained in the list.
young->_gc_prev = (uintptr_t)prev;

/* Move the objects that still have the flag to unreachable. Look for
* them from both ends of the list at once and stop when all of them are
* found. The links between consecutive ones are already those of the
* unreachable list.
*/
PyGC_Head *front = unreachable; // last object taken from the front
PyGC_Head *back = unreachable; // first object taken from the back
PyGC_Head *fwd = GC_NEXT(young);
PyGC_Head *bwd = prev;
while (state.unreachable > 0) {
if (fwd->_gc_next & NEXT_MASK_UNREACHABLE) {
prev = GC_PREV(fwd);
// NOTE: This may pollute the unreachable list head's 'next'
// pointer with the flag. It is repaired below.
front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)fwd);
_PyGCHead_SET_PREV(fwd, front);
do {
front = fwd;
fwd = (PyGC_Head*)(fwd->_gc_next & ~NEXT_MASK_UNREACHABLE);
state.unreachable--;
} while (fwd->_gc_next & NEXT_MASK_UNREACHABLE);
prev->_gc_next = (uintptr_t)fwd;
_PyGCHead_SET_PREV(fwd, prev);
if (state.unreachable == 0) {
break;
}
}
else {
fwd = GC_NEXT(fwd);
}
if (bwd->_gc_next & NEXT_MASK_UNREACHABLE) {
PyGC_Head *last = bwd;
PyGC_Head *next = (PyGC_Head*)(bwd->_gc_next & ~NEXT_MASK_UNREACHABLE);
do {
gc = bwd;
bwd = GC_PREV(bwd);
state.unreachable--;
} while (bwd->_gc_next & NEXT_MASK_UNREACHABLE);
bwd->_gc_next = (uintptr_t)next;
_PyGCHead_SET_PREV(next, bwd);

last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back);
_PyGCHead_SET_PREV(back, last);
back = gc;
}
else {
bwd = GC_PREV(bwd);
}
}
front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back);
_PyGCHead_SET_PREV(back, front);
// don't let the pollution of the list head's next pointer leak
unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE;
}
Expand Down Expand Up @@ -1179,37 +1246,15 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
/* Leave everything reachable from outside base in base, and move
* everything else (in base) to unreachable.
*
* NOTE: This used to move the reachable objects into a reachable
* set instead. But most things usually turn out to be reachable,
* so it's more efficient to move the unreachable things. It "sounds slick"
* to move the unreachable objects, until you think about it - the reason it
* pays isn't actually obvious.
*
* Suppose we create objects A, B, C in that order. They appear in the young
* generation in the same order. If B points to A, and C to B, and C is
* reachable from outside, then the adjusted refcounts will be 0, 0, and 1
* respectively.
*
* When move_unreachable finds A, A is moved to the unreachable list. The
* same for B when it's first encountered. Then C is traversed, B is moved
* _back_ to the reachable list. B is eventually traversed, and then A is
* moved back to the reachable list.
*
* So instead of not moving at all, the reachable objects B and A are moved
* twice each. Why is this a win? A straightforward algorithm to move the
* reachable objects instead would move A, B, and C once each.
*
* The key is that this dance leaves the objects in order C, B, A - it's
* reversed from the original order. On all _subsequent_ scans, none of
* them will move. Since most objects aren't in cycles, this can save an
* unbounded number of moves across an unbounded number of later collections.
* It can cost more only the first time the chain is scanned.
*
* Drawback: move_unreachable is also used to find out what's still trash
* after finalizers may resurrect objects. In _that_ case most unreachable
* objects will remain unreachable, so it would be more efficient to move
* the reachable objects instead. But this is a one-time cost, probably not
* worth complicating the code to speed just a little.
* When move_unreachable finds A, A is assumed to be unreachable. The
* same for B when it's first encountered. Then C is traversed, B is
* found to be reachable and is traversed, and then A is. None of them
* moves: the list stays in the order A, B, C.
*/
gc_list_init(unreachable);
move_unreachable(base, unreachable); // gc_prev is pointer again
Expand Down
Loading