From d7de1165dc61583ed76cba22e702393a4a751807 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 22:40:02 +0100 Subject: [PATCH 1/7] gh-159137: Keep surviving objects in order in the garbage collector --- InternalDocs/garbage_collector.md | 66 ++--- ...10-10-00-00-01.gh-issue-159137.gcOrder.rst | 4 + Python/gc.c | 251 ++++++++++++------ 3 files changed, 203 insertions(+), 118 deletions(-) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 0ef45ff8e02bc5f..593eb0853828db6 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -260,8 +260,10 @@ ended having `gc_ref == 0` but is referenced still by the `link_1` object that is reachable from the outside. To obtain the set of objects that are really unreachable, the garbage collector re-scans the container objects using the `tp_traverse` slot; this time with a different traverse function that marks objects with -`gc_ref == 0` as "tentatively unreachable" and then moves them to the -tentatively unreachable list. The following image depicts the state of the lists in a +`gc_ref == 0` as "tentatively unreachable". The figures show these objects in a +tentatively unreachable list of their own. The implementation only sets the +`NEXT_MASK_UNREACHABLE` flag on them and leaves them where they are until the scan +ends. The following image depicts the state of the lists in a moment when the GC processed the `link_3` and `link_4` objects but has not processed `link_1` and `link_2` yet. @@ -275,13 +277,12 @@ already in what will become the reachable list): When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are -reachable from it, moving them to the end of the list of reachable objects (where -they started originally) and setting its `gc_ref` field to 1. This is what happens -to `link_2` and `link_3` below as they are reachable from `link_1`. From the +reachable from it. Those that the scan has not processed yet get their `gc_ref` +field set to 1, so that the GC knows they are reachable when it gets to them. This is +what happens to `link_2` below as it is reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` -the GC knows that `link_3` is reachable after all, so it is moved back to the -original list and its `gc_ref` field is set to 1 so that if the GC visits it again, -it will know that it's reachable. To avoid visiting an object twice, the GC marks all +the GC knows that `link_3` is reachable after all, so it stops being tentatively +unreachable. To avoid visiting an object twice, the GC marks all objects that have already been visited once (by unsetting the `PREV_MASK_COLLECTING` flag) so that if an object that has already been processed is referenced by some other object, the GC does not process it twice. @@ -289,42 +290,43 @@ object, the GC does not process it twice. ![gc-image5](images/python-cyclic-gc-5-new-page.png) Notice that an object that was marked as "tentatively unreachable" and was later -moved back to the reachable list will be visited again by the garbage collector -as now all the references that the object has need to be processed as well. This -process is really a breadth first search over the object graph. Once all the objects -are scanned, the GC knows that all container objects in the tentatively unreachable -list are really unreachable and can thus be garbage collected. +found to be reachable has to be traversed as well, as now all the references that +the object has need to be processed. The scan has already passed it, so the GC puts it +in a small stack and traverses the objects in the stack before it moves on. Once all +the objects are scanned, the GC knows that all container objects that are still +tentatively unreachable are really unreachable and can thus be garbage collected. +Only then are they moved to the unreachable list. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage -the GC algorithms require. +the GC algorithms require. The stack has a fixed size: when it is full, an object +found to be reachable is moved to the right of the object being scanned instead, +where the scan gets to it again. -Why moving unreachable objects is better ----------------------------------------- - -It sounds logical to move the unreachable objects under the premise that most objects -are usually reachable, until you think about it: the reason it pays isn't actually -obvious. +Why reachable objects are not moved +----------------------------------- Suppose we create objects A, B, C in that order. They appear in the young generation in the same order. If B points to A, and C to B, and C is reachable from outside, then the adjusted refcounts after the first step of the algorithm runs will be 0, 0, and 1 respectively because the only reachable object from the outside is C. -When the next step of the algorithm finds A, A is moved to the unreachable list. The -same for B when it's first encountered. Then C is traversed, B is moved *back* to -the reachable list. B is eventually traversed, and then A is moved back to the reachable -list. - -So instead of not moving at all, the reachable objects B and A are each moved twice. -Why is this a win? A straightforward algorithm to move the reachable objects instead -would move A, B, and C once each. The key is that this dance leaves the objects in -order C, B, A - it's reversed from the original order. On all *subsequent* scans, -none of them will move. Since most objects aren't in cycles, this can save an -unbounded number of moves across an unbounded number of later collections. The only -time the cost can be higher is the first time the chain is scanned. +When the next step of the algorithm finds A, A is marked as tentatively unreachable. The +same for B when it's first encountered. Then C is traversed, B is found to be reachable +and is traversed from the stack, and then A is. The list is still A, B, C. + +The GC used to move A and B to a tentatively unreachable list, and then *back* to the +end of the reachable list. That dance left the objects in order C, B, A, and on all +*subsequent* scans none of them was tentatively unreachable again. But it left the list +in the order in which the objects are reached, and that order has nothing to do with +where the objects are in memory. Objects are added to the list as they are allocated, +so the order of the list starts close to the order in memory, and a walk of the list +is then a walk through memory. Once the list is in a different order, every step of +every walk of it is a cache miss, in this collection and in all the later ones. +Keeping the order costs a push and a pop each time an object such as B or A is found +to be reachable after the scan has passed it, which is much less. Destroying unreachable objects ============================== diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst new file mode 100644 index 000000000000000..bfadf46169920ea --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst @@ -0,0 +1,4 @@ +Speed up the garbage collector by keeping the objects that survive a +collection in their original order. They were left in the order in which the +collector reached them, which made every later walk of the generation miss the +CPU cache. diff --git a/Python/gc.c b/Python/gc.c index bb20dae5a6543fa..7e83561a1442df8 100644 --- a/Python/gc.c +++ b/Python/gc.c @@ -40,7 +40,9 @@ typedef struct _gc_runtime_state GCState; // Lowest bit of _gc_next is used for UNREACHABLE flag. // -// This flag represents the object is in unreachable list in move_unreachable() +// move_unreachable() sets this flag on the objects it assumes unreachable. +// While it runs they stay where they are in the list being scanned, so that +// list is not a normal list either; when it ends they are in unreachable list. // // Although this flag is used only in move_unreachable(), move_unreachable() // doesn't clear this flag to skip unnecessary iteration. @@ -49,6 +51,9 @@ typedef struct _gc_runtime_state GCState; // most gc_list_* functions for it. #define NEXT_MASK_UNREACHABLE (1) +// Number of objects visit_reachable() can queue for traversal. +#define REACHABLE_STACK_SIZE 1024 + #define AS_GC(op) _Py_AS_GC(op) #define FROM_GC(gc) _Py_FROM_GC(gc) @@ -192,14 +197,14 @@ _gc_next takes these values: NEXT_MASK_UNREACHABLE flag described below. NEXT_MASK_UNREACHABLE - move_unreachable() then moves objects not reachable (whether directly or - indirectly) from outside the generation into an "unreachable" set and - set this flag. + move_unreachable() sets this flag on objects not reachable (whether + directly or indirectly) from outside the generation, and moves them into + an "unreachable" set when its scan ends. Objects that are found to be reachable have gc_refs set to 1. - When this flag is set for the reachable object, the object must be in - "unreachable" set. - The flag is unset and the object is moved back to "reachable" set. + When this flag is set for the reachable object, the scan has already + passed it. The flag is unset and the object is traversed without being + moved. move_legacy_finalizers() will remove this flag from "unreachable" set. */ @@ -500,23 +505,31 @@ subtract_refs(PyGC_Head *containers) } } +/* State shared by move_unreachable() and visit_reachable(). */ +struct reachable_state { + PyGC_Head *tail; /* where to insert objects after the scanned one */ + PyGC_Head *first; /* at or before every object with the flag */ + Py_ssize_t unreachable; /* objects with NEXT_MASK_UNREACHABLE */ + Py_ssize_t top; + PyGC_Head *stack[REACHABLE_STACK_SIZE]; +}; + /* A traversal callback for move_unreachable. */ static int visit_reachable(PyObject *op, void *arg) { - PyGC_Head *reachable = arg; + struct reachable_state *state = arg; OBJECT_STAT_INC(object_visits); if (!_PyObject_IS_GC(op)) { return 0; } PyGC_Head *gc = AS_GC(op); - const Py_ssize_t gc_refs = gc_get_refs(gc); // Ignore objects in other generation. - // This also skips objects "to the left" of the current position in - // move_unreachable's scan of the 'young' list - they've already been - // traversed, and no longer have the PREV_MASK_COLLECTING flag. + // This also skips objects already known to be reachable: those that + // move_unreachable has traversed, is traversing, or has in the stack. + // They no longer have the PREV_MASK_COLLECTING flag. if (! gc_is_collecting(gc)) { return 0; } @@ -527,25 +540,35 @@ visit_reachable(PyObject *op, void *arg) if (gc->_gc_next & NEXT_MASK_UNREACHABLE) { /* This had gc_refs = 0 when move_unreachable got * to it, but turns out it's reachable after all. - * Move it back to move_unreachable's 'young' list, - * and move_unreachable will eventually get to it - * again. + * It is "to the left" of the scan and _gc_prev is + * a pointer again. Leave it where it is and let + * move_unreachable traverse it from the stack. */ - // Manually unlink gc from unreachable list because the list functions - // don't work right in the presence of NEXT_MASK_UNREACHABLE flags. - PyGC_Head *prev = GC_PREV(gc); PyGC_Head *next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); - _PyObject_ASSERT(FROM_GC(prev), - prev->_gc_next & NEXT_MASK_UNREACHABLE); - _PyObject_ASSERT(FROM_GC(next), - next->_gc_next & NEXT_MASK_UNREACHABLE); - prev->_gc_next = gc->_gc_next; // copy NEXT_MASK_UNREACHABLE - _PyGCHead_SET_PREV(next, prev); - - gc_list_append(gc, reachable); - gc_set_refs(gc, 1); + state->unreachable--; + if (state->top < REACHABLE_STACK_SIZE) { + gc->_gc_next = (uintptr_t)next; + gc_clear_collecting(gc); + state->stack[state->top++] = gc; + } + else { + /* No room. Move it to the right of the object being scanned + * instead, and move_unreachable will get to it again. + */ + PyGC_Head *prev = GC_PREV(gc); + prev->_gc_next = (prev->_gc_next & NEXT_MASK_UNREACHABLE) + | (uintptr_t)next; + _PyGCHead_SET_PREV(next, prev); + if (state->first == gc) { + state->first = next; + } + gc->_gc_next = state->tail->_gc_next; + state->tail->_gc_next = (uintptr_t)gc; + state->tail = gc; + gc_set_refs(gc, 1); + } } - else if (gc_refs == 0) { + else if (gc_get_refs(gc) == 0) { /* This is in move_unreachable's 'young' list, but * the traversal hasn't yet gotten to it. All * we need to do is tell move_unreachable that it's @@ -558,7 +581,8 @@ visit_reachable(PyObject *op, void *arg) * list, and move_unreachable will eventually get to it. */ else { - _PyObject_ASSERT_WITH_MSG(op, gc_refs > 0, "refcount is too small"); + _PyObject_ASSERT_WITH_MSG(op, gc_get_refs(gc) > 0, + "refcount is too small"); } return 0; } @@ -574,21 +598,35 @@ visit_reachable(PyObject *op, void *arg) * doubly linked list after this function. * But _gc_next in unreachable list has NEXT_MASK_UNREACHABLE flag. * So we can not gc_list_* functions for unreachable until we remove the flag. + * + * The objects that stay in young keep their relative order. The order is + * the order of allocation, which is close to the order in memory, and every + * walk of the list in this and in later collections is faster for it. */ static void move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) { + struct reachable_state state; + state.first = NULL; + state.unreachable = 0; + state.top = 0; // previous elem in the young list, used for restore gc_prev. PyGC_Head *prev = young; PyGC_Head *gc = GC_NEXT(young); - - /* Invariants: all objects "to the left" of us in young are reachable - * (directly or indirectly) from outside the young list as it was at entry. + // Number of objects scanned, and where the flag was set: for the first + // time since no object had it, and for the last time. + Py_ssize_t pos = 0; + Py_ssize_t first_pos = 0; + Py_ssize_t last_pos = 0; + PyGC_Head *last = NULL; + + /* Invariants: all objects "to the left" of us in young without + * NEXT_MASK_UNREACHABLE are reachable (directly or indirectly) from + * outside the young list as it was at entry. * - * All other objects from the original young "to the left" of us are in - * unreachable now, and have NEXT_MASK_UNREACHABLE. All objects to the - * left of us in 'young' now have been scanned, and no objects here - * or to the right have been scanned yet. + * All objects to the left of us in 'young' now have been scanned, + * their _gc_prev is a pointer again, and no objects here or to the + * right have been scanned yet. */ while (gc != young) { @@ -597,55 +635,105 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) * original 'young'. Mark it as such, and traverse * its pointers to find any other objects that may * be directly reachable from it. Note that the - * call to tp_traverse may append objects to young, - * so we have to wait until it returns to determine - * the next object to visit. + * call to tp_traverse may insert objects in young + * after gc, so we have to wait until it returns to + * determine the next object to visit. */ PyObject *op = FROM_GC(gc); traverseproc traverse = Py_TYPE(op)->tp_traverse; _PyObject_ASSERT_WITH_MSG(op, gc_get_refs(gc) > 0, "refcount is too small"); - // NOTE: visit_reachable may change gc->_gc_next when - // young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before! - (void) traverse(op, - visit_reachable, - (void *)young); // relink gc_prev to prev element. _PyGCHead_SET_PREV(gc, prev); // gc is not COLLECTING state after here. gc_clear_collecting(gc); + state.tail = gc; + (void) traverse(op, + visit_reachable, + (void *)&state); + while (state.top > 0) { + op = FROM_GC(state.stack[--state.top]); + traverse = Py_TYPE(op)->tp_traverse; + (void) traverse(op, + visit_reachable, + (void *)&state); + } prev = gc; + gc = (PyGC_Head*)gc->_gc_next; + pos++; } else { /* This *may* be unreachable. To make progress, * assume it is. gc isn't directly reachable from * any object we've already traversed, but may be * reachable from an object we haven't gotten to yet. - * visit_reachable will eventually move gc back into - * young if that's so, and we'll see it again. + * visit_reachable will eventually clear the flag + * if that's so. */ - // Move gc to unreachable. - // No need to gc->next->prev = prev because it is single linked. - prev->_gc_next = gc->_gc_next; - - // We can't use gc_list_append() here because we use - // NEXT_MASK_UNREACHABLE here. - PyGC_Head *last = GC_PREV(unreachable); - // NOTE: Since all objects in unreachable set has - // NEXT_MASK_UNREACHABLE flag, we set it unconditionally. - // But this may pollute the unreachable list head's 'next' pointer - // too. That's semantically senseless but expedient here - the - // damage is repaired when this function ends. - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); - _PyGCHead_SET_PREV(gc, last); - gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)gc; + // gc_refs is not needed anymore: relink gc_prev. + _PyGCHead_SET_PREV(gc, prev); + prev = gc; + if (state.unreachable++ == 0) { + state.first = gc; + first_pos = pos; + } + last = gc; + last_pos = pos++; + uintptr_t next = gc->_gc_next; + gc->_gc_next = (NEXT_MASK_UNREACHABLE | next); + gc = (PyGC_Head*)next; } - gc = (PyGC_Head*)prev->_gc_next; } // young->_gc_prev must be last element remained in the list. young->_gc_prev = (uintptr_t)prev; - // don't let the pollution of the list head's next pointer leak + + /* The objects that still have the flag are unreachable. Move each run + * of them to unreachable. The links inside a run are already the ones + * the unreachable list needs. + */ + if (state.unreachable == 0) { + return; + } + if (state.unreachable == last_pos - first_pos + 1) { + // Everything from 'first' to 'last' has the flag: it is one run. + gc = state.first; + prev = GC_PREV(gc); + PyGC_Head *next = (PyGC_Head*)(last->_gc_next & ~NEXT_MASK_UNREACHABLE); + prev->_gc_next = (uintptr_t)next; + _PyGCHead_SET_PREV(next, prev); + + unreachable->_gc_next = (uintptr_t)gc; + _PyGCHead_SET_PREV(gc, unreachable); + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); + unreachable->_gc_prev = (uintptr_t)last; + return; + } + last = unreachable; + gc = state.first; + while (state.unreachable > 0) { + if (!(gc->_gc_next & NEXT_MASK_UNREACHABLE)) { + gc = GC_NEXT(gc); + continue; + } + PyGC_Head *run = gc; + prev = GC_PREV(run); + PyGC_Head *end; + do { + end = gc; + gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); + state.unreachable--; + } while (gc->_gc_next & NEXT_MASK_UNREACHABLE); + prev->_gc_next = (uintptr_t)gc; + _PyGCHead_SET_PREV(gc, prev); + + // NOTE: This may pollute the unreachable list head's 'next' pointer + // with the flag. The damage is repaired below. + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)run); + _PyGCHead_SET_PREV(run, last); + last = end; + } + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); + unreachable->_gc_prev = (uintptr_t)last; unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE; } @@ -1181,35 +1269,26 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) { * * NOTE: This used to move the reachable objects into a reachable * set instead. But most things usually turn out to be reachable, - * so it's more efficient to move the unreachable things. It "sounds slick" - * to move the unreachable objects, until you think about it - the reason it - * pays isn't actually obvious. + * so it's more efficient to move the unreachable things. * * Suppose we create objects A, B, C in that order. They appear in the young * generation in the same order. If B points to A, and C to B, and C is * reachable from outside, then the adjusted refcounts will be 0, 0, and 1 * respectively. * - * When move_unreachable finds A, A is moved to the unreachable list. The - * same for B when it's first encountered. Then C is traversed, B is moved - * _back_ to the reachable list. B is eventually traversed, and then A is - * moved back to the reachable list. - * - * So instead of not moving at all, the reachable objects B and A are moved - * twice each. Why is this a win? A straightforward algorithm to move the - * reachable objects instead would move A, B, and C once each. - * - * The key is that this dance leaves the objects in order C, B, A - it's - * reversed from the original order. On all _subsequent_ scans, none of - * them will move. Since most objects aren't in cycles, this can save an - * unbounded number of moves across an unbounded number of later collections. - * It can cost more only the first time the chain is scanned. + * When move_unreachable finds A, A is assumed to be unreachable. The + * same for B when it's first encountered. Then C is traversed, B is + * found to be reachable and is traversed right away, and then A is. + * None of them moves. * - * Drawback: move_unreachable is also used to find out what's still trash - * after finalizers may resurrect objects. In _that_ case most unreachable - * objects will remain unreachable, so it would be more efficient to move - * the reachable objects instead. But this is a one-time cost, probably not - * worth complicating the code to speed just a little. + * This used to move A and B to the unreachable list, and then back to the + * end of the reachable list, which left the objects in order C, B, A. On + * all _subsequent_ scans none of them was assumed to be unreachable again. + * But that order is the order in which the objects are reached, and it has + * nothing to do with where the objects are in memory. Every walk of the + * list, in every later collection, then misses the cache at each step. + * That costs much more than finding B and A to be reachable after + * the fact each time. */ gc_list_init(unreachable); move_unreachable(base, unreachable); // gc_prev is pointer again From 3a73e715170d5453e8796c95304dcff10339b859 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 22:49:14 +0100 Subject: [PATCH 2/7] Drop the NEWS entry and the doc changes --- InternalDocs/garbage_collector.md | 66 +++++++++---------- ...10-10-00-00-01.gh-issue-159137.gcOrder.rst | 4 -- 2 files changed, 32 insertions(+), 38 deletions(-) delete mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 593eb0853828db6..0ef45ff8e02bc5f 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -260,10 +260,8 @@ ended having `gc_ref == 0` but is referenced still by the `link_1` object that is reachable from the outside. To obtain the set of objects that are really unreachable, the garbage collector re-scans the container objects using the `tp_traverse` slot; this time with a different traverse function that marks objects with -`gc_ref == 0` as "tentatively unreachable". The figures show these objects in a -tentatively unreachable list of their own. The implementation only sets the -`NEXT_MASK_UNREACHABLE` flag on them and leaves them where they are until the scan -ends. The following image depicts the state of the lists in a +`gc_ref == 0` as "tentatively unreachable" and then moves them to the +tentatively unreachable list. The following image depicts the state of the lists in a moment when the GC processed the `link_3` and `link_4` objects but has not processed `link_1` and `link_2` yet. @@ -277,12 +275,13 @@ already in what will become the reachable list): When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are -reachable from it. Those that the scan has not processed yet get their `gc_ref` -field set to 1, so that the GC knows they are reachable when it gets to them. This is -what happens to `link_2` below as it is reachable from `link_1`. From the +reachable from it, moving them to the end of the list of reachable objects (where +they started originally) and setting its `gc_ref` field to 1. This is what happens +to `link_2` and `link_3` below as they are reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` -the GC knows that `link_3` is reachable after all, so it stops being tentatively -unreachable. To avoid visiting an object twice, the GC marks all +the GC knows that `link_3` is reachable after all, so it is moved back to the +original list and its `gc_ref` field is set to 1 so that if the GC visits it again, +it will know that it's reachable. To avoid visiting an object twice, the GC marks all objects that have already been visited once (by unsetting the `PREV_MASK_COLLECTING` flag) so that if an object that has already been processed is referenced by some other object, the GC does not process it twice. @@ -290,43 +289,42 @@ object, the GC does not process it twice. ![gc-image5](images/python-cyclic-gc-5-new-page.png) Notice that an object that was marked as "tentatively unreachable" and was later -found to be reachable has to be traversed as well, as now all the references that -the object has need to be processed. The scan has already passed it, so the GC puts it -in a small stack and traverses the objects in the stack before it moves on. Once all -the objects are scanned, the GC knows that all container objects that are still -tentatively unreachable are really unreachable and can thus be garbage collected. -Only then are they moved to the unreachable list. +moved back to the reachable list will be visited again by the garbage collector +as now all the references that the object has need to be processed as well. This +process is really a breadth first search over the object graph. Once all the objects +are scanned, the GC knows that all container objects in the tentatively unreachable +list are really unreachable and can thus be garbage collected. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage -the GC algorithms require. The stack has a fixed size: when it is full, an object -found to be reachable is moved to the right of the object being scanned instead, -where the scan gets to it again. +the GC algorithms require. -Why reachable objects are not moved ------------------------------------ +Why moving unreachable objects is better +---------------------------------------- + +It sounds logical to move the unreachable objects under the premise that most objects +are usually reachable, until you think about it: the reason it pays isn't actually +obvious. Suppose we create objects A, B, C in that order. They appear in the young generation in the same order. If B points to A, and C to B, and C is reachable from outside, then the adjusted refcounts after the first step of the algorithm runs will be 0, 0, and 1 respectively because the only reachable object from the outside is C. -When the next step of the algorithm finds A, A is marked as tentatively unreachable. The -same for B when it's first encountered. Then C is traversed, B is found to be reachable -and is traversed from the stack, and then A is. The list is still A, B, C. - -The GC used to move A and B to a tentatively unreachable list, and then *back* to the -end of the reachable list. That dance left the objects in order C, B, A, and on all -*subsequent* scans none of them was tentatively unreachable again. But it left the list -in the order in which the objects are reached, and that order has nothing to do with -where the objects are in memory. Objects are added to the list as they are allocated, -so the order of the list starts close to the order in memory, and a walk of the list -is then a walk through memory. Once the list is in a different order, every step of -every walk of it is a cache miss, in this collection and in all the later ones. -Keeping the order costs a push and a pop each time an object such as B or A is found -to be reachable after the scan has passed it, which is much less. +When the next step of the algorithm finds A, A is moved to the unreachable list. The +same for B when it's first encountered. Then C is traversed, B is moved *back* to +the reachable list. B is eventually traversed, and then A is moved back to the reachable +list. + +So instead of not moving at all, the reachable objects B and A are each moved twice. +Why is this a win? A straightforward algorithm to move the reachable objects instead +would move A, B, and C once each. The key is that this dance leaves the objects in +order C, B, A - it's reversed from the original order. On all *subsequent* scans, +none of them will move. Since most objects aren't in cycles, this can save an +unbounded number of moves across an unbounded number of later collections. The only +time the cost can be higher is the first time the chain is scanned. Destroying unreachable objects ============================== diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst deleted file mode 100644 index bfadf46169920ea..000000000000000 --- a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-00-00-01.gh-issue-159137.gcOrder.rst +++ /dev/null @@ -1,4 +0,0 @@ -Speed up the garbage collector by keeping the objects that survive a -collection in their original order. They were left in the order in which the -collector reached them, which made every later walk of the generation miss the -CPU cache. From 51dc0ca4bbd42860ea7a0dc84c8283d2317af0b6 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 22:57:01 +0100 Subject: [PATCH 3/7] Simplify --- Python/gc.c | 184 ++++++++++++++++++---------------------------------- 1 file changed, 62 insertions(+), 122 deletions(-) diff --git a/Python/gc.c b/Python/gc.c index 7e83561a1442df8..45876d7e042152b 100644 --- a/Python/gc.c +++ b/Python/gc.c @@ -41,8 +41,7 @@ typedef struct _gc_runtime_state GCState; // Lowest bit of _gc_next is used for UNREACHABLE flag. // // move_unreachable() sets this flag on the objects it assumes unreachable. -// While it runs they stay where they are in the list being scanned, so that -// list is not a normal list either; when it ends they are in unreachable list. +// They stay in the list being scanned until the scan ends. // // Although this flag is used only in move_unreachable(), move_unreachable() // doesn't clear this flag to skip unnecessary iteration. @@ -51,9 +50,6 @@ typedef struct _gc_runtime_state GCState; // most gc_list_* functions for it. #define NEXT_MASK_UNREACHABLE (1) -// Number of objects visit_reachable() can queue for traversal. -#define REACHABLE_STACK_SIZE 1024 - #define AS_GC(op) _Py_AS_GC(op) #define FROM_GC(gc) _Py_FROM_GC(gc) @@ -203,8 +199,7 @@ NEXT_MASK_UNREACHABLE Objects that are found to be reachable have gc_refs set to 1. When this flag is set for the reachable object, the scan has already - passed it. The flag is unset and the object is traversed without being - moved. + passed it. The flag is unset and the object is traversed in place. move_legacy_finalizers() will remove this flag from "unreachable" set. */ @@ -507,11 +502,10 @@ subtract_refs(PyGC_Head *containers) /* State shared by move_unreachable() and visit_reachable(). */ struct reachable_state { - PyGC_Head *tail; /* where to insert objects after the scanned one */ - PyGC_Head *first; /* at or before every object with the flag */ - Py_ssize_t unreachable; /* objects with NEXT_MASK_UNREACHABLE */ + PyGC_Head *young; + Py_ssize_t unreachable; /* objects with NEXT_MASK_UNREACHABLE */ Py_ssize_t top; - PyGC_Head *stack[REACHABLE_STACK_SIZE]; + PyGC_Head *stack[1024]; /* reachable objects to traverse */ }; /* A traversal callback for move_unreachable. */ @@ -525,11 +519,12 @@ visit_reachable(PyObject *op, void *arg) } PyGC_Head *gc = AS_GC(op); + const Py_ssize_t gc_refs = gc_get_refs(gc); // Ignore objects in other generation. - // This also skips objects already known to be reachable: those that - // move_unreachable has traversed, is traversing, or has in the stack. - // They no longer have the PREV_MASK_COLLECTING flag. + // This also skips objects "to the left" of the current position in + // move_unreachable's scan of the 'young' list - they've already been + // traversed, and no longer have the PREV_MASK_COLLECTING flag. if (! gc_is_collecting(gc)) { return 0; } @@ -540,35 +535,32 @@ visit_reachable(PyObject *op, void *arg) if (gc->_gc_next & NEXT_MASK_UNREACHABLE) { /* This had gc_refs = 0 when move_unreachable got * to it, but turns out it's reachable after all. - * It is "to the left" of the scan and _gc_prev is - * a pointer again. Leave it where it is and let - * move_unreachable traverse it from the stack. + * Leave it where it is, so that the list keeps its + * order, and let move_unreachable traverse it. */ - PyGC_Head *next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); + gc->_gc_next &= ~NEXT_MASK_UNREACHABLE; state->unreachable--; - if (state->top < REACHABLE_STACK_SIZE) { - gc->_gc_next = (uintptr_t)next; + if (state->top < (Py_ssize_t)Py_ARRAY_LENGTH(state->stack)) { gc_clear_collecting(gc); state->stack[state->top++] = gc; } else { - /* No room. Move it to the right of the object being scanned - * instead, and move_unreachable will get to it again. + /* No room. Move it to the end of 'young' and + * move_unreachable will eventually get to it again. */ + // Manually unlink gc because the list functions don't work + // right in the presence of NEXT_MASK_UNREACHABLE flags. PyGC_Head *prev = GC_PREV(gc); + PyGC_Head *next = GC_NEXT(gc); prev->_gc_next = (prev->_gc_next & NEXT_MASK_UNREACHABLE) | (uintptr_t)next; _PyGCHead_SET_PREV(next, prev); - if (state->first == gc) { - state->first = next; - } - gc->_gc_next = state->tail->_gc_next; - state->tail->_gc_next = (uintptr_t)gc; - state->tail = gc; + + gc_list_append(gc, state->young); gc_set_refs(gc, 1); } } - else if (gc_get_refs(gc) == 0) { + else if (gc_refs == 0) { /* This is in move_unreachable's 'young' list, but * the traversal hasn't yet gotten to it. All * we need to do is tell move_unreachable that it's @@ -581,8 +573,7 @@ visit_reachable(PyObject *op, void *arg) * list, and move_unreachable will eventually get to it. */ else { - _PyObject_ASSERT_WITH_MSG(op, gc_get_refs(gc) > 0, - "refcount is too small"); + _PyObject_ASSERT_WITH_MSG(op, gc_refs > 0, "refcount is too small"); } return 0; } @@ -599,26 +590,19 @@ visit_reachable(PyObject *op, void *arg) * But _gc_next in unreachable list has NEXT_MASK_UNREACHABLE flag. * So we can not gc_list_* functions for unreachable until we remove the flag. * - * The objects that stay in young keep their relative order. The order is - * the order of allocation, which is close to the order in memory, and every - * walk of the list in this and in later collections is faster for it. + * The objects left in young keep their order. It is the order in which they + * were allocated, so walking the list visits memory mostly in address order. */ static void move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) { struct reachable_state state; - state.first = NULL; + state.young = young; state.unreachable = 0; state.top = 0; // previous elem in the young list, used for restore gc_prev. PyGC_Head *prev = young; PyGC_Head *gc = GC_NEXT(young); - // Number of objects scanned, and where the flag was set: for the first - // time since no object had it, and for the last time. - Py_ssize_t pos = 0; - Py_ssize_t first_pos = 0; - Py_ssize_t last_pos = 0; - PyGC_Head *last = NULL; /* Invariants: all objects "to the left" of us in young without * NEXT_MASK_UNREACHABLE are reachable (directly or indirectly) from @@ -635,9 +619,9 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) * original 'young'. Mark it as such, and traverse * its pointers to find any other objects that may * be directly reachable from it. Note that the - * call to tp_traverse may insert objects in young - * after gc, so we have to wait until it returns to - * determine the next object to visit. + * call to tp_traverse may append objects to young, + * so we have to wait until it returns to determine + * the next object to visit. */ PyObject *op = FROM_GC(gc); traverseproc traverse = Py_TYPE(op)->tp_traverse; @@ -647,20 +631,17 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) _PyGCHead_SET_PREV(gc, prev); // gc is not COLLECTING state after here. gc_clear_collecting(gc); - state.tail = gc; + // NOTE: visit_reachable may change gc->_gc_next when + // young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before! (void) traverse(op, visit_reachable, (void *)&state); while (state.top > 0) { op = FROM_GC(state.stack[--state.top]); - traverse = Py_TYPE(op)->tp_traverse; - (void) traverse(op, + (void) Py_TYPE(op)->tp_traverse(op, visit_reachable, (void *)&state); } - prev = gc; - gc = (PyGC_Head*)gc->_gc_next; - pos++; } else { /* This *may* be unreachable. To make progress, @@ -670,70 +651,42 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) * visit_reachable will eventually clear the flag * if that's so. */ - // gc_refs is not needed anymore: relink gc_prev. _PyGCHead_SET_PREV(gc, prev); - prev = gc; - if (state.unreachable++ == 0) { - state.first = gc; - first_pos = pos; - } - last = gc; - last_pos = pos++; - uintptr_t next = gc->_gc_next; - gc->_gc_next = (NEXT_MASK_UNREACHABLE | next); - gc = (PyGC_Head*)next; + gc->_gc_next |= NEXT_MASK_UNREACHABLE; + state.unreachable++; } + prev = gc; + gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); } // young->_gc_prev must be last element remained in the list. young->_gc_prev = (uintptr_t)prev; - /* The objects that still have the flag are unreachable. Move each run - * of them to unreachable. The links inside a run are already the ones - * the unreachable list needs. - */ - if (state.unreachable == 0) { - return; - } - if (state.unreachable == last_pos - first_pos + 1) { - // Everything from 'first' to 'last' has the flag: it is one run. - gc = state.first; - prev = GC_PREV(gc); - PyGC_Head *next = (PyGC_Head*)(last->_gc_next & ~NEXT_MASK_UNREACHABLE); - prev->_gc_next = (uintptr_t)next; - _PyGCHead_SET_PREV(next, prev); - - unreachable->_gc_next = (uintptr_t)gc; - _PyGCHead_SET_PREV(gc, unreachable); - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)last; - return; - } - last = unreachable; - gc = state.first; - while (state.unreachable > 0) { - if (!(gc->_gc_next & NEXT_MASK_UNREACHABLE)) { - gc = GC_NEXT(gc); - continue; - } - PyGC_Head *run = gc; - prev = GC_PREV(run); - PyGC_Head *end; - do { - end = gc; - gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); + /* Move the objects that still have the flag to unreachable. */ + PyGC_Head *last = unreachable; + PyGC_Head *next; + for (gc = GC_NEXT(young); state.unreachable > 0; gc = next) { + next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); + if (gc->_gc_next & NEXT_MASK_UNREACHABLE) { + prev = GC_PREV(gc); + prev->_gc_next = (uintptr_t)next; + _PyGCHead_SET_PREV(next, prev); + + // We can't use gc_list_append() here because we use + // NEXT_MASK_UNREACHABLE here. + // NOTE: Since all objects in unreachable set has + // NEXT_MASK_UNREACHABLE flag, we set it unconditionally. + // But this may pollute the unreachable list head's 'next' pointer + // too. That's semantically senseless but expedient here - the + // damage is repaired when this function ends. + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); + _PyGCHead_SET_PREV(gc, last); + gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); + unreachable->_gc_prev = (uintptr_t)gc; + last = gc; state.unreachable--; - } while (gc->_gc_next & NEXT_MASK_UNREACHABLE); - prev->_gc_next = (uintptr_t)gc; - _PyGCHead_SET_PREV(gc, prev); - - // NOTE: This may pollute the unreachable list head's 'next' pointer - // with the flag. The damage is repaired below. - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)run); - _PyGCHead_SET_PREV(run, last); - last = end; - } - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)last; + } + } + // don't let the pollution of the list head's next pointer leak unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE; } @@ -1267,10 +1220,6 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) { /* Leave everything reachable from outside base in base, and move * everything else (in base) to unreachable. * - * NOTE: This used to move the reachable objects into a reachable - * set instead. But most things usually turn out to be reachable, - * so it's more efficient to move the unreachable things. - * * Suppose we create objects A, B, C in that order. They appear in the young * generation in the same order. If B points to A, and C to B, and C is * reachable from outside, then the adjusted refcounts will be 0, 0, and 1 @@ -1278,17 +1227,8 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) { * * When move_unreachable finds A, A is assumed to be unreachable. The * same for B when it's first encountered. Then C is traversed, B is - * found to be reachable and is traversed right away, and then A is. - * None of them moves. - * - * This used to move A and B to the unreachable list, and then back to the - * end of the reachable list, which left the objects in order C, B, A. On - * all _subsequent_ scans none of them was assumed to be unreachable again. - * But that order is the order in which the objects are reached, and it has - * nothing to do with where the objects are in memory. Every walk of the - * list, in every later collection, then misses the cache at each step. - * That costs much more than finding B and A to be reachable after - * the fact each time. + * found to be reachable and is traversed, and then A is. None of them + * moves: the list stays in the order A, B, C. */ gc_list_init(unreachable); move_unreachable(base, unreachable); // gc_prev is pointer again From 40e93649f181f074104f0a6b489b3b70696bf39f Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 22:58:40 +0100 Subject: [PATCH 4/7] Update the internal docs --- InternalDocs/garbage_collector.md | 61 +++++++++++++++---------------- 1 file changed, 29 insertions(+), 32 deletions(-) diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 0ef45ff8e02bc5f..70d0ec0fc4ef25e 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -260,8 +260,10 @@ ended having `gc_ref == 0` but is referenced still by the `link_1` object that is reachable from the outside. To obtain the set of objects that are really unreachable, the garbage collector re-scans the container objects using the `tp_traverse` slot; this time with a different traverse function that marks objects with -`gc_ref == 0` as "tentatively unreachable" and then moves them to the -tentatively unreachable list. The following image depicts the state of the lists in a +`gc_ref == 0` as "tentatively unreachable". The images show these objects in a +list of their own; the implementation sets the `NEXT_MASK_UNREACHABLE` flag on them +and leaves them where they are until the scan ends. The following image depicts the +state of the lists in a moment when the GC processed the `link_3` and `link_4` objects but has not processed `link_1` and `link_2` yet. @@ -275,13 +277,12 @@ already in what will become the reachable list): When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are -reachable from it, moving them to the end of the list of reachable objects (where -they started originally) and setting its `gc_ref` field to 1. This is what happens -to `link_2` and `link_3` below as they are reachable from `link_1`. From the +reachable from it. Those that the scan has not processed yet get their `gc_ref` +field set to 1, so that the GC knows they are reachable when it gets to them. This is +what happens to `link_2` below as it is reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` -the GC knows that `link_3` is reachable after all, so it is moved back to the -original list and its `gc_ref` field is set to 1 so that if the GC visits it again, -it will know that it's reachable. To avoid visiting an object twice, the GC marks all +the GC knows that `link_3` is reachable after all, so it is no longer tentatively +unreachable. To avoid visiting an object twice, the GC marks all objects that have already been visited once (by unsetting the `PREV_MASK_COLLECTING` flag) so that if an object that has already been processed is referenced by some other object, the GC does not process it twice. @@ -289,42 +290,38 @@ object, the GC does not process it twice. ![gc-image5](images/python-cyclic-gc-5-new-page.png) Notice that an object that was marked as "tentatively unreachable" and was later -moved back to the reachable list will be visited again by the garbage collector -as now all the references that the object has need to be processed as well. This -process is really a breadth first search over the object graph. Once all the objects -are scanned, the GC knows that all container objects in the tentatively unreachable -list are really unreachable and can thus be garbage collected. +found to be reachable has to be traversed as well, as now all the references that +the object has need to be processed. The scan has already passed it, so the GC puts +it in a small stack and traverses the objects in the stack before it moves on. Once +all the objects are scanned, the GC knows that all container objects that are still +tentatively unreachable are really unreachable and can thus be garbage collected. +They are then moved to the unreachable list. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage -the GC algorithms require. +the GC algorithms require. The stack has a fixed size: when it is full, an object +found to be reachable is moved to the end of the list instead, where the scan gets +to it again. -Why moving unreachable objects is better ----------------------------------------- - -It sounds logical to move the unreachable objects under the premise that most objects -are usually reachable, until you think about it: the reason it pays isn't actually -obvious. +Why reachable objects are not moved +----------------------------------- Suppose we create objects A, B, C in that order. They appear in the young generation in the same order. If B points to A, and C to B, and C is reachable from outside, then the adjusted refcounts after the first step of the algorithm runs will be 0, 0, and 1 respectively because the only reachable object from the outside is C. -When the next step of the algorithm finds A, A is moved to the unreachable list. The -same for B when it's first encountered. Then C is traversed, B is moved *back* to -the reachable list. B is eventually traversed, and then A is moved back to the reachable -list. - -So instead of not moving at all, the reachable objects B and A are each moved twice. -Why is this a win? A straightforward algorithm to move the reachable objects instead -would move A, B, and C once each. The key is that this dance leaves the objects in -order C, B, A - it's reversed from the original order. On all *subsequent* scans, -none of them will move. Since most objects aren't in cycles, this can save an -unbounded number of moves across an unbounded number of later collections. The only -time the cost can be higher is the first time the chain is scanned. +When the next step of the algorithm finds A, A is marked as tentatively unreachable. +The same for B when it's first encountered. Then C is traversed, B is found to be +reachable and is traversed from the stack, and then A is. The list is still A, B, C. + +Objects are added to the list as they are allocated, so the order of the list is +close to the order of the objects in memory, and a walk of the list goes through +memory mostly in address order. The collector walks the list several times in every +collection. Keeping the order keeps those walks friendly to the CPU cache, in this +collection and in all the later ones. Destroying unreachable objects ============================== From 514d0f40c52ac04f112ac5088f7666db528f4f6f Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 23:13:50 +0100 Subject: [PATCH 5/7] Move runs of unreachable objects at once --- Python/gc.c | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/Python/gc.c b/Python/gc.c index 45876d7e042152b..484827c04943df9 100644 --- a/Python/gc.c +++ b/Python/gc.c @@ -661,31 +661,31 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) // young->_gc_prev must be last element remained in the list. young->_gc_prev = (uintptr_t)prev; - /* Move the objects that still have the flag to unreachable. */ + /* Move the objects that still have the flag to unreachable. The links + * between consecutive ones are already those of the unreachable list. + */ PyGC_Head *last = unreachable; - PyGC_Head *next; - for (gc = GC_NEXT(young); state.unreachable > 0; gc = next) { - next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); - if (gc->_gc_next & NEXT_MASK_UNREACHABLE) { - prev = GC_PREV(gc); - prev->_gc_next = (uintptr_t)next; - _PyGCHead_SET_PREV(next, prev); - - // We can't use gc_list_append() here because we use - // NEXT_MASK_UNREACHABLE here. - // NOTE: Since all objects in unreachable set has - // NEXT_MASK_UNREACHABLE flag, we set it unconditionally. - // But this may pollute the unreachable list head's 'next' pointer - // too. That's semantically senseless but expedient here - the - // damage is repaired when this function ends. - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); - _PyGCHead_SET_PREV(gc, last); - gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)gc; + gc = GC_NEXT(young); + while (state.unreachable > 0) { + if (!(gc->_gc_next & NEXT_MASK_UNREACHABLE)) { + gc = GC_NEXT(gc); + continue; + } + prev = GC_PREV(gc); + // NOTE: This may pollute the unreachable list head's 'next' pointer + // with the flag. The damage is repaired when this function ends. + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); + _PyGCHead_SET_PREV(gc, last); + do { last = gc; + gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); state.unreachable--; - } + } while (gc->_gc_next & NEXT_MASK_UNREACHABLE); + prev->_gc_next = (uintptr_t)gc; + _PyGCHead_SET_PREV(gc, prev); } + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); + unreachable->_gc_prev = (uintptr_t)last; // don't let the pollution of the list head's next pointer leak unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE; } From 56186144eaa2c622d18299f1ab321a121214c4f6 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sat, 10 Oct 2026 23:54:07 +0100 Subject: [PATCH 6/7] Add a NEWS entry --- .../2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst new file mode 100644 index 000000000000000..95ae3265e154bc0 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst @@ -0,0 +1,3 @@ +Speed up the garbage collector on large heaps by keeping the objects that +survive a collection in the order in which they were allocated. +:func:`gc.get_objects` returns them in that order. From 6c08848505cbcfbe23e0d0edfa9736f04f3f2de8 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sun, 11 Oct 2026 02:47:48 +0100 Subject: [PATCH 7/7] Look for the unreachable objects from both ends of the list --- Python/gc.c | 70 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 48 insertions(+), 22 deletions(-) diff --git a/Python/gc.c b/Python/gc.c index 484827c04943df9..e159329d7ad1cb9 100644 --- a/Python/gc.c +++ b/Python/gc.c @@ -661,31 +661,57 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) // young->_gc_prev must be last element remained in the list. young->_gc_prev = (uintptr_t)prev; - /* Move the objects that still have the flag to unreachable. The links - * between consecutive ones are already those of the unreachable list. + /* Move the objects that still have the flag to unreachable. Look for + * them from both ends of the list at once and stop when all of them are + * found. The links between consecutive ones are already those of the + * unreachable list. */ - PyGC_Head *last = unreachable; - gc = GC_NEXT(young); + PyGC_Head *front = unreachable; // last object taken from the front + PyGC_Head *back = unreachable; // first object taken from the back + PyGC_Head *fwd = GC_NEXT(young); + PyGC_Head *bwd = prev; while (state.unreachable > 0) { - if (!(gc->_gc_next & NEXT_MASK_UNREACHABLE)) { - gc = GC_NEXT(gc); - continue; + if (fwd->_gc_next & NEXT_MASK_UNREACHABLE) { + prev = GC_PREV(fwd); + // NOTE: This may pollute the unreachable list head's 'next' + // pointer with the flag. It is repaired below. + front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)fwd); + _PyGCHead_SET_PREV(fwd, front); + do { + front = fwd; + fwd = (PyGC_Head*)(fwd->_gc_next & ~NEXT_MASK_UNREACHABLE); + state.unreachable--; + } while (fwd->_gc_next & NEXT_MASK_UNREACHABLE); + prev->_gc_next = (uintptr_t)fwd; + _PyGCHead_SET_PREV(fwd, prev); + if (state.unreachable == 0) { + break; + } + } + else { + fwd = GC_NEXT(fwd); } - prev = GC_PREV(gc); - // NOTE: This may pollute the unreachable list head's 'next' pointer - // with the flag. The damage is repaired when this function ends. - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); - _PyGCHead_SET_PREV(gc, last); - do { - last = gc; - gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); - state.unreachable--; - } while (gc->_gc_next & NEXT_MASK_UNREACHABLE); - prev->_gc_next = (uintptr_t)gc; - _PyGCHead_SET_PREV(gc, prev); - } - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)last; + if (bwd->_gc_next & NEXT_MASK_UNREACHABLE) { + PyGC_Head *last = bwd; + PyGC_Head *next = (PyGC_Head*)(bwd->_gc_next & ~NEXT_MASK_UNREACHABLE); + do { + gc = bwd; + bwd = GC_PREV(bwd); + state.unreachable--; + } while (bwd->_gc_next & NEXT_MASK_UNREACHABLE); + bwd->_gc_next = (uintptr_t)next; + _PyGCHead_SET_PREV(next, bwd); + + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back); + _PyGCHead_SET_PREV(back, last); + back = gc; + } + else { + bwd = GC_PREV(bwd); + } + } + front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back); + _PyGCHead_SET_PREV(back, front); // don't let the pollution of the list head's next pointer leak unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE; }