diff --git a/InternalDocs/garbage_collector.md b/InternalDocs/garbage_collector.md index 0ef45ff8e02bc5f..70d0ec0fc4ef25e 100644 --- a/InternalDocs/garbage_collector.md +++ b/InternalDocs/garbage_collector.md @@ -260,8 +260,10 @@ ended having `gc_ref == 0` but is referenced still by the `link_1` object that is reachable from the outside. To obtain the set of objects that are really unreachable, the garbage collector re-scans the container objects using the `tp_traverse` slot; this time with a different traverse function that marks objects with -`gc_ref == 0` as "tentatively unreachable" and then moves them to the -tentatively unreachable list. The following image depicts the state of the lists in a +`gc_ref == 0` as "tentatively unreachable". The images show these objects in a +list of their own; the implementation sets the `NEXT_MASK_UNREACHABLE` flag on them +and leaves them where they are until the scan ends. The following image depicts the +state of the lists in a moment when the GC processed the `link_3` and `link_4` objects but has not processed `link_1` and `link_2` yet. @@ -275,13 +277,12 @@ already in what will become the reachable list): When the GC encounters an object which is reachable (`gc_ref > 0`), it traverses its references using the `tp_traverse` slot to find all the objects that are -reachable from it, moving them to the end of the list of reachable objects (where -they started originally) and setting its `gc_ref` field to 1. This is what happens -to `link_2` and `link_3` below as they are reachable from `link_1`. From the +reachable from it. Those that the scan has not processed yet get their `gc_ref` +field set to 1, so that the GC knows they are reachable when it gets to them. This is +what happens to `link_2` below as it is reachable from `link_1`. From the state in the previous image and after examining the objects referred to by `link_1` -the GC knows that `link_3` is reachable after all, so it is moved back to the -original list and its `gc_ref` field is set to 1 so that if the GC visits it again, -it will know that it's reachable. To avoid visiting an object twice, the GC marks all +the GC knows that `link_3` is reachable after all, so it is no longer tentatively +unreachable. To avoid visiting an object twice, the GC marks all objects that have already been visited once (by unsetting the `PREV_MASK_COLLECTING` flag) so that if an object that has already been processed is referenced by some other object, the GC does not process it twice. @@ -289,42 +290,38 @@ object, the GC does not process it twice. ![gc-image5](images/python-cyclic-gc-5-new-page.png) Notice that an object that was marked as "tentatively unreachable" and was later -moved back to the reachable list will be visited again by the garbage collector -as now all the references that the object has need to be processed as well. This -process is really a breadth first search over the object graph. Once all the objects -are scanned, the GC knows that all container objects in the tentatively unreachable -list are really unreachable and can thus be garbage collected. +found to be reachable has to be traversed as well, as now all the references that +the object has need to be processed. The scan has already passed it, so the GC puts +it in a small stack and traverses the objects in the stack before it moves on. Once +all the objects are scanned, the GC knows that all container objects that are still +tentatively unreachable are really unreachable and can thus be garbage collected. +They are then moved to the unreachable list. Pragmatically, it's important to note that no recursion is required by any of this, and neither does it in any other way require additional memory proportional to the number of objects, number of pointers, or the lengths of pointer chains. Apart from `O(1)` storage for internal C needs, the objects themselves contain all the storage -the GC algorithms require. +the GC algorithms require. The stack has a fixed size: when it is full, an object +found to be reachable is moved to the end of the list instead, where the scan gets +to it again. -Why moving unreachable objects is better ----------------------------------------- - -It sounds logical to move the unreachable objects under the premise that most objects -are usually reachable, until you think about it: the reason it pays isn't actually -obvious. +Why reachable objects are not moved +----------------------------------- Suppose we create objects A, B, C in that order. They appear in the young generation in the same order. If B points to A, and C to B, and C is reachable from outside, then the adjusted refcounts after the first step of the algorithm runs will be 0, 0, and 1 respectively because the only reachable object from the outside is C. -When the next step of the algorithm finds A, A is moved to the unreachable list. The -same for B when it's first encountered. Then C is traversed, B is moved *back* to -the reachable list. B is eventually traversed, and then A is moved back to the reachable -list. - -So instead of not moving at all, the reachable objects B and A are each moved twice. -Why is this a win? A straightforward algorithm to move the reachable objects instead -would move A, B, and C once each. The key is that this dance leaves the objects in -order C, B, A - it's reversed from the original order. On all *subsequent* scans, -none of them will move. Since most objects aren't in cycles, this can save an -unbounded number of moves across an unbounded number of later collections. The only -time the cost can be higher is the first time the chain is scanned. +When the next step of the algorithm finds A, A is marked as tentatively unreachable. +The same for B when it's first encountered. Then C is traversed, B is found to be +reachable and is traversed from the stack, and then A is. The list is still A, B, C. + +Objects are added to the list as they are allocated, so the order of the list is +close to the order of the objects in memory, and a walk of the list goes through +memory mostly in address order. The collector walks the list several times in every +collection. Keeping the order keeps those walks friendly to the CPU cache, in this +collection and in all the later ones. Destroying unreachable objects ============================== diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst new file mode 100644 index 000000000000000..95ae3265e154bc0 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2026-10-10-23-50-00.gh-issue-159137.gcOrdr.rst @@ -0,0 +1,3 @@ +Speed up the garbage collector on large heaps by keeping the objects that +survive a collection in the order in which they were allocated. +:func:`gc.get_objects` returns them in that order. diff --git a/Python/gc.c b/Python/gc.c index bb20dae5a6543fa..e159329d7ad1cb9 100644 --- a/Python/gc.c +++ b/Python/gc.c @@ -40,7 +40,8 @@ typedef struct _gc_runtime_state GCState; // Lowest bit of _gc_next is used for UNREACHABLE flag. // -// This flag represents the object is in unreachable list in move_unreachable() +// move_unreachable() sets this flag on the objects it assumes unreachable. +// They stay in the list being scanned until the scan ends. // // Although this flag is used only in move_unreachable(), move_unreachable() // doesn't clear this flag to skip unnecessary iteration. @@ -192,14 +193,13 @@ _gc_next takes these values: NEXT_MASK_UNREACHABLE flag described below. NEXT_MASK_UNREACHABLE - move_unreachable() then moves objects not reachable (whether directly or - indirectly) from outside the generation into an "unreachable" set and - set this flag. + move_unreachable() sets this flag on objects not reachable (whether + directly or indirectly) from outside the generation, and moves them into + an "unreachable" set when its scan ends. Objects that are found to be reachable have gc_refs set to 1. - When this flag is set for the reachable object, the object must be in - "unreachable" set. - The flag is unset and the object is moved back to "reachable" set. + When this flag is set for the reachable object, the scan has already + passed it. The flag is unset and the object is traversed in place. move_legacy_finalizers() will remove this flag from "unreachable" set. */ @@ -500,11 +500,19 @@ subtract_refs(PyGC_Head *containers) } } +/* State shared by move_unreachable() and visit_reachable(). */ +struct reachable_state { + PyGC_Head *young; + Py_ssize_t unreachable; /* objects with NEXT_MASK_UNREACHABLE */ + Py_ssize_t top; + PyGC_Head *stack[1024]; /* reachable objects to traverse */ +}; + /* A traversal callback for move_unreachable. */ static int visit_reachable(PyObject *op, void *arg) { - PyGC_Head *reachable = arg; + struct reachable_state *state = arg; OBJECT_STAT_INC(object_visits); if (!_PyObject_IS_GC(op)) { return 0; @@ -527,23 +535,30 @@ visit_reachable(PyObject *op, void *arg) if (gc->_gc_next & NEXT_MASK_UNREACHABLE) { /* This had gc_refs = 0 when move_unreachable got * to it, but turns out it's reachable after all. - * Move it back to move_unreachable's 'young' list, - * and move_unreachable will eventually get to it - * again. + * Leave it where it is, so that the list keeps its + * order, and let move_unreachable traverse it. */ - // Manually unlink gc from unreachable list because the list functions - // don't work right in the presence of NEXT_MASK_UNREACHABLE flags. - PyGC_Head *prev = GC_PREV(gc); - PyGC_Head *next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); - _PyObject_ASSERT(FROM_GC(prev), - prev->_gc_next & NEXT_MASK_UNREACHABLE); - _PyObject_ASSERT(FROM_GC(next), - next->_gc_next & NEXT_MASK_UNREACHABLE); - prev->_gc_next = gc->_gc_next; // copy NEXT_MASK_UNREACHABLE - _PyGCHead_SET_PREV(next, prev); - - gc_list_append(gc, reachable); - gc_set_refs(gc, 1); + gc->_gc_next &= ~NEXT_MASK_UNREACHABLE; + state->unreachable--; + if (state->top < (Py_ssize_t)Py_ARRAY_LENGTH(state->stack)) { + gc_clear_collecting(gc); + state->stack[state->top++] = gc; + } + else { + /* No room. Move it to the end of 'young' and + * move_unreachable will eventually get to it again. + */ + // Manually unlink gc because the list functions don't work + // right in the presence of NEXT_MASK_UNREACHABLE flags. + PyGC_Head *prev = GC_PREV(gc); + PyGC_Head *next = GC_NEXT(gc); + prev->_gc_next = (prev->_gc_next & NEXT_MASK_UNREACHABLE) + | (uintptr_t)next; + _PyGCHead_SET_PREV(next, prev); + + gc_list_append(gc, state->young); + gc_set_refs(gc, 1); + } } else if (gc_refs == 0) { /* This is in move_unreachable's 'young' list, but @@ -574,21 +589,28 @@ visit_reachable(PyObject *op, void *arg) * doubly linked list after this function. * But _gc_next in unreachable list has NEXT_MASK_UNREACHABLE flag. * So we can not gc_list_* functions for unreachable until we remove the flag. + * + * The objects left in young keep their order. It is the order in which they + * were allocated, so walking the list visits memory mostly in address order. */ static void move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) { + struct reachable_state state; + state.young = young; + state.unreachable = 0; + state.top = 0; // previous elem in the young list, used for restore gc_prev. PyGC_Head *prev = young; PyGC_Head *gc = GC_NEXT(young); - /* Invariants: all objects "to the left" of us in young are reachable - * (directly or indirectly) from outside the young list as it was at entry. + /* Invariants: all objects "to the left" of us in young without + * NEXT_MASK_UNREACHABLE are reachable (directly or indirectly) from + * outside the young list as it was at entry. * - * All other objects from the original young "to the left" of us are in - * unreachable now, and have NEXT_MASK_UNREACHABLE. All objects to the - * left of us in 'young' now have been scanned, and no objects here - * or to the right have been scanned yet. + * All objects to the left of us in 'young' now have been scanned, + * their _gc_prev is a pointer again, and no objects here or to the + * right have been scanned yet. */ while (gc != young) { @@ -605,46 +627,91 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable) traverseproc traverse = Py_TYPE(op)->tp_traverse; _PyObject_ASSERT_WITH_MSG(op, gc_get_refs(gc) > 0, "refcount is too small"); - // NOTE: visit_reachable may change gc->_gc_next when - // young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before! - (void) traverse(op, - visit_reachable, - (void *)young); // relink gc_prev to prev element. _PyGCHead_SET_PREV(gc, prev); // gc is not COLLECTING state after here. gc_clear_collecting(gc); - prev = gc; + // NOTE: visit_reachable may change gc->_gc_next when + // young->_gc_prev == gc. Don't do gc = GC_NEXT(gc) before! + (void) traverse(op, + visit_reachable, + (void *)&state); + while (state.top > 0) { + op = FROM_GC(state.stack[--state.top]); + (void) Py_TYPE(op)->tp_traverse(op, + visit_reachable, + (void *)&state); + } } else { /* This *may* be unreachable. To make progress, * assume it is. gc isn't directly reachable from * any object we've already traversed, but may be * reachable from an object we haven't gotten to yet. - * visit_reachable will eventually move gc back into - * young if that's so, and we'll see it again. + * visit_reachable will eventually clear the flag + * if that's so. */ - // Move gc to unreachable. - // No need to gc->next->prev = prev because it is single linked. - prev->_gc_next = gc->_gc_next; - - // We can't use gc_list_append() here because we use - // NEXT_MASK_UNREACHABLE here. - PyGC_Head *last = GC_PREV(unreachable); - // NOTE: Since all objects in unreachable set has - // NEXT_MASK_UNREACHABLE flag, we set it unconditionally. - // But this may pollute the unreachable list head's 'next' pointer - // too. That's semantically senseless but expedient here - the - // damage is repaired when this function ends. - last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc); - _PyGCHead_SET_PREV(gc, last); - gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable); - unreachable->_gc_prev = (uintptr_t)gc; + _PyGCHead_SET_PREV(gc, prev); + gc->_gc_next |= NEXT_MASK_UNREACHABLE; + state.unreachable++; } - gc = (PyGC_Head*)prev->_gc_next; + prev = gc; + gc = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE); } // young->_gc_prev must be last element remained in the list. young->_gc_prev = (uintptr_t)prev; + + /* Move the objects that still have the flag to unreachable. Look for + * them from both ends of the list at once and stop when all of them are + * found. The links between consecutive ones are already those of the + * unreachable list. + */ + PyGC_Head *front = unreachable; // last object taken from the front + PyGC_Head *back = unreachable; // first object taken from the back + PyGC_Head *fwd = GC_NEXT(young); + PyGC_Head *bwd = prev; + while (state.unreachable > 0) { + if (fwd->_gc_next & NEXT_MASK_UNREACHABLE) { + prev = GC_PREV(fwd); + // NOTE: This may pollute the unreachable list head's 'next' + // pointer with the flag. It is repaired below. + front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)fwd); + _PyGCHead_SET_PREV(fwd, front); + do { + front = fwd; + fwd = (PyGC_Head*)(fwd->_gc_next & ~NEXT_MASK_UNREACHABLE); + state.unreachable--; + } while (fwd->_gc_next & NEXT_MASK_UNREACHABLE); + prev->_gc_next = (uintptr_t)fwd; + _PyGCHead_SET_PREV(fwd, prev); + if (state.unreachable == 0) { + break; + } + } + else { + fwd = GC_NEXT(fwd); + } + if (bwd->_gc_next & NEXT_MASK_UNREACHABLE) { + PyGC_Head *last = bwd; + PyGC_Head *next = (PyGC_Head*)(bwd->_gc_next & ~NEXT_MASK_UNREACHABLE); + do { + gc = bwd; + bwd = GC_PREV(bwd); + state.unreachable--; + } while (bwd->_gc_next & NEXT_MASK_UNREACHABLE); + bwd->_gc_next = (uintptr_t)next; + _PyGCHead_SET_PREV(next, bwd); + + last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back); + _PyGCHead_SET_PREV(back, last); + back = gc; + } + else { + bwd = GC_PREV(bwd); + } + } + front->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)back); + _PyGCHead_SET_PREV(back, front); // don't let the pollution of the list head's next pointer leak unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE; } @@ -1179,37 +1246,15 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) { /* Leave everything reachable from outside base in base, and move * everything else (in base) to unreachable. * - * NOTE: This used to move the reachable objects into a reachable - * set instead. But most things usually turn out to be reachable, - * so it's more efficient to move the unreachable things. It "sounds slick" - * to move the unreachable objects, until you think about it - the reason it - * pays isn't actually obvious. - * * Suppose we create objects A, B, C in that order. They appear in the young * generation in the same order. If B points to A, and C to B, and C is * reachable from outside, then the adjusted refcounts will be 0, 0, and 1 * respectively. * - * When move_unreachable finds A, A is moved to the unreachable list. The - * same for B when it's first encountered. Then C is traversed, B is moved - * _back_ to the reachable list. B is eventually traversed, and then A is - * moved back to the reachable list. - * - * So instead of not moving at all, the reachable objects B and A are moved - * twice each. Why is this a win? A straightforward algorithm to move the - * reachable objects instead would move A, B, and C once each. - * - * The key is that this dance leaves the objects in order C, B, A - it's - * reversed from the original order. On all _subsequent_ scans, none of - * them will move. Since most objects aren't in cycles, this can save an - * unbounded number of moves across an unbounded number of later collections. - * It can cost more only the first time the chain is scanned. - * - * Drawback: move_unreachable is also used to find out what's still trash - * after finalizers may resurrect objects. In _that_ case most unreachable - * objects will remain unreachable, so it would be more efficient to move - * the reachable objects instead. But this is a one-time cost, probably not - * worth complicating the code to speed just a little. + * When move_unreachable finds A, A is assumed to be unreachable. The + * same for B when it's first encountered. Then C is traversed, B is + * found to be reachable and is traversed, and then A is. None of them + * moves: the list stays in the order A, B, C. */ gc_list_init(unreachable); move_unreachable(base, unreachable); // gc_prev is pointer again