Skip to content

Reuse Element Data through a bounded pool for each Flow Element - #248

Merged
jwrosewell merged 9 commits into
rcfrom
remote/element-data-pools
Nov 21, 2026
Merged

jwrosewell merged 9 commits into
rcfrom
remote/element-data-pools

Conversation

@jwrosewell

Copy link
Copy Markdown
Contributor

The problem

Every request makes new Element Data for each Flow Element that adds some, and the garbage collector frees it later. The specification now asks implementations that can to reuse Element Data through an object pool, and requires code comments and documentation to say that a reference to Element Data kept after its Flow Data is closed is not supported.

This draft is stacked on the draft that makes writes to a finished Flow Data fail, branch remote/finished-writes-fail at eae137e, because a pooled instance has to come back open for writing, which needs that draft's record of when a Flow Element has finished. It retargets to rc once that draft merges.

The specification

From resource-cleanup.md, Reusing Element Data, in specifications#190 at a90ba516.

Where the language supports it, implementations SHOULD reuse Element Data instances through an object pool, to avoid the memory overhead of creating them for every request. An instance goes back to its pool only when the Flow Data that holds it is disposed and any lazy task writing it has finished, and an instance held by a cache never goes back to a pool, because the cache shares it with other requests. An instance taken from a pool starts with no values and open for writing.

Keeping a reference to Element Data after the Flow Data that holds it is disposed is not supported, because a pool may give the instance to another request. Code comments and documentation MUST say so, and implementations do not check for it when the Element Data is read.

From resource-cleanup.md, Flow Data.

Element Data instances that do not require cleanup MUST be left alone, apart from going back to an object pool as reusing Element Data describes, in order for result caching to work correctly.

In Python a Flow Data is disposed by FlowData.close(), which a with statement calls.

The change

At the head, 5bdd9b3.

  1. ElementDataPool, new in the pipeline core, keeps up to DEFAULT_MAXIMUM_INSTANCES, twice the number of processors, of the Element Data one Flow Element makes. take() hands out an instance an earlier request has finished with, after its _clear() has dropped its values and opened it for writing, or None when the pool holds none, and the Flow Element then makes a new one. give_back() keeps an instance unless the pool is full or the Flow Element that made it has a cache. Requests on several threads share the pool, so it takes a threading.Lock of its own. That lock guards state shared between requests, and no Flow Data or Element Data is locked.
  2. ElementData._clear() opens an instance for writing. ElementDataDictionary also drops its contents, with a new dictionary so that one returned earlier by as_dictionary() is unchanged. The Device Detection SwigData drops its native results, and the IP Intelligence SwigData its native results and the addresses it echoes.
  3. FlowData.close() closes each Element Data as before, then gives it back to the pool of the Flow Element that made it, if that element keeps one as _element_data_pool. An instance held under two keys goes back once, and a second close() does nothing, so one instance never reaches two requests. A Flow Data that is not closed gives nothing back, and its instances go to the garbage collector as before.
  4. The Device Detection and IP Intelligence on-premise engines, and the pipeline core's Sequence, JSON Bundler, JavaScript Builder and Set Header elements, take their Element Data from a pool. Both engines refuse a cache, and the four core elements are Flow Elements with no cache, so each of their instances goes back.
  5. The FlowData.close and ElementData docstrings say that a reference to Element Data kept after its Flow Data is closed is not supported, because another request may then change its values, and that reads are not checked for it.

An instance is cleared when it is taken, not when it goes back, so its values stay readable after close() until another request takes it. None of the elements adopted here writes its Element Data from a lazy task, so close() gives each back at once.

No existing public name changes. The new public names are ElementDataPool and DEFAULT_MAXIMUM_INSTANCES in fiftyone_pipeline_core.elementdata_pool, and the rest are private.

Flow Elements not adopted

  • The Cloud Request Engine and the cloud engines built on CloudEngine (Device Detection, IP Intelligence, country translation and hardware profile). The call to the cloud service is a lazy property that thread-safety.md lines 164 to 186 puts in a lazy task, and an instance goes back only once any lazy task writing it has finished, so these follow the lazy task pattern into this repository.
  • The derived property elements. Every derived property element in a Pipeline writes the one instance under derived that the first of them makes, so they are left for a change of their own.
  • The translation engines, whose subclasses choose their Element Data through _create_element_data, so each kind needs its own _clear() first.
  • The usage sharing element, which makes no Element Data.

Commits

  1. 4948144, the new tests alone, on the code of remote/finished-writes-fail.
  2. 5bdd9b3, the change.

Released tests

The tests of the last release, main at 612aa28, ran unaltered against this branch for each package whose code or tests it touches and for the two examples packages that use the adopted engines and elements. The names of the failing tests are the same as on rc at 2373d14 and on remote/finished-writes-fail at eae137e, package by package, with none new and none fixed.

Package rc 2373d14 remote/finished-writes-fail eae137e This branch
fiftyone_pipeline_core 480 passed, 1 failed 480 passed, 1 failed 480 passed, 1 failed
fiftyone_pipeline_engines 34 passed, 31 failed 34 passed, 31 failed 34 passed, 31 failed
fiftyone_devicedetection_onpremise 67 passed, 6 failed 67 passed, 6 failed 67 passed, 6 failed
fiftyone_ipintelligence_onpremise 52 passed, 1 failed 52 passed, 1 failed 52 passed, 1 failed
fiftyone_devicedetection_examples 32 passed, 6 failed, 1 error 32 passed, 6 failed, 1 error 32 passed, 6 failed, 1 error
fiftyone_ipintelligence_examples 26 passed, 4 failed, 1 skipped 26 passed, 4 failed, 1 skipped 26 passed, 4 failed, 1 skipped

The released tests that exercise what this touches, on this branch.

Released test file What it covers This branch
fiftyone_pipeline_core test_close.py Closing a Flow Data, and reading Element Data that needs no cleanup after it is closed 7 passed
fiftyone_pipeline_core test_core.py Flow Data and Element Data 18 passed
fiftyone_pipeline_core test_javascriptbuilder*.py (5 files) The JavaScript Builder element 208 passed
fiftyone_pipeline_core test_jsonbundler*.py (6 files) The JSON Bundler element 19 passed
fiftyone_pipeline_core test_setheader.py The Set Header element 12 passed, 1 failed as on rc
fiftyone_pipeline_core test_web.py, test_web_client_side.py, test_web_integration_elements.py The web integration, which closes its Flow Data, and the Sequence element 47 passed
fiftyone_devicedetection_onpremise test_devicedetection.py Many requests each closed by a with statement, and test_flowdata_close_releases_results, which reads after closing 33 passed, 5 failed as on rc
fiftyone_devicedetection_onpremise test_swig.py, test_properties.py, test_read_by_index.py, test_swig_data_index.py Reading SwigData 17 passed
fiftyone_ipintelligence_onpremise test_ipintelligence.py Requests and test_flowdata_close_releases_results 15 passed, 1 failed as on rc
fiftyone_ipintelligence_onpremise test_read_by_index.py, test_swig_data_decimal_places.py, test_swig_data_index.py Reading SwigData 15 passed

The failures that are the same on rc are test_setheader.py test_get_response_header_name_1_SetHeaderbrowserAccept_CH, five in test_devicedetection.py (test_excluded_property, test_missing_property_service_not_found_anywhere, test_missing_property_service_not_found_in_current, test_validate_data_false, test_validate_data_true), test_update_matched_useragent.py test_turned_off_leaves_user_agents_empty, test_ipintelligence.py test_missing_licence_key, the 31 in fiftyone_pipeline_engines (test_datafile_update_http.py, test_datafile_update_service.py, test_engines.py, test_licence_key_masking.py), and the cloud examples, run here with no resource key.

The red and the green

Both ran through a throwaway build of each commit, without macOS, on Ubuntu and Windows with Python 3.10 to 3.14.

  • Red: run 44190837294, whose workflow commit d1dd73f sits on 4948144. All ten Build and Test jobs failed in the unit test step, and only on the 15 new tests, 9 in pipeline core, 2 in engines, 2 in Device Detection on-premise and 2 in IP Intelligence on-premise.
  • Green: run 44192246193, whose workflow commit 99f85ef sits on 5bdd9b3. All ten Build and Test jobs passed.

Both throwaway branches are deleted.

Local counts

On Windows with Python 3.14.5, with the free Lite and ASN data files.

  • The 15 new tests: all 15 fail on the code of remote/finished-writes-fail and pass on this branch. In pipeline core, 9 tests count the Element Data the four core elements make over 200 requests (one each), the instances made with more Flow Data open at once than a pool's bound and from four threads at once, and check that a reused instance has none of the last request's values, is open for writing and then refuses writes, is given back once when closed twice, and that the instances of Flow Data that is not closed go to the garbage collector. In fiftyone_pipeline_engines, 2 tests check that an instance a cache holds is never reused and that one from an engine without a cache is. In each on-premise package, 2 tests count one SwigData over 200 requests and check that a reused instance gives the new request's results.
  • Each touched package's own tests on this branch. fiftyone_pipeline_core 580 passed. fiftyone_pipeline_engines 129 passed. fiftyone_devicedetection_onpremise 106 passed. fiftyone_ipintelligence_onpremise 81 passed. On remote/finished-writes-fail the same suites gave 571, 127, 104 and 79 passed, the difference being the new tests.

What remains

  1. Retarget to rc once remote/finished-writes-fail merges.
  2. Both on-premise packages now import ElementDataPool from fiftyone_pipeline_core and still ask for fiftyone_pipeline_core>=4.5.42. Their minimum moves to the core release that carries the pool when they are released, so that an on-premise package is never installed beside a core without it.
  3. Pooling for the cloud engines follows the lazy task pattern into this repository. Then close() gives an instance back only once any lazy task writing it has finished.
  4. Pooling for the derived property and translation elements, each in a change of its own.

Before you upgrade

  1. What changed. Once a Flow Data is closed, by close() or at the end of its with block, a later request reuses the Element Data it held from the Device Detection and IP Intelligence on-premise engines and from the Sequence, JSON Bundler, JavaScript Builder and Set Header elements. A reference to that Element Data kept after closing can then show another request's values. Closing a Flow Data a second time now does nothing.
  2. Why. specifications#190 at a90ba516, resource-cleanup.md lines 89 to 101 says "implementations SHOULD reuse Element Data instances through an object pool" and "Keeping a reference to Element Data after the Flow Data that holds it is disposed is not supported, because a pool may give the instance to another request. Code comments and documentation MUST say so". No released test changes.
  3. What to change. Read or copy the values you need before closing the Flow Data, and keep no Element Data from it afterwards. Values already read, such as an AspectPropertyValue or the JSON from the JSON Bundler, are separate objects and stay as they were. Code that never closes its Flow Data is unaffected, as its Element Data is left to the garbage collector.

jwrosewell and others added 7 commits November 17, 2026 22:16
Each package that defines a Flow Element gains a test that processes every
Flow Element the package and its examples define, then writes to its
Element Data and expects an error. It also checks that it covers every Flow
Element class the package's own modules define.

The pipeline core test also checks that a finished Flow Data refuses
Evidence, Element Data and errors, that a second process raises even when
process exceptions are suppressed, and that Flow Elements sharing one
Element Data write it while each processes.
FlowData raises once processing has finished when Evidence, Element Data or
an error is added, and a second process raises without adding an error.

ElementData records when the Flow Element that writes it has finished, and
then setting a public attribute, or a write method such as the derived
property data's set_value, raises. A Flow Element that shares the instance
opens it again while it processes. Names that start with an underscore hold
the data's own state, such as values kept once read, and closing the Flow
Data is not a write.

The checks read a field set when processing ends and take no lock.
The pipeline core test counts the Element Data the Sequence, JSON Bundler,
JavaScript Builder and Set Header elements make over 200 requests, each
closing its Flow Data, and expects one each rather than one for each
request. With more Flow Data open at once than a pool's bound, the
instances made stay at the bound and the pool keeps no more than it.
Threads processing at once make no more instances than there are threads.

A reused instance has none of the last request's values, is open for
writing while its Flow Element processes and refuses writes once it has
finished. Closing a Flow Data twice gives an instance back once, and the
instances of a Flow Data that is not closed go to the garbage collector.

The engines test checks that an instance a cache holds is never reused.
The Device Detection and IP Intelligence on-premise tests count the
results data the engines make over 200 requests and expect one, and check
that a reused instance gives the new request's results.
ElementDataPool, in the pipeline core, keeps up to twice the number of
processors of the Element Data one Flow Element makes, behind a lock of its
own, as requests on several threads share it. take() hands out an instance
an earlier request finished with, after ElementData._clear() has dropped
its values and opened it for writing, or None, when the element makes a new
one.

FlowData.close() gives each Element Data back to the pool of the Flow
Element that made it, after closing it. Data from an element with a cache
is never given back, as the cache may share it with other requests, and a
second close does nothing. A Flow Data that is not closed leaves its
instances to the garbage collector.

The Device Detection and IP Intelligence on-premise engines and the
Sequence, JSON Bundler, JavaScript Builder and Set Header elements take
their Element Data from a pool. The FlowData.close and ElementData
docstrings say that a reference to Element Data kept after its Flow Data is
closed is not supported, and reads are not checked for it.
The contents of ElementDataDictionary, AspectDataDictionary and CloudData,
and a hardware profile, which is itself a dict, can still be written once
the Flow Element that provides them has finished. Pipeline core now checks
each dict write method and that the contents stay a dict. Each package's
check over every Flow Element also writes an item, and the Device
Detection cloud check writes to a hardware profile in each way.
FlowData.close() read _element_data_pool from the FlowElement, so a
user's element with an attribute of that name holding something else
made close() fail. FlowElement.get_element_data_pool() returns that
attribute only when it is an ElementDataPool, and None otherwise, and
close() asks for the pool through it.

The tests for this are in the commit before this one. That commit's
message describes another change by mistake. Its test file is the one
this message describes.
The resource cleanup page of the specification says code comments and
documentation must say that keeping a reference to Element Data after
its Flow Data is disposed is not supported. The comments said so, and
now the Pipeline Core readme does too.

Two comments say why the code is as it is. The pool's cache check is on
the element, not the instance, and close() gives data back only once
its close has returned, so data written by a lazy task goes back only
after that task has finished.
Base automatically changed from remote/finished-writes-fail to rc November 21, 2026 16:54
rc now holds the read only Flow Data and finished Element Data from
#247, whose earlier commits this branch also carried, and the linked
list of Element Data from #250. The merge result is rc with this
branch's pool commits replayed on it, and nothing else from the
branch's copies of #247.

In flowdata.py, _give_back sits after rc's _GIVEN table. close() keeps
rc's call of the close method, which opens the Element Data itself, in
place of the branch's own _set_finished calls around it, and then gives
the instance back to its pool. The second close guard and the once only
give back for data held under two keys are kept, and they read the
Element Data through rc's data view.

The core readme keeps rc's web integration section and adds the
section saying Element Data is reused.
With the Element Data in a linked list, close() reads it through the
data view. One instance added under a second key is in the list twice,
and close() still gives it back to its pool once.
@jwrosewell
jwrosewell marked this pull request as ready for review November 21, 2026 19:28
@jwrosewell
jwrosewell merged commit ad8c2a7 into rc Nov 21, 2026
3 checks passed
@jwrosewell
jwrosewell deleted the remote/element-data-pools branch November 21, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants