Skip to content

BUG: Refuse writes through a view of a finished Flow Data's Element Data - #256

Merged
jwrosewell merged 2 commits into
rcfrom
fix/finished-flow-data-view
Nov 22, 2026
Merged

jwrosewell merged 2 commits into
rcfrom
fix/finished-flow-data-view

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The unit tests of fiftyone_pipeline_core fail on rc as it stands, so every full build of rc, and of any pull request into it, stops in the Pipeline's unit tests.

  • Run 44935249761 built rc at d85e95b with nothing changed but its own workflow file. Its job Ubuntu_Python_3.12 gave "11 failed, 604 passed" for the core package.
  • On today's rc, 60e1b5e, the same tests fail here (Windows, Python 3.12.10, pytest -m "not live" tests in pipeline/fiftyone_pipeline_core): 11 failed, 616 passed.
SUBFAILED[[]=], [update], [setdefault], [|=], [del], [pop], [popitem], [clear] tests/test_finished_writes.py::FinishedFlowDataTests::test_the_element_data_dictionary_cannot_be_written_once_processed
SUBFAILED[data] tests/test_finished_writes.py::FinishedFlowDataTests::test_the_dictionaries_cannot_be_replaced_once_processed
FAILED tests/test_finished_writes.py::FinishedFlowDataTests::test_data_given_before_processing_is_read_only_after
FAILED tests/test_flowdata_element_data_list.py::ElementDataListTests::test_setting_data_replaces_the_whole_list_at_once

The nightly on main builds rc as well, and will show the same failure as soon as its Configure job passes again. With the next release the tests reach main itself.

Cause

Two changes merged into rc a day apart, each green on its own, and they do not agree about FlowData.data.

  • #250 keeps the Element Data in a linked list, and data gives an ElementDataView of it. Each read gives a new view, a view adds to the Flow Data when a key is set, and a removal is a TypeError.
  • #247 makes a Flow Data read only once processing has finished. It was written for data as a dict, and its __setattr__ wraps a replacement for data in a dict that refuses writes.

Put together, in flowdata.py at 60e1b5e8:

  1. A view's __setitem__ (lines 98 to 101) calls _add with no check, so flowdata.data["key"] = value, update and setdefault add to a finished Flow Data. Only set_element_data checks processed.
  2. A view has no |=, so flowdata.data |= {...} raises 'ElementDataView' object has no attribute '__ior__' whether the Flow Data has finished or not.
  3. A removal from a finished Flow Data raises "Element Data is not removed from a FlowData", which does not say that the Flow Data has finished.
  4. __setattr__ (lines 218 and 219) copies a replacement for data into a _FlowDataDict before the data property reads it. The copy is made by dict, which does not call the mapping's own items(), so the property no longer builds the list from the mapping it was given. That is what Keep a Flow Data's Element Data in a linked list, added with no lock #250's own test of a replacement checks.
  5. data gives a new view for each read, so the test that a refused replacement leaves the Flow Data holding what it held (assertIs(before, flowdata.data)) cannot pass.

What changed

All in pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py.

  1. ElementDataView.__setitem__ raises once the Flow Data has finished, with the message set_element_data gives: "This FlowData has finished processing, so ElementData cannot be added to it. A FlowData is read only once processing has finished." update and setdefault set keys through it, so they raise too.
  2. ElementDataView has |=, which adds as update does and returns the view. On a finished Flow Data it raises as above.
  3. A removal is still a TypeError, at any time, because Element Data is never removed. Once the Flow Data has finished the message adds "This FlowData has finished processing, so it is read only". del, pop, popitem and clear all go through it.
  4. FlowData.__setattr__ wraps only errors in a dict that refuses writes. A replacement for data goes to the data property as it was given. A replacement for data, errors or evidence on a finished Flow Data still raises, as before.
  5. Once processing has finished, data gives the same view for each read. Nothing can be added by then, so one view always shows the whole list. While Element Data can still be added, each read gives a view of its own, as Keep a Flow Data's Element Data in a linked list, added with no lock #250 designed, so a reader part way through a list is not upset by an add.

No test of #247 or #250 is changed.

Before you upgrade

Nothing changes for code that only reads a Flow Data, or that adds to it while it processes. Three things are different from rc today, and none of them is in a release yet:

How it was verified

Windows, Python 3.12.10, pytest, with the Pipeline packages installed from the checkout after ci/write-package-versions.ps1 and the JavaScript templates submodule checked out.

Commit fiftyone_pipeline_core, pytest -m "not live" tests
rc 60e1b5e 11 failed, 616 passed, 276 subtests passed
d9926b6, the new tests only The two test files this touches: 14 failed, 32 passed. The three new failures are the |=, one view and removal message tests
07497d9, the change 622 passed, 285 subtests passed, 0 failed
  • The first commit adds four tests to tests/test_flowdata_element_data_list.py. Three fail without the change: an in-place union through a view (TypeError: unsupported operand type(s) for |=), a finished Flow Data giving the same view each time, and a removal from a finished Flow Data naming the reason. The fourth passes before and after, and records that each read gives its own view while Element Data can be added.
  • With the change, the other Pipeline packages give what they gave on rc: fiftyone_pipeline_engines_fiftyone 155 passed, fiftyone_pipeline_cloudrequestengine 137 passed and 4 deselected, and fiftyone_pipeline_engines 132 passed with the same 8 failures in tests/test_package_requirements.py that rc has. Those 8 have another cause and are not changed here.
  • The one write through flowdata.data outside the tests, in cloudrequestengine.py line 793, runs inside the engine's process, before the Flow Data has finished.
  • Not run here: the Device Detection, IP Intelligence and Location packages, which need the native build and the data files, and Linux and macOS. This pull request's full build covers them.

What remains

  • fiftyone_pipeline_engines still fails 8 tests on rc, all in tests/test_package_requirements.py. Its rows name classes that a later change removed. That needs its own pull request.

…ed Flow Data gives one view

Three tests of the Element Data view fail today. An in-place union
through a view raises TypeError, as the view has no such operator. A
finished Flow Data gives a new view for each read, although nothing can
be added to it. A removal from a finished Flow Data does not say that
the Flow Data has finished processing.

A fourth, which passes today, records that each read gives its own view
while Element Data can still be added.
A Flow Data is read only once processing has finished, and its Element
Data is kept in a linked list read through views. The two did not meet:
a view added to a finished Flow Data, and a replacement for the list was
copied into a dict before the data property read it, so the property
never called the mapping's own items().

A view now refuses to set a key once the Flow Data has finished, with the
message every other write gives, and so update, setdefault and |= do too.
|= is new, and adds as update does. A removal is still a TypeError, as
Element Data is never removed, and says when the Flow Data has finished.

Setting data passes the mapping given to the data property as it is, and
only errors is kept in a dict that refuses writes.

A finished Flow Data gives the same view for each read, as nothing can be
added to it. While Element Data can be added, each read still gives a
view of its own.
@jwrosewell
jwrosewell marked this pull request as ready for review November 22, 2026 08:58
@jwrosewell
jwrosewell merged commit f432c30 into rc Nov 22, 2026
7 checks passed
@jwrosewell
jwrosewell deleted the fix/finished-flow-data-view branch November 22, 2026 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant