Repository navigation
BUG: Refuse writes through a view of a finished Flow Data's Element Data - #256
Merged
Merged
Conversation
…ed Flow Data gives one view Three tests of the Element Data view fail today. An in-place union through a view raises TypeError, as the view has no such operator. A finished Flow Data gives a new view for each read, although nothing can be added to it. A removal from a finished Flow Data does not say that the Flow Data has finished processing. A fourth, which passes today, records that each read gives its own view while Element Data can still be added.
A Flow Data is read only once processing has finished, and its Element Data is kept in a linked list read through views. The two did not meet: a view added to a finished Flow Data, and a replacement for the list was copied into a dict before the data property read it, so the property never called the mapping's own items(). A view now refuses to set a key once the Flow Data has finished, with the message every other write gives, and so update, setdefault and |= do too. |= is new, and adds as update does. A removal is still a TypeError, as Element Data is never removed, and says when the Flow Data has finished. Setting data passes the mapping given to the data property as it is, and only errors is kept in a dict that refuses writes. A finished Flow Data gives the same view for each read, as nothing can be added to it. While Element Data can be added, each read still gives a view of its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The unit tests of
fiftyone_pipeline_corefail onrcas it stands, so every full build ofrc, and of any pull request into it, stops in the Pipeline's unit tests.rcat d85e95b with nothing changed but its own workflow file. Its job Ubuntu_Python_3.12 gave "11 failed, 604 passed" for the core package.rc, 60e1b5e, the same tests fail here (Windows, Python 3.12.10,pytest -m "not live" testsinpipeline/fiftyone_pipeline_core): 11 failed, 616 passed.The nightly on
mainbuildsrcas well, and will show the same failure as soon as its Configure job passes again. With the next release the tests reachmainitself.Cause
Two changes merged into
rca day apart, each green on its own, and they do not agree aboutFlowData.data.datagives anElementDataViewof it. Each read gives a new view, a view adds to the Flow Data when a key is set, and a removal is aTypeError.dataas a dict, and its__setattr__wraps a replacement fordatain a dict that refuses writes.Put together, in
flowdata.pyat 60e1b5e8:__setitem__(lines 98 to 101) calls_addwith no check, soflowdata.data["key"] = value,updateandsetdefaultadd to a finished Flow Data. Onlyset_element_datachecksprocessed.|=, soflowdata.data |= {...}raises'ElementDataView' object has no attribute '__ior__'whether the Flow Data has finished or not.__setattr__(lines 218 and 219) copies a replacement fordatainto a_FlowDataDictbefore thedataproperty reads it. The copy is made bydict, which does not call the mapping's ownitems(), so the property no longer builds the list from the mapping it was given. That is what Keep a Flow Data's Element Data in a linked list, added with no lock #250's own test of a replacement checks.datagives a new view for each read, so the test that a refused replacement leaves the Flow Data holding what it held (assertIs(before, flowdata.data)) cannot pass.What changed
All in
pipeline/fiftyone_pipeline_core/src/fiftyone_pipeline_core/flowdata.py.ElementDataView.__setitem__raises once the Flow Data has finished, with the messageset_element_datagives: "This FlowData has finished processing, so ElementData cannot be added to it. A FlowData is read only once processing has finished."updateandsetdefaultset keys through it, so they raise too.ElementDataViewhas|=, which adds asupdatedoes and returns the view. On a finished Flow Data it raises as above.TypeError, at any time, because Element Data is never removed. Once the Flow Data has finished the message adds "This FlowData has finished processing, so it is read only".del,pop,popitemandclearall go through it.FlowData.__setattr__wraps onlyerrorsin a dict that refuses writes. A replacement fordatagoes to thedataproperty as it was given. A replacement fordata,errorsorevidenceon a finished Flow Data still raises, as before.datagives the same view for each read. Nothing can be added by then, so one view always shows the whole list. While Element Data can still be added, each read gives a view of its own, as Keep a Flow Data's Element Data in a linked list, added with no lock #250 designed, so a reader part way through a list is not upset by an add.No test of #247 or #250 is changed.
Before you upgrade
Nothing changes for code that only reads a Flow Data, or that adds to it while it processes. Three things are different from
rctoday, and none of them is in a release yet:flowdata.dataafterprocess()has returned now raises, as Refuse writes to a finished Flow Data and to finished Element Data #247 intended for every write to a finished Flow Data. In the last releasedatais a plain dict and takes the write, so this is new to anyone upgrading, and it arrives with Refuse writes to a finished Flow Data and to finished Element Data #247 and not with this pull request alone.del,pop,popitemandclearonflowdata.dataraiseTypeErroras onrc, with a longer message once processing has finished.flowdata.data |= mappingworks before processing has finished.How it was verified
Windows, Python 3.12.10, pytest, with the Pipeline packages installed from the checkout after
ci/write-package-versions.ps1and the JavaScript templates submodule checked out.fiftyone_pipeline_core,pytest -m "not live" testsrc60e1b5e|=, one view and removal message teststests/test_flowdata_element_data_list.py. Three fail without the change: an in-place union through a view (TypeError: unsupported operand type(s) for |=), a finished Flow Data giving the same view each time, and a removal from a finished Flow Data naming the reason. The fourth passes before and after, and records that each read gives its own view while Element Data can be added.rc:fiftyone_pipeline_engines_fiftyone155 passed,fiftyone_pipeline_cloudrequestengine137 passed and 4 deselected, andfiftyone_pipeline_engines132 passed with the same 8 failures intests/test_package_requirements.pythatrchas. Those 8 have another cause and are not changed here.flowdata.dataoutside the tests, incloudrequestengine.pyline 793, runs inside the engine'sprocess, before the Flow Data has finished.What remains
fiftyone_pipeline_enginesstill fails 8 tests onrc, all intests/test_package_requirements.py. Its rows name classes that a later change removed. That needs its own pull request.