Skip to content

BUG: Require fiftyone_pipeline_core at the version that holds the names four packages import - #257

Draft
jwrosewell wants to merge 2 commits into
rcfrom
fix/package-requirement-rows
Draft

jwrosewell wants to merge 2 commits into
rcfrom
fix/package-requirement-rows

Conversation

@jwrosewell

Copy link
Copy Markdown
Contributor

Problem

The unit tests of fiftyone_pipeline_engines fail on rc as it stands, in tests/test_package_requirements.py, so every full build of rc stops in the Pipeline's unit tests.

  • Run 44935249761 built rc at d85e95b with nothing changed but its own workflow file. Its job Ubuntu_Python_3.12 gave "8 failed, 130 passed" for the engines package.
  • On today's rc, 60e1b5e, the same 8 fail here (Windows, Python 3.12.10): 8 failed, 132 passed.
AssertionError: 'MissingPropertyReason' not found in {'MissingPropertyService'}
AssertionError: 'OnPremiseMissingPropertyService' not found in {'MissingPropertyService'}
AssertionError: 'CloudMissingPropertyService' not found in {'OnPremiseMissingPropertyService', 'CloudData'}
AssertionError: False is not true : device-detection/fiftyone_devicedetection_onpremise does not import OnPremiseMissingPropertyService from fiftyone_pipeline_engines.missingproperty_service

The nightly on main builds rc as well, and will show the same failure as soon as its Configure job passes again. With the next release the test reaches main itself.

Cause

  • #221, merged on 16 November, added MissingPropertyReason and OnPremiseMissingPropertyService to fiftyone_pipeline_engines.missingproperty_service, and CloudMissingPropertyService to fiftyone_pipeline_cloudrequestengine.clouddata.
  • #246, merged on 19 November, gave the released missing property messages again and removed all three.
  • #231, merged on 21 November, added the test. Its four rows, and the comments on four requirements in setup.py files, name those three classes. So the test has failed from the commit that added it.

The test's purpose still holds. A package that imports a name from another package here must require that package at the version the name came in at, or upgrading it alone leaves an older copy of the other in place and the import fails.

What the rows should be

I compared every from <package>.<module> import <name> in the source of each published package on rc with the module the named package holds at its latest release tag. There are 77 such imports between packages. Four name something the latest release does not hold, and all four are in fiftyone_pipeline_core, whose latest release is 4.5.43 (tag fiftyone_pipeline_core/4.5.43).

Importing package Name Module Requirement on rc
fiftyone_pipeline_engines _contents_property fiftyone_pipeline_core.elementdata fiftyone_pipeline_core, no minimum
fiftyone_devicedetection_cloud _CheckedDictWrites fiftyone_pipeline_core.elementdata none of its own
fiftyone_devicedetection_onpremise ElementDataPool fiftyone_pipeline_core.elementdata_pool, a new module >=4.5.42
fiftyone_ipintelligence_onpremise ElementDataPool fiftyone_pipeline_core.elementdata_pool >=4.5.42

So each of the four can be installed today beside a core that it cannot import from. ci/write-package-versions.ps1 gives 4.5.44 as the version this checkout releases core at.

What changed

  1. tests/test_package_requirements.py has the four rows above in place of the four that named removed classes, each at 4.5.44, and PACKAGE_FOLDERS names the core package. The test's code is unchanged.
  2. The four importing packages require fiftyone_pipeline_core>=4.5.44 in setup.py, each with a comment naming what it imports. For fiftyone_devicedetection_cloud that is a new requirement of its own.
  3. Four other requirements keep the versions Require the version of each package that holds the names imported #231 gave them, and only their comments change, because the comments named classes that do not exist: fiftyone_pipeline_engines>=4.5.44 in the Cloud Request Engine and in both on-premise packages, and fiftyone_pipeline_cloudrequestengine>=4.5.45 in fiftyone_devicedetection_cloud. Each comment now says "The release this package is built and tested with." The second commit's message says three where four is right.
  4. pipeline/fiftyone_pipeline_core/tox.ini lists -e../fiftyone_pipeline_core/ among its dependencies, as Require the version of each package that holds the names imported #231 did for the engines package in its own tox.ini. The core package's test environment installs the engines package from the checkout, which now requires a core that no package index has, so core has to come from the checkout before its wheel is built. The wheel replaces it before the tests run.

No source file of any package changes.

Before you upgrade

Installing or upgrading fiftyone_pipeline_engines, fiftyone_devicedetection_cloud, fiftyone_devicedetection_onpremise or fiftyone_ipintelligence_onpremise at this release brings fiftyone_pipeline_core 4.5.44 or later with it. Nothing else changes.

A question for the review

The four requirements in point 3 were raised by #231 for names that are gone, and nothing imported today needs those versions. I left them where they are, because lowering a requirement is a decision of its own and a higher minimum only asks for packages that are released together anyway. If the rule is that a minimum stays only while a row of the test gives its reason, they go back to what the last release has: no minimum on fiftyone_pipeline_engines in the Cloud Request Engine, >=4.5.42 in the on-premise packages, and no requirement on the Cloud Request Engine in fiftyone_devicedetection_cloud.

How it was verified

Windows, Python 3.12.10, with COMMON_CI naming a checkout of common-ci's main and the package versions written by ci/write-package-versions.ps1 (core 4.5.44, engines 4.5.44, Cloud Request Engine 4.5.45).

  • First commit, a5bcbcf, the rows only: pytest tests/test_package_requirements.py gives 4 failed, 3 passed and 8 subtests passed. The imports and the modules holding the names pass for all four rows. The four failures are the requirements, for example device-detection/fiftyone_devicedetection_onpremise requires fiftyone_pipeline_core >=4.5.42, below the 4.5.44 that holds ElementDataPool.
  • Second commit, 7aaa1b2: the same file gives 3 passed and 12 subtests passed.
  • tox -e py312 in pipeline/fiftyone_pipeline_engines: 132 passed, 0 failed, where rc gives 8 failed and 132 passed.
  • tox -e py312 in pipeline/fiftyone_pipeline_core: the environment is built and the tests run, giving the 11 failures rc has in that package and 616 passed. Those 11 have another cause, fixed in #256.
  • The same environment with the new tox.ini line taken out is not built: ERROR: No matching distribution found for fiftyone_pipeline_core>=4.5.44 (from fiftyone-pipeline-engines).
  • tox -e py312 --notest in device-detection/fiftyone_devicedetection_cloud builds its environment with fiftyone_pipeline_cloudrequestengine>=4.5.45 fiftyone_pipeline_core>=4.5.44, both from the checkout.
  • Invoke-Pester ci/write-package-versions.Tests.ps1: 4 passed.
  • Not run here: the on-premise packages' environments, which need the native build, and Linux and macOS. Every tox.ini that installs the engines package from the checkout already lists core from the checkout, apart from core's own, which this changes.

What remains

… hold

The four rows of the requirements test named MissingPropertyReason,
CloudMissingPropertyService and OnPremiseMissingPropertyService in the
engines and Cloud Request Engine packages. Those classes were removed
before the test merged, so 8 of its checks fail on every build.

The rows are now the four names a package here imports from another
that the other's last release does not hold, all of them in
fiftyone_pipeline_core, whose last release is 4.5.43: _contents_property,
_CheckedDictWrites and, twice, ElementDataPool.

The imports and the modules holding the names pass. The requirements
fail, as none of the four packages requires fiftyone_pipeline_core at
4.5.44.
…es four packages import

Four packages import a name from fiftyone_pipeline_core that its last
release, 4.5.43, does not hold, and none required the version that does.
Upgrading one of them alone left the older core in place, and the import
failed.

fiftyone_pipeline_engines (_contents_property),
fiftyone_devicedetection_cloud (_CheckedDictWrites) and the two
on-premise packages (ElementDataPool) now require
fiftyone_pipeline_core>=4.5.44.

The comments on three other requirements named classes that no longer
exist. The versions they require are unchanged, and each comment now
says only that it is the release the package is built and tested with.

The core package's tox.ini takes core from the checkout before its
wheel is built, as the engines package does, because the engines package
it installs now requires a core that a package index may not have yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant