Skip to content

Budget 2026 slower data fixes, PR C of #1418: electricity at the ONS level, pension income by area - #1420

Merged
juaristi22 merged 47 commits into
mainfrom
uk-budget-2026-data-fixes-c-1418
Oct 30, 2026
Merged

juaristi22 merged 47 commits into
mainfrom
uk-budget-2026-data-fixes-c-1418

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

PR C of microcosm#1418 covers the Budget 2026 data fixes that weren't quick ones. It is stacked on PR A (#1419), whose branch is merged in. #1418 stays open.

Ready for review. Merging waits on CI at the head and on María. The feed is re-pinned onto PolicyEngine/chronicle#394's merge (53815b2), which carries SPI Table 3.10 and the October 2026 EFO. Anthony's two rounds, Vahid's review and the independent review are answered (commits 36 and 37), and so are the seven re-reviews since (commits 38, 39, 41 to 45).

A national built from commit 40 (staging only) stopped at certification on two £1m-and-over SPI rows. Whether to hold the new £1m-and-over other-income row out, as the ten other £1m-and-over cells are held, is María's call; the national is rebuilt after it.

A deadline: from 2026-11-27, test_no_committed_exclusion_is_about_to_lapse (commit 42) fails on every PR until orr.rail_government_support, which expires on 2026-12-11, is renewed or bound.

Not in this PR:

Commits, in order

  1. The energy stage's level becomes ONS household spending (UK: data fixes for the Budget 2026 household measures #1418 item 12; the commit itself is titled for electricity, and gas follows in commit 6).

    • Before: the lcfs_consumption stage raked kWh to DESNZ Energy Trends' calendar-2024 volume. Priced at QEP, that sits about 5% above ONS Consumer Trends for both fuels. The 30 September 2026 release revised 2024 spending up, electricity from £26.10bn to £28.13bn and gas from £14.64bn to £18.30bn.
    • The new declaration: a fuel declared in spend_level_fuels takes the kWh ONS household spending buys: its class (04.5.1, 04.5.2) for 2024, less QEP's UK average fixed cost on every UK domestic meter DESNZ counts (Great Britain's sub-national count plus Northern Ireland's row, now vendored), divided by QEP's UK average unit cost. Electricity: 84.9 TWh, against Energy Trends' 91.7 TWh.
    • Expected frame spend: on the last national's design weights, each fuel's frame spend sits about 1% below ONS, where Energy Trends put it about 5% above. The 1% is the standing charges of meters that aren't frame households' main homes. The next build measures it.
    • One function, three places: energy_pricing.published_energy_level carries the level to the stage's rake, to the uk_stage_lcfs_consumption_energy_rake gate's recomputation, and to the factor that levels the dense build's local DESNZ kWh rows. So the 980 local electricity rows move with it (Great Britain's cells now sum to 82.7 TWh, not 89.2), and no £/kWh bridge is needed.
    • Unchanged: the H5's energy_price_basis stays calendar 2024.
    • Text updates: the texts that described the level as Energy Trends now say what it is.
  2. The Gambling Commission's machine yield by category is vendored (item 5, row 56).

    • OOTLAR 1.50 moves machines with a maximum charge above £1, or a maximum prize above £100, to 25% from 1 April 2027. That means Category B.
    • excise_facts.json gains the Commission's FY2024-25 machine GGY by category for Great Britain, all venues: Category B £2.29bn of £2.59bn.
    • The split itself is held for the engine input.
  3. Re-pin of the gate digests and the coverage manifest.

  4. SPI pension income by area (item 14).

    • hmrc.pension_income.amount: each area's taxpayers' private and State Pension income together, SPI 2023-24 Tables 3.15 and 3.14, count times mean.
    • It grows at ITL Table 2.5's all-ranges total income growth (×1.1340). The national State Pension and private pension bands grew ×1.1348 together, a 0.1% gap, before UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index: ×1.151 since, a 1.5% gap (commit 36).
    • A fan-out sum bridge reconciles the cells per grain to those two families summed. This is the first bridge to sum several fan-outs, and it refuses a control that is missing one of them.
    • At source the bands sit 3.0% above the constituency cells and 3.2% above the authority cells. The area tables leave out the taxpayers HMRC places in no area, who are more often pensioners: ITL Tables 2.1 and 2.2 put 1.8% of taxpayers of State Pension age in no region, against 0.8% of all taxpayers. The bridge therefore allowed 4% at source and 0.6% in the index gap. Commits 27 and 36 revise both: 2.5% to 3.7% at source, and 2% in the index gap after UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index.
    • The frame reads total_pension_income_annual.
  5. Merge of PR A, then a test-stub fix and the H2 fixture.

  6. Gas follows electricity (María's ruling; commit 30 takes gas back to Energy Trends): ONS 04.5.2 gives 236.5 TWh against 252.3 TWh. Great Britain's local gas cells now sum to 234.0 TWh, not 249.7.

  7. The ONS electricity and gas rows are bound again (commit 30 holds the gas row out again). Decision D1's two measure exclusions are deleted (María's ruling: PR C carries the electricity deletion PR A had planned).

  8. The 29 register entries tracking UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 now track UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095, which UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 closed into. Text only.

  9. The taxpayers with pension income by area (María's ruling, option c).

    • hmrc.pension_income.count binds at HMRC's own area level, grown by ITL Table 2.5's all-ranges taxpayer growth.
    • It is a local family with no higher control, because no national row counts the union of State Pension and private pension recipients. It is declared a subset of ITL's taxpayers.
  10. Re-pin of the gate digests, the coverage manifest and the H2 fixture.

  11. The cars fuel duty hold quotes the 30 September ONS edition (María's ruling: refresh text that quotes the old release). Text only: ONS 07.2.2 for 2025 converts to about £13.1bn of duty, 9% below the OBR cars line, where the June edition gave £13.5bn and 7%.

  12. Rural Fuel Duty Relief on the dense build (item 6, row 3).

    • Before: policyengine-uk relieves petrol and diesel duty for a household with in_rural_fuel_duty_relief_area, which no build wrote.
    • The areas: HMRC Notice 2001 opens the relief in the Inner and Outer Hebrides, the Northern Isles, the Islands of the Clyde, the Isles of Scilly, the post town of Hawes and sixteen mainland postcode districts. It names the island groups without bounding them, so tools/build_uk_rfdr_area_layer.py declares Microcosm's translation into postcodes, each part with its sources. Islands that share a sector with the mainland (Raasay, Seil, Easdale, Luing, Kerrera, Lismore) and Hawes are listed by unit postcode, and mainland units inside island districts are taken out. The points the sources leave open are recorded in the tool; none changes an area's flag.
    • The layer: from the ONS postcode lookup (August 2026), uk/rfdr_area_layer.json lists the output areas more than half of whose postcodes are in the scheme: 41 in England and 1,053 in Scotland.
    • The node: uk.full.geography.rfdr sets the flag on each placed household from its assigned output area, after the local-authority resolve, on the BRMA node's pattern. --rfdr-localisation area is the default on the atomic assignment; none is for measurement builds, and a release candidate refuses it. Northern Ireland has no area in the scheme.
    • The national: it has no areas, so it leaves the engine's default of false (María's ruling).
    • The column joins the export surface's reviewed extras. The next dense build measures the flag.
  13. Re-pin of the gate digests for the RFDR export column.

  14. The six texts that still said UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 tracks an item now name UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095, finishing commit 8. Text only.

  15. Merge of PR A's newer commits (the UC take-up gate at 0.06 of DWP's caseload, and the care-hours test). The gate digests are recomputed on the merged gates.json.

  16. OBR's betting and gaming receipts are bound (María's ruling).

    • The old reasoning: the exclusion held the row out as an identity. The stage levels each gambling base to HMRC's FY2024-25 receipts, or Machine Games Duty's liabilities, over the engine's rates, so the engine charges the receipts back by construction.
    • Why it no longer holds: that is true only at prior weights. Calibration moves the weights of the households carrying the bases, and nothing holds their level.
    • Before, on PR A's national (run uk-national-1418-7757add7a, H5 a912a7044836), at final weights:
      • the twelve bases shared out by betting and gaming stakes each sit 12.3% above HMRC's level;
      • the National Lottery base, on its own stakes, sits 2.4% above;
      • Machine Games Duty comes to £0.688bn against £0.612bn of liabilities;
      • at the engine's 2024 rates the bases carry £3.972bn of duty against OBR's £3.616bn (+9.9%). At HMRC's levels it would be £3.619bn.
    • The change: the exclusion is deleted, so the national binds the row as it binds the ONS energy rows. The tests and notes that called the row held out now say it is bound.
    • Undone by commit 40, pending María: the national solve can't measure the row's 2024 duty, so the exclusion is restored.
  17. Merge of PR A's review round (the UC take-up gate at 0.055, and the Carer's Allowance wording), with the gate and spine-part digests and the coverage manifest recomputed.

  18. The H2 fixture, regenerated on the merged tree.

  19. The local references, membership and compile-parity receipt are regenerated on the 272c987 pin. It ran from the feed's local cut, whose source hash matches the pin.

    • Pension income by area: 2,022 cells, 2,016 of them compiled. The two authorities without SPI local measures and the City of London's support-floor exclusion sign off the other six, as for the other SPI local families.
    • Energy: the DESNZ electricity and gas rows move by exactly the new level factor over the old, ×0.9258 on all 980 electricity rows and ×0.9371 on all 977 gas rows. No row's status changes.
    • Nothing else moves. The target-surface gate passes, and its count pins follow.
  20. Merge of main after PR A merged (two changelog lines).

  21. The Carer's Allowance take-up draws are balanced by weight within strata, for the next pair build (option B of the pool investigation).

    • Before: the newly entitled carers' identity-keyed uniforms were compared with 0.6 one by one, so the claiming share reached 0.6 only in expectation. On the few source households over the limit, one draw put the 2031 taper's cost well below its expectation.
    • The change: the uniforms are now placed by weight within strata, as policyengine-uk's stratified_uniform does (an engine test holds the two identical). The strata are:
      • the route (in-household or not);
      • whether the benefit unit reports a means-tested benefit at the survey year;
      • the survey-year weekly earnings band, one weekly rate wide above the limit and open above three.
    • Result: each stratum claims 0.6 of its eligible weight to within one record, and the outside pool's draw is balanced the same way at its measured share. The draws stay keyed on identity.
    • Unchanged: the rate, the pool and the caseload.
    • Receipt: it lists each stratum's realised share against its target, support-banded.
  22. Re-pin of the coverage manifest and the H2 fixture for the stage.

  23. Anthony's review: the scorer test's pin. test_uk_local_candidate_scorer.py pins the 27,208 active local references that commit 19 left (the 2,016 pension-income cells added).

  24. Anthony's review: the Rural Fuel Duty Relief checks.

    • In area mode the node refuses a Great Britain output-area code that isn't well-formed for its system, and a frame with Great Britain households none of which is flagged. Commit 31 makes the second check per support system, and commit 36 applies it at full scale only.
    • The layer tool's column check tests what it means.
    • Tests cover the release refusal of --rfdr-localisation none and the committed-layer requirement.
  25. Anthony's review: the energy level's bound and Northern Ireland's meters.

    • The energy rake gate holds each ONS-spend level to between 0.85 and 1.0 of the Energy Trends volume (0.926 and 0.937 now), so a wrong input row the recomputation shares fails it.
    • Northern Ireland's electricity meters stay in the standing-charge subtraction, and the level block says why: its tariffs carry the charge in the unit rate. The level doesn't move.
    • The gate battery digests are recomputed.
  26. Anthony's review: the Carer's Allowance strata are enforced. The stage refuses a take-up or pool stratum whose drawn weight misses its share by more than one record. The receipt records the check, and the H2 fixture carries the stage's new notes.

  27. Anthony's review: the pension bridge's tolerance. spi_pension_amount_bands_vs_area allows the larger measured gap plus half a point, 3.7%, not 4%. The coverage manifest is regenerated for commits 23 to 27.

  28. Anthony's re-review: the council tax pin. The support-floor membership has the City of London's two SPI pension income cells (51 rows, 95 area ids). This was the only line both engine-free jobs failed on.

  29. SPI Table 3.7's other income by total-income band is bound (María's ruling of 29 October). No row held that column, so the Income Tax on savings rows were met mostly through it.

  30. Gas back on Energy Trends; the QEP payment method checked directly (María's ruling of 29 October, Anthony's re-review 2 and 3).

    • ONS 04.5.2 counts bottled gas (COICOP 04.5.2.2), which the frame holds in liquid_fuel_consumption. Bottled gas is 2.2% to 3.2% of the class on ONS's CPI weights.
    • So gas and the local DESNZ gas rows return to the Energy Trends level, and ons.household_gas_expenditure returns to the measure exclusions, tracked on UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095. Electricity stays on ONS 04.5.1.
    • Every QEP payment method's electricity level falls inside the ratio bound, so the gate now checks the stage's declared payment_method against qep_payment_method.
    • The gate battery digests are recomputed.
  31. Anthony's re-review: count pins, RFDR per system, the Carer's Allowance tolerance, and 7b.

    • The second council tax assertion and the subset relations (22) are re-pinned.
    • The RFDR node refuses each support system with households none of which is flagged.
    • The Carer's Allowance stage refuses a stratum off its share by more than half its heaviest record, the most the construction allows. A stage test covers the pool branch. Commit 36 rotates each stratum by a start, which makes the bound one record.
    • The RFDR layer is rebuilt from the registered NSPL zip (downloaded from R2, sha256 checked). It flags the same output areas, and a test holds its provenance to the zip.
  32. SPI savings rows on the OBR's savings income; SPI Table 3.10 bound (María's rulings of 29 October).

    • The Table 3.7 interest and other-income rows grow by the October 2026 EFO's Table 4.5 savings income growth, vendored as obr_savings_income_growth_facts.json, in place of ×1.192.
    • Table 3.10's individuals with self-employment income, and that income, bind in the eight bands from £1 to £10,000. The others are signed out row by row.
  33. Re-pin onto Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) chronicle#394's merge (53815b2).

    • On the national surface only the intended values move: the 13 interest rows up 13.4%, 13 other-income rows and 16 Table 3.10 rows.
    • obr.winter_fuel_allowance is pinned to March's expenditure table, since October's FY2025-26 outturn would match it twice.
    • On the local surface only the 977 DESNZ gas cells move, back to the Energy Trends level after commit 30 (×1.067); the statuses and the references are unchanged.
  34. The national calibration test's pin moves to 1,403 active references. It had stopped engine-free CI on commit 33; the 94 test files that read the regenerated surfaces pass locally.

  35. Each UK active reference count is pinned once in the tests (María's request).

    • National: test_uk_target_references.py's ACTIVE_REFERENCE_COUNT, which now carries the count's history.
    • Local: the battery bindings test's candidate_targets.
    • The other copies check against the committed membership, or drop an incidental count. A change to the target set still fails one test, and no longer one per copy, each found in its own CI round.
  36. The independent review and Vahid's review (María's rulings of 29 October where the points were hers).

    • Pension income by area.
      • The bridge's index bound is 0.02: the 1.5% gap after UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index, plus half a point. A test recomputes every fan-out bridge's gap from the committed registers.
      • The source factor is bounded on both sides, at 1.025 to 1.037, because the gap runs one way.
      • The rounded-thousand floor defers the pension income counts HMRC publishes at 5,000 or fewer, and their amounts with them (17 constituency and 3 local-authority cells of each). The bridge still counts them at source.
      • A deferred cell joins the reconciliation grown by its bound cells' own index, as an off-surface cell does. The dense target surface compiles end to end, and the pension cells take ×1.045 at both grains.
      • The SPI-frame fence and both adjudication records name the pension metrics.
    • Table 3.10's amounts grow by the count index times mixed income per head.
    • Electricity is measured at 2024, as road fuel is, with a signed matched-period hold.
    • The zero-support floor covers Table 3.7's other income and Table 3.10's bands.
    • RFDR: sampled rungs check the codes only, and the tool pins the NSPL registration and gains --check.
    • Carer's Allowance: each stratum's placement is rotated by a start drawn per stratum. So the stage refuses past one record, not half.
    • Cache keys and timing: the stages key their caches on the modules they read, and the energy level is computed once per process.
    • Texts: the stale texts are corrected, and the savings path's notes record its scope against ONS's household interest.
  37. Regenerated surfaces: the national and local references, the census, both parity receipts, the coverage manifest, the H2 fixture and the gate digests.

    • On the national surface only the intended values move: the eight Table 3.10 amount rows (×1.0763, the count index), and electricity onto ONS's calendar-2024 figure (×0.975).
    • On the local surface only the 40 pension income cells move, from bound to deferred, leaving 27,168 active.
  38. The re-review of commits 36 and 37.

    • tools/diagnose_uk_consumption_taxes.py reads each row's measurement period from the contract.
    • Stage-level tests pin the Carer's Allowance placement and the RFDR strictness switch through the graph. A test checks each fan-out bridge's source half on the committed registers, and another the surface's deferral wiring.
    • The Carer's Allowance start is keyed on each stratum's members.
    • Only the transferred spine gate keys on the energy level.
    • The small-cell floor reads published values.
    • The widened floor and fence records say they are pending María's confirmation.
    • The texts are fixed.
    • The local references regenerate unchanged, and the national compile-parity receipt changes only in the income-anchor rows' rationale.
  39. The round-3 re-review of commit 38.

    • The fan-out bridges' gaps are checked by running the runtime surface on the committed registers, at both grains.
    • A Carer's Allowance stratum's start is keyed on its lowest-draw member, so a membership change re-draws a stratum only when that member changes.
    • The certifier compares support-floor drops on the floor's signature fields only.
    • The diagnostic tool weights every row at the calibration year and reads periods from the run's registry.
    • A test pins the take-up strata's route.
    • The RFDR layer is rebuilt with its corrected description (same areas).
    • The texts are fixed.
  40. The betting and gaming row is held out again, pending María's decision.

    • The first national built from this PR (at 10f434d) refused at the national problem. The row declares the frame's period, 2024, but its value is the engine's gambling_duty, not a stored column.
    • The solve measures a declared period only on a stored column or a CGT dated measure, so it can't read the engine's 2024 duty. No PR C national had been built since commit 16, and the tests don't run the national problem's measure routes.
    • María's signed exclusion of 23 October is restored as it was. Its adjudication now notes that her 28 October bind ruling can't take effect until the solve can measure the duty, either through a dated duty measure or through bases bound at the frame's period.
    • Main holds the other excise rows out under María's 23 October ruling, since the build sets their level; they would hit the same refusal if bound. The row is still measured on every evaluation.
  41. The round-4 re-review of commits 39 and 40.

    • A test runs the solve's own period check over every bound row of the packaged contract, so a binding the solve can't measure now fails in CI.
    • The bridge mutation's scale is derived from the baseline, and the committed deferrals are pinned.
    • The diagnostic tool reads the solved weights as the certifier does.
    • The texts are fixed.
  42. The round-5 re-review of commit 41.

    • A test fails 14 days before any committed measure exclusion lapses, so a lapse shows in CI and not at a national's target step.
    • The diagnostic tool's own weight path is tested on synthetic release and solved H5s. Its id check refuses a household on either side alone. With --registry, its default year is the run's calibration period, and its texts say it needs a national H5.
    • The period test reads the release's calibration year and also runs the solve's fact-period check.
  43. The round-6 re-review of commit 42.

    • The exclusion lapse clock also reads the register's support floor, and passes on an empty register.
    • The period test runs the solve's fact-period check on every binding kind, so the CGT provider rows are covered, and a required row missing from the membership fails.
    • The diagnostic tool weights its composition with the solved weights and refuses a registry with mixed periods when no --year is given.
  44. The round-7 re-review of commit 43.

    • The period test reads the rows the national compiles: national-line targets, less rows signed out of a fan-out and rows held on their own, except held rows in the support floor's scope, which are compiled as probes.
    • A deferred required row is flagged like a missing one. The national compiles only the active references, so this test is the only guard there.
  45. The round-8 re-review of commit 44. The period test's docstrings describe the deferral check accurately, and its plain-column check again covers the national-geography targets on the local line only, which the dense build compiles.

Checks run locally

All on the merged tree, with policyengine-uk 2.162.14:

  • the touched engine-free tests: energy pricing, local energy targets, stage health, local census, cross-grain declarations, fan-out bridges, local targets, population targets, measure simulation, data-target parity, gate-battery pins and contract;
  • for RFDR: the layer and node tests, country spec, Orrery contract and publication, full-build preparation, CLI and calibration graph, national graph, rowwise national role, terminal gates and dense sign-off draft;
  • the annual-income engine test;
  • the coverage manifest, local census and parity-register checks.

Also run since: the RFDR engine admission test (test_uk_full_graph_admission.py), which passes, and the local surface tests after commit 19 (battery bindings, local targets, target references, local census, local release preflight, fan-out bridges, cross-grain declarations), which pass.

For commits 23 to 27: the RFDR layer and node tests, the atomic, BRMA, rent, property and home-value graph tests, the full target and population graphs, the CLI, Carer's Allowance, stage health, energy pricing, fan-out bridges, cross-grain declarations, national reconciliation, country spec, scorer, battery bindings, gate-battery pins and contract tests, and the coverage manifest check.

For commits 28 to 33: 31 touched engine-free test files (target references and membership, local targets, battery bindings, local census, local release preflight, fan-out bridges, cross-grain declarations, the full target graph, energy pricing, domestic energy targets, the constituency energy resource, high-value homes, stage health, measure simulation, Carer's Allowance, the RFDR layer and graph, subset relations, council tax, HMRC uprating, population targets, the feed pin, full targets, the scorer, national reconciliation, bus and CGT targets, LCFS consumption, country spec, gate-battery pins and contract), the HMRC uprating and RFDR admission engine tests, the coverage manifest check, and the feed-gated high-value home stock check in the re-pin. CI runs the rest.

For commits 36 and 37: the 107 engine-free test files that read the regenerated surfaces or that this round touches; the engine tests for HMRC uprating, Carer's Allowance and graph admission; the coverage manifest check; the H2 fixture's readers; tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the committed layer byte for byte; and tools/uk_target_surface_receipt.py, which compiles the dense target surface end to end on the pinned feed. All pass. The vendoring register's check that each consumer names its file failed once and is fixed in commit 36. The Orrery reconstruction test needs a Node package this checkout doesn't have; CI runs it.

For commit 38: the 31 engine-free test files this round touches or that read what it changed; the Carer's Allowance and HMRC uprating engine tests; the coverage manifest check; the round-2 mutations, each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which still reproduces the committed layer byte for byte. All pass.

For commit 39: the 34 engine-free test files this round touches or that read what it changed; the Carer's Allowance and weight-year engine tests; the coverage manifest check; the re-review's mutations (local-authority pension cells x1.005, and the take-up strata without the route), each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the rebuilt layer byte for byte. All pass.

For commits 40 to 45: the touched engine-free test files (measure simulation, measurement period, consumption-tax diagnostics, excise quantity inputs, uprating holds, support floor, fan-out bridges and off-surface cells). All pass.

CI runs the suites (this PR targets main).

🤖 Generated with Claude Code

juaristi22 and others added 10 commits October 28, 2026 20:28
…g buys

The lcfs_consumption stage raked electricity kWh to DESNZ Energy Trends'
calendar-2024 volume, which priced at QEP sits about 5% above ONS
Consumer Trends 04.5.1 on the 30 September 2026 release. A fuel the
stage now declares in spend_level_fuels takes its level from ONS
household spending instead: the class for the level year, less QEP's
United Kingdom average fixed cost on every UK domestic meter DESNZ
counts (Great Britain's sub-national count with Northern Ireland's row,
now vendored), over QEP's United Kingdom average unit cost. Electricity
is declared, giving 84.9 TWh against Energy Trends' 91.7 TWh; gas keeps
Energy Trends.

published_energy_level carries the new level everywhere it was read:
the stage's rake, the energy_rake gate's recomputation, and the factor
that puts the dense build's local DESNZ electricity rows on the stage's
level, so those rows move with it. The texts that described the level
as Energy Trends follow (gate notes, census fence, cross-grain route,
holds, adjudication, contract notes), and the ONS resource's purpose now
names the release its rows come from.

microcosm#1418, item 12 of the Budget data fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OOTLAR 1.50 moves machines with a maximum charge to play above GBP 1,
or a maximum prize above GBP 100, to Machine Games Duty's higher rate
from 1 April 2027: Category B. The Machine Games Duty standard-rate base
the stage levels has no category split, so the Gambling Commission's
FY2024-25 gaming machine gross gambling yield by machine category
(Industry Statistics sheet 4, all venues, Great Britain) is vendored
beside it, a sixth excise_facts.json selection. The split column that
reads it waits for the engine input (policyengine-uk#2719).

microcosm#1418, item 5 of the Budget data fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The energy_rake gate's notes moved the gates manifest digest, its
fingerprint and the spine part's manifest digest (the policy digest is
unchanged), and the stage's declaration and vendored resources moved the
release input coverage manifest's source digest. Regenerated and
checked; the local references, the local compile parity receipt and the
H2 fixture follow in the pin pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ands

The dense surface had no pension income below the national SPI bands.
hmrc.pension_income.amount binds each constituency's and local
authority's taxpayers' private and State Pension income together from
SPI 2023-24 Tables 3.15 and 3.14 (count times mean). It is grown to
calendar 2025 by ITL Table 2.5's all-ranges total income growth, which
the national State Pension and private pension bands match together to
within 0.1%. The frame reads total_pension_income_annual, the engine's
sum of the two pensions on the annual simulation.

A fan-out sum bridge reconciles the cells per grain to the national
bands, and may now sum several fan-outs: the State Pension and private
pension bands together are the area cells' quantity. A control missing
one of its fan-outs is refused. At source the bands sit 3.0% above the
constituency cells and 3.2% above the local-authority cells, since the
area tables leave out the taxpayers HMRC places in no area, who are
more often pensioners; the bridge refuses past 4% at source or 0.6% in
the index gap. The two authorities without SPI measures are deferred as
for employment, and the metric is appended last at both grains.

Counts are not bound here: the area tables count taxpayers with any
pension, a union no national row binds, and the bridge for them is a
ruling still owed (microcosm#1418, item 14 of the Budget data fixes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR C stacks on PR A. The two conflicts were the contract's target counts
(PR A's three ITL Table 2.6 savings rows and PR C's pension income by
area: 452 targets, 391 national, 61 local) and the release input
coverage manifest, regenerated on the merged tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine-free annual measure test checks every annual measure is one
the resolver knows; total_pension_income_annual reads the engine's
total_pension_income, which the stub now sums from its two pensions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lcfs_consumption stage's spend_level declaration is mirrored into
the fixture's stage parameters and the graph's contract hashes move with
it and with the re-vendored resources; nothing else moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
María's ruling of 2026-10-28 on PR C: gas follows electricity. The 30
September 2026 Consumer Trends release puts 2024 gas spending at
GBP 18.30bn, so the stage's Energy Trends level priced at QEP sat about
5% above ONS for both fuels. The stage now declares gas in
spend_level_fuels: ONS 04.5.2 less QEP's United Kingdom average gas
standing charge on every UK domestic gas meter, over the unit cost,
236.5 TWh against Energy Trends' 252.3 TWh. Great Britain's local gas
cells follow at 234.0 TWh. The texts that named Energy Trends for gas
now name the shared level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decision D1 held ons.household_electricity_expenditure and
ons.household_gas_expenditure out of the objective as published-source
residuals against the stage's DESNZ-volume level. With the stage's level
now the kWh ONS household spending buys, the frame starts about 1% below
each row at design weights, so both exclusions are deleted and the
national binds ONS 04.5.1 and 04.5.2 again (María's rulings of
2026-10-28 on microcosm#1418: PR C carries the electricity deletion PR A
planned, and gas follows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1278 closed into microcosm#1095 on 25 October, which now
tracks every open UK data item. The 29 register entries whose tracking
still named #1278 (five measure exclusions, two small-cell rules and 22
uprating holds) name #1095 (María's ruling of 2026-10-28). Text only:
no entry's dates, reason or scope changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 7 commits October 28, 2026 21:16
María's ruling of 2026-10-28 on PR C (option c): SPI Tables 3.14 and
3.15 count each area's taxpayers with any pension income, State or
private, and no national row counts that union, since the national SPI
bands count State Pension and private pension recipients apart and most
taxpayers of State Pension age are both. hmrc.pension_income.count binds
the cells at HMRC's own area level, grown by ITL Table 2.5's all-ranges
taxpayer growth, as a local family with no higher control, declared a
subset of ITL's taxpayers. The metric follows the amount, appended last
at both grains; the same two authorities are deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The energy_rake gate's notes moved the gates manifest digest, its
fingerprint and the spine part's manifest digest again (the policy digest
is unchanged); the stage's gas declaration moved the coverage manifest's
source digest and the fixture's stage parameters and contract hashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The obr.fuel_duties_cars reason converted ONS 07.2.2 for calendar 2025
to duty on the June Consumer Trends edition; the feed carries the 30
September 2026 edition, whose figure is 3.2% lower, so the conversion is
about GBP 13.1bn, 9 percent below the OBR cars line. Text only (María's
ruling of 2026-10-28: refresh reason text that quotes the old release).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk relieves petrol and diesel duty for a household with
in_rural_fuel_duty_relief_area, which no Microcosm build wrote. HMRC
Notice 2001 (SI 2011/2935 Schedule 1) opens the relief in the Inner and
Outer Hebrides, the Northern Isles, the Islands of the Clyde, the Isles
of Scilly, the post town of Hawes and sixteen mainland postcode
districts, and names the island groups without bounding them.

tools/build_uk_rfdr_area_layer.py declares Microcosm's translation into
postcode districts, with unit postcodes for the islands that share a
sector with the mainland (Raasay, Seil, Easdale, Luing, Kerrera,
Lismore) and for Hawes, and mainland units inside island districts
excluded, each part with its sources. From the ONS postcode lookup it
writes uk/rfdr_area_layer.json: the output areas more than half of
whose postcodes are in the scheme, 41 in England and 1,053 in Scotland.

The post-geography node uk.full.geography.rfdr sets the flag on each
placed household from its assigned output area, after the local
authority resolve, on the BRMA node's pattern: --rfdr-localisation area
is the default on the atomic assignment, none is for measurement
builds and a release candidate refuses it, and the layer is pinned like
the BRMA layer. Northern Ireland has no area in the scheme, and the
national line leaves the engine's default of false (María's ruling of
2026-10-28). The column joins the export surface's reviewed extras.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The uk_export_surface gate's allowed_extra_columns gains the RFDR flag,
moving the policy, gates manifest and fingerprint digests and the
release-cut part's digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracking fields moved to #1095 in c4b08a5, but six texts still
said an open item is tracked on #1278: the obr.fuel_duties_cars
reason's business-use rebind, two uprating-hold reasons, and the
high-value homes coverage decision (#1240 fix 6) in the local census
description, the scope constant's comment and a test comment. Text
only, completing María's ruling of 2026-10-28; texts citing #1278 as
where something was decided or changed are history and stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o PR C

PR A allows the UC take-up stage gate 0.06 of DWP's caseload and names
the in-household care hours carrier among the release-dropped columns.
The gate battery's policy, manifest and fingerprint digests and the
spine part's manifest digest are recomputed on the merged gates.json,
which carries both PR A's tolerance and PR C's notes and RFDR export
column; the coverage manifest and the H2 fixture don't move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 3 commits October 28, 2026 22:57
The obr.betting_and_gaming_duties exclusion held the row out as an
identity: the lcfs_consumption stage levels each gambling base to
HMRC's FY2024-25 receipts, or Machine Games Duty's liabilities, over
the engine's rates (microcosm#1359), so the engine charged the receipts
back by construction. That holds only at prior weights. Calibration
moves the weights of the households that carry the bases, and nothing
held their level: on the 3c0d11b6 national the bases sat about 12%
above HMRC's receipts at final weights (Machine Games Duty £0.684bn
against £0.609bn). The level is not held at final weights, so the
exclusion is deleted and the national binds the row, as it binds the
ONS electricity and gas rows (María's ruling of 2026-10-28).

The notes and tests that described the row as held out now say it is
bound, and the consumption-tax tool's docstring names the bound rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR A narrows the UC take-up stage gate to 0.055 and rewords the Carer's
Allowance pool and the unmapped account codes. The gate battery's
policy, manifest and fingerprint digests and both spine-part digests
are recomputed on the merged gates.json, and the coverage manifest is
regenerated; the H2 fixture follows in its own commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR A's Carer's Allowance wording and PR C's stages both reach the spine
graph; tools/graph_uk_spine_fixture.py regenerates it on the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 2 commits October 29, 2026 04:24
The local surface regenerates on the 272c987 pin for PR C's two local
changes, from the feed's local cut (constituency and local-authority
rows and the statistical_scope rows but the small areas, as the
272c987 re-pin cut it; its source hash matches the pin):

- SPI pension income and the taxpayers with it by area add 2,022 cells,
  2,016 of them compiled. The two authorities without SPI local
  measures and the City of London's support-floor exclusion sign off
  the other six, as for the other SPI local families.
- The DESNZ electricity and gas rows move with the energy level, by
  exactly the new factor over the old: x0.9258 on all 980 electricity
  rows and x0.9371 on all 977 gas rows, with no status changing.

Nothing else moves: the references only gain the pension cells, and the
receipt replaces the energy rows and adds the pension rows. The
target-surface gate passes; its count pins follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 changed the base branch from uk-budget-2026-static-fixes-1418 to main October 29, 2026 10:04
juaristi22 and others added 2 commits October 29, 2026 11:22
The stage compared each newly entitled carer's identity-keyed uniform
with the take-up rate on its own, so the claiming share reached 0.6
only in expectation. On the few source households over the limit, one
draw had put the 2031 taper's cost well below its expectation (the
1418-ca-pool investigation, option B).

weight_balanced_uniforms places a stratum's uniforms by weight: each
record takes the midpoint of its slice of the stratum's cumulative
weight, in the order of its draw, as policyengine-uk's
stratified_uniform does (an engine test holds the two identical). The
take-up strata are the route (in-household or not), whether the
benefit unit reports a means-tested benefit at the survey year, and the
survey-year weekly earnings band, one weekly rate wide above the limit
and open above three, read from the engine's carers_allowance_earnings
as its own test reads them. Each stratum then claims 0.6 of its
eligible weight to within one record. The outside pool's draw is
balanced the same way over the other carers who report no allowance, by
receipt and band, at its measured share. The rate, the pool and the
caseload are unchanged, and the draws stay keyed on identity. The
receipt lists each stratum's realised share against its target,
support-banded and guarded as its other shares are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Carer's Allowance take-up stage's reviewed payload gains the
carers_allowance_earnings predictor and the balanced-draw wording, so
the source manifest's hash and the spine graph move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 216f22a5f (round 3)

This checks round 2's items (issuecomment-6513556513) against commit 38 and the reply, then reviews commit 38. Two reviewers re-checked the items and re-ran round 2's mutations, and a verifier checked each new candidate. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/, tools/ or docs/.

Verdict: nothing blocks merge.

Round-2 items

Item Status Evidence
A Resolved ROW_YEARS comes from each row's measurement_period in the contract, and the test requires the two to match. Two residuals remain (5).
B Resolved The stage tests spy on both placements and permute the rows. Dropping either starts= (take-up or pool) now fails, and so do row-position draws.
C Resolved Both tests go through the graph builder. Forcing strictness in the caller or in the kernel fails a test.
D Partly The source half is tested on the committed registers, and the deferral wiring has a test. But the local-authority reading isn't the runtime's (1).
E Resolved An energy_pricing.py edit now moves 22 stage keys, from lcfs_consumption on, and leaves the assembled gate alone. The new param enters the node key, and tests pin both.
F Resolved The floor reads value_before_uprating, and reverting that fails the new test. No reference moved.
G Resolved Simulated with the real helpers: a one-record stratum claims at the rate over identities (0.59 to 0.60 at a rate of 0.6). The start is order-free and the one-record bound holds. One cost remains (2).
H María's The three records say they're pending her confirmation. The filter is named correctly, and no approved_on moved. Two text gaps remain (4).
I Resolved Every listed text is fixed; small residuals are in 6.
#4, #14, #15 Resolved —

New findings

1. The local-authority source check reads a different quantity from the runtime.

  • test_each_committed_bridge_holds_its_declared_source_gap (test_uk_fanout_sum_bridges.py:469-510) reads the pension bridge's local-authority source factor as 1.0321.
  • The runtime surface reads 1.0296, because it also counts the two authorities without SPI measures. The test's docstring says it leaves them out.
  • Mutation run: with the local-authority cells ×1.005, the runtime reads 1.0245 and refuses, below the 1.025 floor. The test reads about 1.027 and passes.
  • So a rise of about 0.45% to 0.7% confined to the local-authority cells keeps CI green and stops every dense build.
  • The wiring test (test_uk_off_surface_cells.py:737) replaces the check with a spy and passes no fan-out controls, so it covers only the hand-off.
  • Suggestion: compute the factor through uk_local_target_surface on the committed pension cells, with the real controls. The review's script does this in about 20 s. Both grains would then read the runtime's quantity.

2. Keying the start on every member re-draws about half a stratum when one member changes.

  • stratum_starts (uk_runtime/carers_allowance_take_up.py:352-382) digests the sorted person ids of each stratum's placed set. Any change to that set re-randomises the start.
  • When one member leaves, about 2r(1−r) of the other members' flags flip, whatever the stratum's size: 0.48 at a rate of 0.6. Keyed on the code alone (round 2's scheme), it was 0.11 at 5 members and under 0.01 at 80.
  • In a stage-like set of 320 eligible carers in 20 strata, one carer changing stratum flips on average 27.5 of the other 319 flags (1.3 with code keying). The weight-based pool share adds to this: a prior-weight change that moves one carer across the pool line re-draws whole take-up strata.
  • So a policyengine-uk re-pin or an SPI refit that moves one carer's band or earnings test re-draws most of that stratum. Household-level comparisons between builds churn, while the aggregates stay within one record.
  • Nothing documents this. The test comment at test_uk_carers_allowance_take_up.py:433-434 ("the stream does not depend on which other people the frame carries") still holds for the uniforms, but no longer for the flags.
  • One alternative keeps G's per-carer property: key the start on the stratum's lowest-draw member.
    • A carer alone claims at about 0.594 at a rate of 0.6.
    • Churn is roughly code keying's plus 0.48/n: 0.20, 0.06, 0.011 and 0.006 at 5, 20, 80 and 200 members.
    • The cost: the full ~48% re-draw comes back when the lowest-draw member leaves, or a lower one arrives, which is about 1 change in n.
  • Otherwise, document the cost.

3. Re-certifying an earlier national will refuse if its floor dropped a row.

  • tools/certify_uk_release_cut.py:306-313 loads the floor from the certifier's own package. uk_runtime/support_floor.py:571-576 then requires each recorded drop's reason, tracking, approved_by, adjudication, approved_on and expires_on to equal that floor's.
  • Commits 36 and 38 changed the reason and the adjudication. A probe confirms that a drop recorded under the floor at fddbc7073, edad8aee9 or ead8943d6 is refused at this head, with "not under the declared signature".
  • Runs with no drops are unaffected. So are the in-graph read-backs, which compare against the run's own record. The runbook certifies at the build's own head, so the standard chain is fine.
  • The next trigger will be María's confirmation edit of the floor's adjudication. After it, a national built at this head that dropped a row couldn't be re-certified at the later head without a rebuild.
  • Per-row exclusions are compared by name only (tools/certify_uk_release_cut.py:298-302), so the floor is the odd one out.
  • Suggestion: compare only the signature fields (tracking, approved_by, approved_on, expires_on), or accept the run's own declaration digest.

4. H: two text gaps.

  • The floor's note asks María to confirm or re-size the amounts scope's 2-row cap (seven families, now nine). But the counts scope widened too, from seven families to eight (Table 3.10's counts), under the same 2-row cap.
  • The cars fuel duty adjudication's new note says "the ruling and its figures are unchanged". Yet this PR restated the reason's figures on the September edition (a58bc1a) and added the June comparison. The note also doesn't say who re-worded it.
  • Older, and not from this PR's edits: the fence's signed reason, and its 23 October re-wording, still name income_tax_annual > 0.

5. The diagnostic tool has two residuals.

  • Weights. Rows measured at 2024 are weighted by the engine's weights at 2024.
    • The national H5 stores the solved weights divided by the engine's 2024→2025 factor (1.0030), and tests/engine/uk/test_uk_weight_year.py pins that the engine returns the solved weights at 2025 and solved ÷ factor at 2024.
    • So on a release H5, every 2024-period row (electricity, road fuel and the five excise rows) reads the calibration's figure ÷ 1.003. An exact fit therefore shows −0.30%, though the tool's comment says it "measures what the national binds".
    • The excise rows have read this way since UK: impute alcohol, tobacco and gambling consumption so policyengine-uk's excise duties stop scoring zero #1359.
    • Suggestion: weight the 2024 values with the calibration-year weights, as the certifier (uk_solve_year_frame) and the scorer do.
  • Periods. The tool takes each row's period from the package's contract, but its value from the run's --registry, whose specs carry their own measurement_period.
    • On a national built before this PR, electricity's registry value is the 2025 fact, which the tool now reads off the 2024 frame: about −2.5 points, spurious.
    • Suggestion: read the period from each registry spec, and fall back to the contract only without --registry.
  • Aside: the docstring at uk_runtime/weight_year.py:11 still says "0.72% from 2024 to 2025". The pinned growth is 0.3%.

6. Smaller items.

  • No test pins the take-up strata's route dimension (this predates commit 38).
    • With take_up_strata = pool_strata, 61 of 800 flags change on _outside_pool_frame, and all 37 tests in the file pass.
    • The new stage test reads its strata from the stage's own call.
  • "Postcode districts and sectors" survives in two places:
    • tools/build_uk_rfdr_area_layer.py:8-9 and :589, which writes it into the committed layer's description, so fixing it moves the layer's digest;
    • uk_runtime/rfdr_area_layer.py:9-10.
  • uk_runtime/graph_evidence.py:342: the spine gate kernel's "Node parameters" list doesn't name energy_pricing_code.
  • The PR body's commit-38 checks line cites "the mutation runs above", which only the reply describes.

Checked and found true

  • Every round-2 mutation, re-run against this head, now fails a test.
  • energy_pricing_code is computed from the module's bytes and enters only the transferred gate's key. Nothing pinned needed regenerating: the H2 fixture has no gate nodes, and the Carer's Allowance stage constants are unchanged.
  • Each start digests exactly the set its placement balances over, from unique person ids: eligible carers for take-up, drawable carers for the pool.
  • The diagnostic's ROW_YEARS follows the calibration's own period rule, and --help runs without the engine.
  • Commit 38 moves no reference or membership. The national compile-parity receipt changes only in rationale text, which now gives the interest counts their own growth.
  • CI at 216f22a5f: every finished job passes. engine-free (3.13) and engine-us (3.13) were still running when this was posted.

Not covered

  • Any build or engine test: everything ran engine-free on Python 3.13.
  • The RFDR --check wasn't re-run against the registered zip.

🤖 Generated with Claude Code

The re-review (issuecomment-6514923017) found nothing blocking; these
are its six points.

1. The fan-out bridges' gaps are checked by running the runtime surface
   itself on the committed registers (cells, controls, small-cell
   deferrals and off-surface cells), so each grain reads what a build
   reads; local-authority pension cells x1.005 now refuse in a test.
2. A Carer's Allowance stratum's start is keyed on its code and its
   lowest-draw member, so a membership change re-draws a stratum only
   when that member changes; a carer alone still claims at the rate.
3. The certifier compares a run's support-floor drops with the floor's
   signature fields only (tracking, approver, approval and expiry), so a
   re-worded floor can still certify a run built under it.
4. The floor's note names both widened caps; the cars fuel duty note
   says what this PR changed and that it awaits María.
5. tools/diagnose_uk_consumption_taxes.py weights every row at --year
   and takes each row's period from the run's registry when given one;
   weight_year's docstring says 0.3%.
6. A test pins the take-up strata's route; the RFDR wording reads
   "districts, unit postcodes and excluded units" in the tool, the
   runtime docstring and the rebuilt layer (same areas; description and
   built_on only); the spine gate docstring names its new param.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-3 re-review. All six points are addressed at 10f434dfd (commit 39), except H's approvals, which stay María's.

  1. Fixed. The source-gap test now runs the runtime surface itself on the committed registers, for every fan-out bridge.
    • It calls uk_local_target_surface with the committed cells at their resolved and published values, each bridge's national control, and the committed small-cell deferrals and off-surface cells.
    • Both grains now read what a build reads. For the pension bridge, the source factors are 1.0295 and 1.0296, and the applied factors ×1.0450 and ×1.0451.
    • test_the_runtime_surface_refuses_local_authority_cells_moved_past_the_floor is your mutation: with the local-authority pension cells ×1.005, the surface now refuses.
    • It runs in under a second.
  2. Fixed with your alternative. Each stratum's start is keyed on its code and its lowest-draw member's identity.
    • A carer alone in a stratum still claims at the rate over identities.
    • A change in membership re-draws the stratum only when that member leaves or a lower one joins.
    • A test checks both: removing another member keeps the start, and removing the lowest-draw member moves it.
  3. Fixed. A recorded drop must now share the floor's signature fields: tracking, approved_by, approved_on and expires_on. The reason and adjudication texts no longer enter the comparison.
    • A test accepts a drop recorded under older texts and refuses each moved signature field.
    • So María's confirmation edit won't stop a national built at this head from being re-certified later.
  4. H's texts fixed.
    • The floor's note now names both caps: the amounts scope's (seven families, now nine) and the counts scope's (seven, now eight).
    • The cars fuel duty note now says what this PR changed. a58bc1a restated the figures on the September edition beside the June ones, and commit 36 added which edition the engine reads were measured on. The note says the change is pending María's confirmation and the ruling itself is unchanged.
    • The fence's signed reason and its 23 October re-wording still say income_tax_annual > 0. That is María's signed text from before this PR, so it is left for her.
  5. Fixed.
    • Weights: every row is weighted at --year, the calibration year, with its values at its own period, so a 2024 row reads on the solved weights. A test with a stand-in engine pins this.
    • Periods: with --registry, each row's period comes from the run's own registry specs; without it, from the contract. A test pins this too.
    • weight_year.py's docstring now says 0.3%, from the vendored pins.
  6. Fixed.
    • The stage test now checks the take-up strata's route: no stratum mixes carers inside and outside the household. Setting take_up_strata = pool_strata fails it.
    • "Districts and sectors" now reads "districts, unit postcodes and excluded units" in the tool, the layer description and the runtime docstring. I rebuilt the layer from the registered zip. It flags the same 41 and 1,053 areas; only its description and built_on changed, and --check reproduces it.
    • The spine gate kernel's docstring names energy_pricing_code.
    • The body's commit-38 checks line no longer cites mutation runs it doesn't describe.

Checks

Run locally for commit 39: the 34 engine-free test files this round touches or that read what it changed; the Carer's Allowance and weight-year engine tests; the coverage manifest check; the re-review's mutations (local-authority pension cells x1.005, and the take-up strata without the route), each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the rebuilt layer byte for byte. All pass.

🤖 Generated with Claude Code

The first PR C national (built from 10f434d) refused at the national
problem: obr.betting_and_gaming_duties declares the frame's period
2024, but its value, the engine's gambling_duty, is not a stored
column. The national solve measures a declared period only on a stored
column or a CGT dated measure, so it cannot read the engine's 2024
duty. Commit 16 had deleted the row's exclusion under María's ruling
of 2026-10-28 to bind it; no PR C national had been built since, and
the tests do not run the national problem's measure routes.

María's signed exclusion of 2026-10-23 is restored as it was, with a
note in its adjudication that the bind ruling cannot take effect until
the solve can measure the duty (a dated duty measure, or bases bound at
the frame's period), pending her decision. The row stays measured on
every evaluation, beside the other excise rows, which main holds for
the same reason. The row's notes, the consumption-tax tool's docstring
and the tests' census follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Commit 40 (5aba83cd2) puts the betting and gaming row back in the held-out set until the solve can measure it. This undoes commit 16's binding, pending María's decision, and it corrects my reply to the independent review's finding 12, which said the OBR total stays bound.

  • What failed. The first national built from this PR (at 10f434dfd, staging local only) refused at uk.full.national_problem. obr.betting_and_gaming_duties declares the frame's period, 2024, but its value is the engine's gambling_duty, which isn't a stored column. The solve measures a declared period only on a stored column or a CGT dated measure, so it can't read the engine's 2024 duty.
  • Why CI didn't catch it. No national had been built from PR C since commit 16 bound the row, and the tests don't run the national problem's measure routes.
  • Other rows. I checked every other bound row that declares a measurement period. All read stored columns or CGT's dated measures. Main holds the other excise rows out for the same reason as this one.
  • The change.
    • María's signed exclusion of 23 October is restored as it was.
    • Its adjudication notes that her 28 October ruling to bind the row can't take effect until the solve can measure the duty, either through a dated duty measure or through bases bound at the frame's period.
    • The row is still measured on every evaluation.
    • The row's notes, the consumption-tax tool's docstring and the tests' exclusion census are updated to match.
  • Next. The national is rebuilding from 5aba83cd2. On it I'll measure each gambling carrier against HMRC's receipts by duty (finding 12).

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 5aba83cd2 (round 4)

This checks round 3's six items (issuecomment-6514923017) against commit 39 (10f434dfd), then reviews commits 39 and 40. A reviewer re-checked the items and re-ran the mutations. A verifier checked commit 40 and every bound row that declares a measurement period. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing blocks merge.

  • All six round-3 items are fixed. H's approvals stay with María.
  • Commit 40 is right:
    • the restored exclusion matches María's signed 23 October entry field for field, apart from the appended adjudication note;
    • every other bound row that declares a measurement period reads a stored column or a CGT dated measure. That is 24 targets (113 references), re-checked independently with the production predicate.
  • The refusal that commit 40 fixes was a miss in the independent review. Round 1 treated binding the betting and gaming row as a design question (finding 12), and didn't check whether the solve could measure it. A cheap engine-free guard would have caught it, and would catch the next one (1).

Round-3 items

Item Status Evidence
1 Resolved The test runs uk_local_target_surface with each bridge's real control, the committed deferrals and the off-surface cells. It reads 1.0295 and 1.0296, applies ×1.0450 and ×1.0451, and refuses under the ×1.005 mutation. Two residuals (3).
2 Resolved The start is keyed on the lowest-draw member, by the placement's own uniform, with ties broken by person id. A carer alone in a stratum claims at 0.59 at a rate of 0.6. Moving one carer between strata now flips 1.7 of the other 319 flags on average, against 1.3 with code keying; the full re-draw happens only when the lowest member changes. Four start mutants fail the new test.
3 Resolved Only the signature fields are compared. A re-worded record certifies, and each moved signature field refuses. Its consequence is finding 2.
4 Resolved Both caps are named, and the cars note says what changed. The fence text stays María's.
5 Resolved Weights are read at --year, values at each row's own period summed to households, and periods come from the run's registry. One residual (4).
6 Resolved The route test fails the mutant, and the wording is fixed. The rebuilt layer flags the same areas, and its provenance digest matches.

New findings

1. No engine-free test checks that a bound row with a declared period can be measured.

  • The solve refuses a bound plain-column row whose declared period (2024) isn't held on the frame. _refuse_ignored_measurement_period (packages/microcosm-build/src/microcosm/build/target_materialization.py:991-1032) raises a MeasurementPeriodError, which stops the whole run.
  • "Held" means one of the frame's own columns (UKFrameTargetAdapter.value_period, uk_runtime/ledger_targets.py:1806-1818) or a CGT dated measure.
  • Nothing runs this rule on the packaged contract:
    • the existing tests use synthetic bindings;
    • the exclusion census simply moved with commit 16;
    • tools/check_uk_measurement_periods.py runs only after a build.
  • That is why commit 16 passed CI and the national refused at uk.full.national_problem, after the spine and engine stages had run.
  • The same refusal will come back if any excise row is bound again, or if one of the five excise exclusions is left to lapse when they expire on 2027-01-08.
  • A guard is cheap. It would run the production predicate over the packaged contract, membership and exclusions, taking the stored columns from uk_spine_graph() / uk_spine_endpoint(...).inputs and the frame period from load_uk_frs_release().
    • A prototype flags exactly obr.betting_and_gaming_duties on commit 16's register, with the build's message, and nothing at this head.
    • It takes milliseconds after the graph import (about 20 s, shared with the graph tests).
    • Its natural home is tests/engine_free/uk/test_uk_measurement_period.py, next to :536.

2. The signature-only floor check no longer ties a drop to the scope it was made under.

  • After commit 39, a recorded drop certifies if its tracking, approved_by, approved_on and expires_on match the floor (uk_runtime/support_floor.py:98, :577-582). Scope membership is re-checked only against the certifier's own floor.
  • Commit 36 widened the floor to Table 3.7's other income and Table 3.10 without moving approved_on. So a national built at this head that drops one of those rows certifies under the 22 October signature, which never covered them. A probe confirms this for drops recorded at edad8aee9 through this head.
  • The reply's point (that María's confirmation edit won't block re-certification) holds only if she confirms by editing the text and leaves approved_on at 22 October. If she moves approved_on, every earlier run with drops refuses again, which is round 3's finding 3 back.
  • Options:
    • record a digest of the scope (families and caps) in each drop, and compare it;
    • or move approved_on with any widening, and let the reader accept named superseded signatures.
  • The share ceiling isn't re-checked at certification either, but that predates this PR.

3. Two test residuals on the bridge.

  • The ×1.005 test is brittle. tests/engine_free/uk/test_uk_fanout_sum_bridges.py:582-595 hard-codes the scale. At ×1.005 the local-authority factor lands only 0.052% below the 1.025 floor. So if the factor legitimately rises past 1.0301, still well inside the bounds, the test fails with "DID NOT RAISE". Deriving the scale from the baseline (baseline ÷ floor × 1.001) keeps it a mutation test.
  • Lost deferrals in the data aren't caught. If a regenerated membership drops the pension deferrals, nothing notices: the factors read 1.0366 and about 1.030, both inside [1.025, 1.037]. Code-level losses are caught elsewhere. A one-line pin on the committed small_cell_deferrals counts (17 and 3) would cover the data.

4. The diagnostic tool assumes the weights were divided for release.

  • It always projects the stored weights to --year (tools/diagnose_uk_consumption_taxes.py:174-213). That equals the solved weights only on H5s divided for release, which started with 0f0bb86 on 14 October.
  • On H5s written as solved, every row now reads +0.3%. These are nationals released before 14 October, and the engine scratch file. The registry-period path is aimed at exactly those older nationals.
  • main()'s --registry wiring and the map_to argument are untested, and --year is hard-coded at 2025 rather than read from the release's calibration year.
  • Suggestion: load the weights the way the certifier does, with load_uk_national_frame and then uk_solve_year_frame.

5. Smaller.

  • The restored betting exclusion still gives María's 23 October reason ("Binding it would move weight onto the households that gamble to hold a level the build already sets"). Her 28 October ruling rejected that argument, and only the appended note says so. Re-signing, or choosing one of the two routes the note names, is her call.
  • Commit 40's message, PR body item 40, the reply and the tool's docstring say main holds the other excise rows "for the same reason". Their registered reason is María's 23 October ruling: the level is set by the build. They would hit the same refusal if bound, but that is a different statement.
  • The cars fuel duty note doesn't say who re-worded it.

Checked and found true

  • Commit 40's entry: its name, reason, tracking, approver and dates are byte-identical to the entry commit 16 deleted, and to main's. Only the adjudication gains a paragraph. The entry isn't expired; the census test, the tool and the row's notes match it; and PR body item 16 says "Undone by commit 40".
  • Period-declared rows: 30 national targets declare 2024. 24 of them are bound:
    • 8 read stored household columns (bus support and fares, electricity, road fuel);
    • 1 reads a CGT dated measure;
    • 15 are CGT threshold rows that carry the year in the measure name.
      The 5 excise rows and the ORR rail row are held. Local bindings can't declare a period (they're refused at load), and the DESNZ kWh rows are resolver measures read at the frame's period.
  • Commit 39: the start keying, the signature fields, the weighting path and the rebuilt RFDR layer are as in the table, and nothing pins the old layer digest.
  • CI at 5aba83cd2: every finished job passes (engine-uk 3.14, integration-uk, lint, select-countries, wheels). The engine-free, engine-uk 3.13 and engine-us jobs were still running when this was posted.

Not covered

  • Any build or engine test: everything ran engine-free on Python 3.13.
  • The national now rebuilding from 5aba83cd2.

🤖 Generated with Claude Code

The re-review (issuecomment-6516126464) found nothing blocking.

1. test_every_bound_row_with_a_declared_period_can_be_measured_there
   runs the solve's own period check over the packaged contract, with
   the spine's stored columns and the release's frame period: it passes
   at this head and flags exactly the betting and gaming row without
   its exclusion, the refusal the first PR C national hit.
3. The bridge mutation's scale is derived from the committed baseline,
   and the pension bridge's committed small-cell deferrals (17 and 3)
   are pinned.
4. tools/diagnose_uk_consumption_taxes.py reads the solved weights as
   the certifier does (load_uk_national_frame, uk_solve_year_frame) by
   household id, and --year defaults to the FRS release's calibration
   year; tests cover the weighting, the households' alignment and
   main()'s --registry wiring.
5. The tool's docstring gives the excise rows' registered reason (María's
   ruling of 2026-10-23), and the cars fuel duty note says who re-worded
   it.

Finding 2 (the floor's drops and its widened scope) is María's, with H.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-4 re-review. Points 1, 3, 4 and 5 are addressed at 125d6e73b (commit 41). Point 2 belongs with H and is María's.

  1. Fixed. test_every_bound_row_with_a_declared_period_can_be_measured_there (in test_uk_measurement_period.py) runs the solve's own _refuse_ignored_measurement_period over every bound plain-column row of the packaged contract that declares a period. The spine's stored columns (uk_spine_endpoint(uk_spine_graph()).inputs) and the release's frame period stand in for the frame, through UKFrameTargetAdapter.value_period. The test does two things:

    • it passes at this head;
    • with the betting and gaming exclusion removed, it flags exactly that row, which is the refusal the first PR C national hit.

    So the next binding of an excise row, or a lapsed excise exclusion, fails in CI and not at the national problem.

  2. María's, with H. Your reading is right: a drop of a newly covered row would certify under the 22 October signature, which never covered it. The fix depends on how María confirms the widening:

    • if she re-signs by moving approved_on, earlier runs with drops need an accepted list of superseded signatures;
    • if she confirms by text, a scope digest in each drop is the stronger check.

    I've left the choice with the widening itself. For the national now building, the floor's receipt will show whether it dropped any row; if it dropped none, nothing rides on this yet.

  3. Fixed. The mutation's scale is now derived from the committed baseline: the local-authority source factor over the floor, ×1.001. The pension bridge's committed small-cell deferrals are pinned at 17 and 3.

  4. Fixed.

    • The tool reads the solved weights as the certifier does (load_uk_national_frame, then uk_solve_year_frame) and aligns them to the engine's households by id. A mismatch refuses.
    • --year defaults to the FRS release's calibration year.
    • Tests cover the weighting, the alignment and main()'s --registry wiring.
  5. Fixed where the text is ours.

    • The tool's docstring now gives the excise rows' registered reason, María's 23 October ruling. "For the same reason" in commit 40's message, the PR body and my commit-40 note was wrong. The body is corrected, and this reply corrects the note.
    • The cars fuel duty note says it was re-worded by the PR C session, committing in María's name, pending her confirmation.
    • The restored betting reason is María's signed text. Re-signing it, or choosing a route to binding, is hers.

Checks

Run locally for commit 41: the touched engine-free files (measurement period, fan-out bridges, consumption-tax diagnostics, measure simulation, support floor and off-surface cells). All pass.

The national is building from 5aba83cd2. Commit 41 changes tests, the tool and texts only, so the build is unaffected.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 125d6e73b (round 5)

This checks round 4's items (issuecomment-6516126464) against commit 41. One reviewer re-ran the new tests and mutated them, and everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing blocks merge, and the four fixes hold. What's left is low severity.

Item Status Evidence
1 Resolved test_every_bound_row_with_a_declared_period_can_be_measured_there runs the solve's own _refuse_ignored_measurement_period, through UKFrameTargetAdapter.value_period, against exactly the spine's carried columns and the release's frame period. It passes engine-free, even with policyengine-uk blocked. With the betting exclusion removed, it flags exactly obr.betting_and_gaming_duties. It isn't vacuous: with no stored columns, it flags the eight stored-column rows.
2 María's The reply's description is right. More precisely, what matters is a drop outside the 22 October scope (Table 3.7's other income or Table 3.10), not any drop at all.
3 Resolved The mutation's scale is now derived from the baseline, so it crosses the floor for any baseline in [1.025, 1.037]. The deferral pin (17 and 3) fails if either grain loses its deferrals.
4 Resolved The solved weights come from load_uk_national_frame and then uk_solve_year_frame, aligned to the engine's households by household_id. Checked on synthetic H5s: a release-divided file gives back the solved weights exactly, a solved file gives them back as stored, and a reversed engine order still aligns.
5 Resolved The tool's docstring and PR body item 40 give the excise rows' registered reason. The cars note names who re-worded it. Commit 40's message keeps the old wording, but that's history.

Remaining, all low severity

1. A lapsed exclusion still passes CI.

  • The exclusion loader (uk_runtime/measure_simulation.py:1770-1845) neither drops nor refuses an expired entry. The window is enforced only when the national targets are applied.
  • So suppose the excise exclusions are still in the register when they expire on 2027-01-08. CI stays green, and every national then refuses at uk.full.national_targets, after the spine, with the window error.
  • The new test catches a deleted exclusion, but not an expired one. The earliest committed exclusion expires on 2026-12-11.
  • Suggestion: a check on the committed register, like the uprating holds' test_no_committed_hold_is_about_to_lapse, would show each lapse in CI ahead of time.

2. The tool's real weight path has no test.

  • The weighting test passes weights=, and the main() test replaces engine_estimates. So a regression anywhere in load, then solve year, then alignment by id would pass CI.
  • Running the default path on a small synthetic H5 inside the test would close this.

3. The tool's scope and robustness.

  • The docstring (:20) and the --dataset help (:244) still say "any H5 the engine loads". The tool now needs a file that load_uk_national_frame accepts.
  • The id check refuses engine households that are missing from the frame, but silently drops frame households the engine lacks. On the default path both sides read one file, so this only matters for callers that pass weights=.
  • With --registry and no --year, the default year comes from the package's release, not the run's.
  • The tool reads the whole H5 a second time, all three tables with Frame validation, after the engine has already loaded it. The household id and weight would be enough, and that matters on the dense file.

4. Test details.

  • The measurability test uses a literal 2025 as the calibration period (test_uk_measurement_period.py:630), rather than load_uk_frs_release().calibration_year.
  • It doesn't run the solve's fact-period check for require_matching_fact_period rows (packages/microcosm-build/src/microcosm/build/target_materialization.py:748-754). No row is affected today.
  • Its second assertion pins today's held betting row, so it will change when María picks a route.

CI and coverage

  • CI at 125d6e73b: lint, select-countries, wheels, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free and engine-us were still running when this was posted.
  • Not covered: any build or engine test.
  • The national building from 5aba83cd2: commit 41 changes only tests, the tool and one register text, so it doesn't affect that build.

🤖 Generated with Claude Code

The re-review (issuecomment-6516738969) found nothing blocking and only
low-severity items.

1. test_no_committed_exclusion_is_about_to_lapse fails 14 days before
   the earliest exclusion expires, as the uprating holds' clock does, so
   a lapse shows in CI and not at a national's target step.
2. A test runs the tool's real weight path on synthetic H5s, one
   divided for release and one written as solved, through
   load_uk_national_frame, uk_solve_year_frame and the alignment by id.
3. The tool says it needs a national H5 load_uk_national_frame accepts,
   refuses a household on either side alone, and with --registry takes
   the run's own calibration period when --year is not given.
4. The measurability test reads the release's calibration year and also
   runs the solve's fact-period check for rows that require their fact
   at the declared period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-5 re-review. The low-severity items are addressed at 162690525 (commit 42).

  1. Fixed. test_no_committed_exclusion_is_about_to_lapse fails 14 days (UK_HOLD_EXPIRY_NOTICE_DAYS) before the earliest exclusion expires, as the uprating holds' clock does. A lapse now shows in CI, not at a national's target step.
  2. Fixed. test_the_tool_reads_the_solved_weights_of_a_release_and_a_solved_file runs the default weight path on two synthetic H5s, one divided for release and one written as solved. The path goes through load_uk_national_frame and uk_solve_year_frame. The test checks that both read back the solved weights, and that the engine path aligns them to a reordered engine by id.
  3. Fixed, except one item.
    • The docstring and --dataset help now say the tool needs a national H5 that load_uk_national_frame accepts.
    • The id check now refuses a household on either side alone.
    • With --registry and no --year, the tool takes the run's own calibration period from the registry. A test covers this.
    • Not changed: the second read of the H5. The solve-year logic needs the frame's weight record, and the tool is a development diagnostic, so I kept the one code path the certifier uses.
  4. Fixed, except one item.
    • The measurability test reads the release's calibration year.
    • It now also runs the solve's fact-period check: every bound row that requires its fact at its declared period resolves to that period in the committed membership.
    • Not changed: its betting assertion. It will change with María's route.

National build: blocked at certification

The national built from 5aba83cd2 got through the solve, but the calibration-seam fit gate blocked the artifact on two rows in the £1m-and-over band:

  • the savings interest amount;
  • the other-income amount, which this PR binds.

Every lower band of both families fits. I've asked María whether to hold the new £1m-and-over other-income row as a measured distortion, as the ten other £1m-and-over cells are held. That needs her signature, so the build waits for her. The figures stay out of this thread because the band rests on the SPI tape.

Checks

Run locally for commit 42: the measurement-period, consumption-tax diagnostics and measure-simulation test files. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 162690525 (round 6)

This checks round 5's items (issuecomment-6516738969) against commit 42. One reviewer ran the new tests under fake clocks and against in-memory mutants. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing in the code blocks merge. The fixes hold, with two small coverage gaps. The open item is the certification block, and that is María's call.

Item Status Evidence
1 Partly test_no_committed_exclusion_is_about_to_lapse uses the holds' clock: 14 days ahead of date.today(). Under fake clocks it passes until 2026-11-26 and fails from 2026-11-27, naming 2026-12-11. It doesn't cover the support_floor block in the same register (finding 1).
2 Resolved The new test runs the real loader and the real solve-year step on two synthetic H5s. Skipping the solve-year step, aligning by position, or reverting the id check each makes it fail.
3 Resolved The id check refuses a household that appears on only one side. The registry's single compiled period sets the default year, with a fallback to the release's year. The second read of the H5 stays, by choice.
4 Partly The test reads the release's calibration year and runs the fact-period check, which catches a mutated plain-column row. It skips the 15 bound CGT rows on the provider route (finding 2).

Findings (all low)

1. The lapse clock skips the support floor.

  • The floor block (uk/calibration_measure_exclusions.json:3-82, in force until 2027-01-08) is enforced at the same targets step. uk_runtime/full_targets.py:122-139 calls the floor's assert_in_force.
  • But the new test reads only the per-row list, and test_uk_support_floor.py pins its own date.
  • Today the floor is covered only by coincidence: seven per-row entries also expire on 2027-01-08. If those are renewed and the floor isn't, CI stays green and every national refuses from 2027-01-09 with "UK support floor is outside its reviewed window".
  • Adding the floor's expires_on to the clock closes it.
  • Also, min() there has no default. If the last per-row exclusion is ever bound, the test errors instead of passing. The sibling clocks share this pattern.

2. The fact-period check misses the CGT provider rows.

  • The test skips any binding with a kind, but the solve checks the fact period before it branches on kind.
  • 18 bound rows require their fact period. The test checks 3 of them (CGT liability, electricity and road fuel), and not the 15 parameter_gated_threshold CGT rows.
  • So a re-pin that resolves one of those rows to a 2025 fact passes CI, and the national stops after the spine. A bound row missing from the membership also passes.
  • The reply's "every bound row" overstates the coverage.

3. Smaller.

  • The registry-year test hard-codes 2026, so it goes vacuous once the release's calibration year is 2026. Using calibration_year + 1 would hold.
  • A registry with mixed periods falls back to the release's year silently.
  • Older than this commit: composition() reads the H5's stored (dataset-year) weights, while the estimates now use the solved weights. Its totals therefore sit about 0.3% off; the shares are unaffected.

The certification block

  • The national built from 5aba83cd2 stopped at the calibration-seam fit gate on two £1m-and-over amount rows: savings interest, and the other income this PR binds. As in the reply, no figures for that band here.
  • Round 1's removed-behaviour pass raised the other-income row: that £1m-and-over cell is bound without the distortion measurement that held ten sibling cells out. The verifier set it aside under the 21 October rule, which holds a row out only on a measured distortion. The build has now measured it.
  • The savings-interest £1m-and-over row was already bound on PR A's certified national. What this PR changed for it is commit 32's uprating: the 13 interest amounts now grow a further ×1.134 on the OBR savings path, which Vahid's finding 1 questioned.
  • So before deciding, it may be worth checking whether holding only the new other-income row clears the gate, or whether the interest row's new level is part of the miss. That is María's call.

CI and coverage

  • CI at 162690525: lint, select-countries, wheels, integration-uk and engine-uk 3.13 pass. The other jobs were still running when this was posted.
  • Not covered: any build or engine test.

🤖 Generated with Claude Code

- The measure-exclusion lapse clock also reads the register's support
  floor, which the national targets refuse at the same step, and passes
  on an empty register instead of erroring.
- The measurability test runs the solve's fact-period check on every
  binding kind, before it branches on kind, as the solve does, so the 15
  CGT provider rows are covered. A required row missing from the committed
  membership also fails. A new test mutates one CGT row to a later fact
  and drops another from the membership; both fail.
- diagnose_uk_consumption_taxes.py weights its composition with the
  solved weights by household id, as the estimates are, reading them once
  in main(). With --registry and no --year, a registry whose specs span
  several periods refuses rather than falling back. The registry-year test
  uses the release's calibration year plus one, not a literal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-6 re-review. All three findings are addressed at 059225c89 (commit 43).

  1. Fixed.
    • test_no_committed_exclusion_is_about_to_lapse also reads the register's support floor (load_uk_support_floor_declaration), which the national targets refuse at the same step. The failure message names every entry that lapses on the earliest date, the floor included.
    • On an empty register the test passes instead of erroring.
    • The three sibling clocks (uprating holds, small-cell rule, local adjudications) predate this PR and are left as they are.
  2. Fixed. You're right that the round-5 reply's "every bound row" overstated the coverage.
    • The helper now runs the fact-period check on every binding kind, before it branches on kind, as target_materialization does. So the 15 parameter_gated_threshold CGT rows are covered.
    • A required row with no active candidate in the committed membership also fails.
    • test_the_fact_period_check_covers_every_binding_kind is the mutation: with one CGT provider row resolved to a 2025 fact, it flags exactly that row; with another row dropped from the membership, it flags exactly that one.
  3. Fixed.
    • The registry-year test uses the release's calibration year plus one.
    • With --registry and no --year, a registry whose specs span several periods now refuses ("pass --year"). A test covers it.
    • composition() is weighted with the solved weights by household id, as the estimates are. main() reads them once and passes them to both.

The certification block

I checked your question against the build's own fit report: is commit 32's interest uprating part of the £1m-and-over miss, or is it only the new other-income row? The answer, and the band's ratios, went to María privately with the decision. The band's figures, and the direction of either miss, stay out of this thread. Before she signs, I've also offered her a local, unsigned re-solve of this national problem with the other-income row held, to confirm the gate clears.

Checks

Run locally for commit 43: the consumption-tax diagnostics, measurement-period and measure-simulation test files, and ruff on the touched files. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 059225c89 (round 7)

This checks round 6's findings (issuecomment-6517309229) against commit 43. One reviewer re-ran round 6's mutations and ran the new clocks under fake dates. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: all three are fixed, and nothing in the code blocks merge. Two low items remain in the measurability helper. The certification block stays with María.

Finding Status Evidence
1 Resolved Under fake clocks, renewing all 39 per-row entries but not the floor now fails from 2026-12-25, naming support_floor. An empty per-row list no longer errors.
2 Resolved The fact-period check now runs before the kind branch. Moving any of the 18 bound required rows to 2025 flags exactly that row, and so does dropping one from the membership; that includes the 15 CGT provider rows. The plain-column part still flags betting when its exclusion is removed.
3 Resolved composition() uses the same refusal and id alignment as the estimates. main() reads the solved weights once and never weights by the stored column. A registry with mixed periods refuses.

Low items

1. The new clock's first deadline. It is by design, but worth noting now: from 2026-11-27, test_no_committed_exclusion_is_about_to_lapse fails on every PR until orr.rail_government_support (expires 2026-12-11) is renewed or bound.

2. The measurability helper's account of holds differs from the solve's (packages/microcosm-build/tests/engine_free/uk/test_uk_measurement_period.py:613-645).

  • Possible false positives. Some of the 15 CGT provider targets fan out into one row per region or band, which the check reaches only since commit 43. Two cases misfire:
    • The helper still checks a fan-out row that is held on its own by a per-row exclusion, although the national drops that spec before materialising it. Example: a re-pin moves one CGT region's fact, and the reviewer holds just that row. The national runs, but this test fails.
    • It also flags a fan-out target whose rows are all signed out one by one.
  • Wider loop. It loops over all 456 contract targets, against the 394-target national membership. So a local-only target that later declares require_matching_fact_period would be flagged too.
  • Latent miss. Held rows inside the floor's scope are compiled as probes and checked by the solve, but the helper skips every held target. No such row declares a period today.
  • Suggestion: skip a row whose own name is held, unless its target is in the floor's scope, and restrict the loop to national targets.

3. The docstrings overstate the "no active candidate" rule (:572-576, :678-681).

  • They present it as the solve's own refusal ("stops the national after the spine"). In fact the national never reads uk/target_reference_membership.json: it compiles uk/target_references.json, which holds active rows only.
  • So a re-pin that defers a required row drops it from the national silently.
  • The test does catch that case, which makes it the only guard. The docstrings should say so rather than describe a build refusal.

CI and coverage

  • CI at 059225c89: lint, select-countries, wheels, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free and engine-us were still running when this was posted.
  • Note on engine-free: that job was cancelled unfinished on the previous three heads by later pushes, so the PR's engine-free result is still to come.
  • Not covered: any build or engine test.

🤖 Generated with Claude Code

The measurability helper now reads the rows the national compiles, as
graph_national does:
- the targets on the national line only (national geography and the
  national calibration line), not every contract target;
- less the rows signed out of a fan-out;
- less the rows a per-row exclusion holds, except held rows in the
  support floor's scope, which the national compiles as probes and so
  checks.

A required row that is deferred (not active) is now flagged, like a
row missing from the membership. The national compiles only the active
references, so either one leaves the national without a refusal, and
this test is the only guard. The docstrings now say so, instead of
describing a build refusal.

A new test covers the three hold cases: a row held on its own with a
moved fact passes; the same row in the floor's scope is flagged; a
fan-out signed out row by row passes. The deferral case is added to the
fact-period test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-7 re-review. Items 2 and 3 are addressed at 54d9d5b6d (commit 44). Item 1 is a deadline for María.

  1. Noted, by design. From 2026-11-27, test_no_committed_exclusion_is_about_to_lapse fails on every PR until orr.rail_government_support (expires 2026-12-11) is renewed or bound. Both are María's signature. I've added the date to the PR body, under what merging waits on, so it doesn't surprise anyone.
  2. Fixed, as you suggested. The helper now reads the rows the national compiles (graph_national):
    • Targets: only those on the national line (national geography and the national calibration line). That is 391 targets, not the contract's 456, so a local-only target can't trip it.
    • Signed out: rows signed out of a fan-out are skipped. A fan-out signed out row by row compiles nothing and isn't flagged.
    • Held: a row held by its own per-row exclusion is skipped, unless its target is in the support floor's scope. Those held rows are compiled as probes, so they're checked, which closes the latent miss.
    • Test: test_the_period_checks_read_the_rows_the_national_compiles covers each case. A CGT region row with a moved fact passes when held on its own, is flagged when its target is in the floor's scope, and passes when the fan-out is signed out row by row.
  3. Fixed.
    • The docstrings now say the national compiles uk/target_references.json, the active references only. So a required row that is deferred or missing from the membership leaves the national without a refusal, and this test is the only guard.
    • I also widened the guard to the deferred case you describe. Before, only a required target with no active row at all failed. Now any required row that is not active fails, and the fact-period test checks it with a deferred CGT age-band row.
    • No required row is deferred in the committed membership.

Checks

Run locally for commit 44: test_uk_measurement_period.py (20 tests) and ruff. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 54d9d5b6d (round 8)

This checks round 7's items (issuecomment-6518130454) against commit 44. One reviewer ran all 20 tests in the file, and re-ran round 7's cases in memory on both the new and the old helper.

Verdict: both items are fixed, and nothing in the code blocks merge. The review has nothing further on the code. What's left is María's: the certification decision, H's approvals, and the 27 November rail deadline, which is now in the body.

  • Item 2: Resolved.
    • The helper uses the national's own predicates: uk_contract_target_calibration_lines and the national-geography contract. The 391 targets are correct: 456 contract targets, 394 of them national, and 3 of those only on the local line.
    • The probe rule matches graph_national (floor.in_scope on the contract id).
    • All of round 7's cases now behave as the national does:
      • a CGT region row held on its own passes;
      • a fan-out signed out row by row passes;
      • a held row in the floor's scope with a moved fact is flagged.
    • Every assertion in the new test fails against commit 43's helper.
  • Item 3: Resolved.
    • The widened rule flags a deferred required row for every deferral class the authoring emits.
    • It never flags signed-out, whole-target-signed or per-row-held rows. The committed membership gives no flags.

Two notes, both on suggestions the review itself made in round 7:

  • "The only guard" was wrong. test_uk_cgt_targets.py pins the CGT rows' names and their active status in uk/target_references.json, so a deferred CGT row on a real re-pin fails those tests as well. The docstrings at :585-589 and :736-738 would be accurate as "no build step refuses; this is the check tied to the period requirement".
  • Narrowing to the national line drops one check from the dense build. The plain-column check no longer covers the 3 high-value-homes targets on the local line only, which the dense build compiles through the same _refuse_ignored_measurement_period. None of them declares a period today, so nothing is needed now. If wanted, run just the plain-column part over national-geography targets on both lines.

CI at 54d9d5b6d: lint, select-countries, wheels, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free and engine-us were still running when this was posted.

🤖 Generated with Claude Code

- The measurability helper's docstrings no longer call the deferral
  check "the only guard". No build step refuses a required row that a
  re-pin defers or drops from the membership, so this is the check tied
  to the period requirement, and test_uk_cgt_targets.py also pins the CGT
  rows' active status.
- The plain-column check again covers the national-geography targets on
  the local line only, which the dense build compiles through the same
  check. The fact-period part stays on the national line. A new test
  declares a period on a local-only high-value-homes target and sees it
  flagged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-8 re-review. Both notes are addressed at 0b276e571 (commit 45).

  • "The only guard." You're right: test_uk_cgt_targets.py also pins the CGT rows' names and active status. The helper's and the test's docstrings now say no build step refuses a deferred or missing required row, and that this is the check tied to the period requirement. They name the CGT tests beside it.
  • The local line. The plain-column check again covers the national-geography targets on the local line only (the three high-value-homes region targets), which the dense build compiles through the same _refuse_ignored_measurement_period. The fact-period part stays on the national line. test_the_plain_column_check_covers_the_local_line_national_targets declares a period on one of them and sees it flagged; the committed contract gives no flags.

What remains is María's: the certification decision on the £1m-and-over other-income row, H's approvals, and the 27 November rail deadline.

Checks

Run locally for commit 45: test_uk_measurement_period.py (21 tests) and ruff. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 0b276e571 (round 9)

Both of round 8's notes are addressed in commit 45. The review has nothing further on the code.

  • The docstrings now say that no build step refuses a deferred or missing required row, and they name test_uk_cgt_targets.py beside the check.
  • The plain-column check again covers the three high-value-homes targets that sit on the local line only, while the fact-period part stays on the national line.
    • To take injected targets, the helper now writes out the national-geography rule itself. It uses the same expression over the same UK_NATIONAL_TARGET_GEOGRAPHY_LEVELS as _uk_contract_targets(national_only=True) (uk_runtime/ledger_targets.py:2028-2029).
    • The only thing it drops is that function's warning about undeclared geography.
  • Tests: the four measurability tests pass at this head (run engine-free), including the new local-line test.

What remains is María's: the £1m-and-over certification decision, H's approvals, and the 27 November deadline for the rail exclusion.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 62f00b6 into main Oct 30, 2026
10 checks passed
juaristi22 added a commit that referenced this pull request Oct 30, 2026
PR C (#1420) merged first, pinning the UK Chronicle feed to 53815b2
(PolicyEngine/chronicle#394), which contains #392's ISA statistics. So
this branch takes main's pin and regenerates on the merged tree with
policyengine-uk 2.164.2 (Vahid's review of #1421):

- Main's versions, since the feed is main's:
  - every file that restates the feed: the pin, the 34 shared vendored
    resources, the DESNZ constituency resource and both memberships;
  - the census, the validation levels and the two tests restating the
    pin.
- Regenerated on the merged tree:
  - every vendored resource, from the 53815b2 artifact. The 34 shared
    ones come out byte-identical to main's, and
    hmrc_isa_statistics_facts.json keeps its 1,074 rows under the new
    feed identity;
  - the national compile-parity receipt. The 2.164.2 pins move #1420's
    eight SPI Table 3.10 self-employment bands by -0.55% and the dividend
    bands by under 0.001%. The savings interest bands now follow the
    OBR's savings income growth, so the pins no longer move them, and the
    2.164.2 changelog says so;
  - the coverage manifest, the H2 fixture and the contract digests.
    The engine pins, the stage projection and the concept coverage were
    already consistent.
- Both sides kept:
  - graph_kernels.py's imports;
  - the vendor selections: the ISA entry beside #1420's OBR savings
    growth;
  - the runbook history: #1420's 53815b2 entry, plus a paragraph saying
    the ISA resource comes from that artifact.
- Vahid's nit on the taper: the docstring states its bound (a factor
  above limit / GBP 18,000 leaves the cap alone binding), and the
  receipt records it. The regenerated fixture binds this module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 30, 2026
…H5 round trip

- After the merge of main, this PR no longer re-pins the UK Chronicle
  feed: #1420 moved it to 53815b2, which carries PolicyEngine/chronicle#392's
  ISA statistics, and #1420's fragment records that re-pin. The
  1415-uk-chronicle-repin-isa-statistics fragment, which announced a re-pin
  to #392 with e56af9a's counts, becomes an added fragment for what this PR
  adds: the vendored hmrc_isa_statistics_facts.json from the pinned feed.
  The method doc names the 53815b2 pin as the source.
- The frame serializer registry's UK single-year round trip passes the ISA
  income contract that _write_uk_single_year_tables requires since the
  microcosm#1360 commit. Engine-free CI stopped on the register test first
  at a4785ef, so this failure only showed at 351fbfa.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 30, 2026
Main now carries #1420 (electricity at the ONS level, pension income by
area, Rural Fuel Duty Relief areas, the balanced Carer's Allowance draw)
and #1421 (policyengine-uk 2.164.2, ISA subscriptions, the Lifetime ISA
anchor, the additive ISA contract), with the feed at Chronicle 53815b2.

Hand-merged:
- uprating_holds.json: main's holds, re-pointed to microcosm#1095, less
  the three DfE in_year_snapshot holds this branch retires (D5), plus the
  signed SLC nation hold (R4) and main's ONS electricity hold.
- gates.json: the export surface keeps main's tax_free_savings_income
  exclusion (microcosm#1360) and drops the all-zero incapacity benefit
  layer as this branch does; its notes name both.
- sources.yaml: main's Carer's Allowance stage note, with its receipt
  floor restated at three source households.
- weight_year.py keeps the docstring that names no lock's growth figure.
- calibration_run.py, test_country_spec.py, contract.py and
  test_contract.py list both sides' new gates in gates.json's order.
- The tests take the merged counts: 458 targets (registry 396, profile
  62; mapped 185, unmapped 211), 44 measure exclusions, and 34 support
  floor rows over 61 area entries ({E06000053: 27, E09000001: 34}).
- The Carer's Allowance receipt test takes main's balanced placement at
  the floor of three; two stale comments restate that floor.

Regenerated on the merged tree with policyengine-uk 2.164.2:
source_stages.json, the release input coverage manifest (143 required,
2 reviewed exclusions), the gate battery, spine-part and release-cut
digests, and the charter-H2 fixture. The national and local references,
membership and compile-parity receipts regenerate next, on feed 53815b2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 30, 2026
The one regeneration this PR owed after merging main: the national and
local references, membership and compile-parity receipts on main's feed
pin, Chronicle 53815b2 (#1420's re-pin), with policyengine-uk 2.165.1.
The vendored resources check against the full artifact unchanged, and
the local census, the local references, the local parity receipt and
the coverage manifest regenerate unchanged.

National surface (1,439 candidates, 396 contract targets):
- 1,409 active references, up from 1,403, and no row without a fact (4
  before): the three OBR council tax nation rows pinned to the UK (D14e,
  held as measured diagnostics) and SLC's three nation rows (R4; Wales
  binds, Scotland and Northern Ireland are held).
- The eight DfE rows select the January 2026 census (observation_period
  2026-01) at model year 2025, with no uprating (D5).
- Scottish Child Payment, the Tax-Free Childcare top-up, Northern
  Ireland's bus receipts and SLC's four England repayment rows take the
  calendar-2025 window (R7).
- obr.council_tax binds the main-residence share of OBR's receipts
  (D14f).
- The national compile-parity receipt: 1,409 compiled, 1,417
  differences (calibration drift 561, fixture only 42, ledger only 814).
  The incumbent's three devolved SLC metrics now match the nation rows
  exactly.

Local surface: 27,168 active references, every status count unchanged.
The membership's signed deferrals fall from 94 to 77: a target scoped to
Wales or Scotland no longer defers the two English micro-authorities
(microcosm#796, via #1095). The MSOA income deferral carries D18a's
signed reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 31, 2026
… surface

CI's engine-free lane failed on the regenerated surface:

- R4's three SLC nation rows declared the frame's own year
  (measurement_period 2024) under a matched-period hold. The solve
  honours a declared period only at the calibration year, or at the
  frame's year for a value the frame stores, and student_loan_repayment
  is computed by the engine (#1420's period check). So the national
  solve would refuse slc.repayments.wales_total, the one that binds.
  María's ruling of 2026-10-30: carry SLC's 2024-25 outturn to 2025 by
  policyengine-uk's OBR average earnings index, since repayments are a
  share of earnings above the plan thresholds. The three rows drop the
  declared period and declare the index, and their matched-period hold
  retires. Scotland and Northern Ireland stay held out of the solve by
  their own signed exclusions (their plan thresholds are not modelled).
- The fixture subset compile read SLC Plan 2 with 2024-25 alone, so R7's
  window refused it. The four Plan 2 rows (full-time and part-time,
  2024-25 and 2025-26) now come verbatim from the pinned feed, and the
  staged-rows test pins the windowed value.
- The pins follow the regenerated surface: 1,409 active references, no
  row without a fact, and 34 support-floor rows over 61 area entries.

The national references, membership and compile-parity receipt are
regenerated for the three uprated rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Oct 31, 2026
María's ruling of 31 October: the 19 OBR targets that window FY2024-25 and
FY2025-26 into calendar 2025 take FY2024-25 from the March 2026 EFO and
FY2025-26 from the October 2026 EFO, each year's latest edition after
PolicyEngine/chronicle#397. Their selectors declare
source_table_by_opening_year, and each note says so.

- Fifteen of the 19 move by under 1%: income tax -0.5% and UC +0.7%. The
  four larger moves are domestic rates +3.2%, Pension Credit +2.3%,
  Jobseeker's Allowance -2.5% and SDLT -1.2%.
- Three report-only diagnostics pinned March's FY2025-26 forecasts, which
  the feed no longer carries: the OBR's capital gains tax receipts, State
  Pension spending and total NICs. They pin October's outturns
  (observation, observed_only), with their notes restated. The NICs total
  now sits 2.1% above the class rows at the window.
- The test feed fixture's captured March rows give way to the feed's
  current ones: October's FY2025-26 income tax, incapacity benefits and
  capital gains tax, and March's FY2024-25 income tax in its v4 form. The
  national calibration tests' synthetic window facts carry each year's
  table.
- Regenerated from the 08962f1 national cut: the national references and
  membership, and the national compile-parity receipt. The receipt's OBR
  rationales name the two editions, and Winter Fuel, which binds FY2025-26
  alone, has a rationale of its own.
- The membership also catches up with policyengine-uk 2.164.1's October-EFO
  indices, which #1421 locked after #1420 had generated it: the eight SPI
  Table 3.10 self-employment bands move -0.55% and the thirteen dividend
  bands under 0.001%. The receipt already had these values.
- docs/uk-income-anchors-280.md describes the map, the re-pin runbook gains
  the 08962f1 entry, and the changelog fragment is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants