Cite US and UK spec sources by symbol or at a commit, never by a drifting line

changelog.d/spec-source-citations-us-uk.fixed.md

@@ -0,0 +1 @@
+Cite source code by symbol, or by line at a named commit, in the US and UK spec packages. A new engine-free guard (`test_spec_source_citations.py`, validators in `test_support/microcosm_build/spec_source_citations.py`) scans every resource in every country package. It refuses a line citation of a file in this repository, pinned or not, including one written as a `PolicyEngine/microcosm` GitHub URL at any ref (`main`, `refs/heads/main`, `feature/x`) or a `microcosm/tools/...` path. It refuses a line citation of any other source unless the citation binds to an inline commit (`commit <sha>`, `at <sha>`, `@<sha>`, or a GitHub `/blob/`, `/tree/`, `/raw/`, `/blame/` or `/commit/` URL): the nearest one in the citation's own clause, else the nearest one before it in its string, unless the citation's clause names a moving ref (`on main`, ``on `main` ``, or a `/blob/main/` or `raw.githubusercontent.com/<owner>/<repo>/main/` URL). A resource's `path`/`commit` mapping pins it too, but only when the cited path ends the pinned path. A commit-like token in no recognised spelling beside a citation is refused as a misspelled pin. So is a line locator after a source name in no recognised spelling (`x.py some_function, lines 12-19`). A truncated hash (`e2de2b41…`) or a hex directory (`at a1b2c3d/foo.py`) pins nothing, and a quoted locator (`x.py:"12-19"`) is a line citation like any other. Each URL is read the way a browser and then GitHub read it first. Its authority is parsed as the WHATWG URL Standard parses it: `\` read as `/`, any number of slashes after `https:` or in a `//host/...` reference, a user and a port dropped, and the host percent-decoded, mapped to ASCII as IDNA maps it and lower-cased, so `%67ithub.com`, `ｇithub.com` and `user@github.com:443` are `github.com`. Then `%` escapes of unreserved characters and `/` in its path are decoded, `.` and `..` segments resolved, and its query dropped, so `%74ools/x.py` and `x.py?plain=1#L12` are line citations of `tools/x.py` and `x.py`. A URL runs to whitespace, so a quote or a bracket in its path or query (`x.py?q='a'#L12`) is part of it. Markdown and prose end a URL at a quote, a parenthesis or an angle bracket, so a string is read that way too where it differs: in `[https://a/?x](https://github.com/...#L12)` the label's query does not hide the link. A string is also read with its Markdown and HTML escapes decoded (`x\.py`, `&#35;L12`): as Markdown decodes them and as HTML does, which leaves a backslash as written (`https:\&#47;github.com` names github.com), each as text and as a URL holds what it decodes to, with references decoded as the HTML standard decodes them. Each reading is also read with its ASCII tabs and line breaks deleted, as the URL Standard reads a URL in an HTML attribute. A quoted attribute value or a Markdown destination in angle brackets that starts a URL is read as the browser follows it, its escapes decoded before its ends are stripped and its spaces percent-encoded, and a URL is read through a Unicode space such as U+00A0, which the URL Standard percent-encodes. A destination in angle brackets is found after the spaces, tabs and one line break CommonMark allows before its `<`. Where a string holds a `?` and a `#`, its URLs are also read through whitespace in their query, which GitHub ignores (`href=".../x.py?a b#L12"`). A citation any reading finds is checked. Every line citation in a string that holds a dot segment (`..` as a whole path segment, either dot perhaps `%2e`) is refused, pinned or not: whitespace, a backslash or markup in the segment it removes can make a renderer remove another one than any reading does, as in `[a]( <https://github.com/.../tools/a b/../x.py#L12>)` or `[a](https://github.com/.../tools/a\b/../x.py#L12)`, where Markdown writes the backslash `%5C`. No resource cites a line in such a string. GitHub reads `%2f` as `/` before it resolves a path, so a `..` beside one is a dot segment too (`tools/a b%2f..%2fx.py`). A string is also refused where any of its readings' sources holds this repository's GitHub path, `PolicyEngine/microcosm/`, with a line fragment (`#L12`) at the first `#` after it, whatever file the link names and whether or not a reading finds the citation. The path is read as a browser and then GitHub read one: a backslash as `/`, as in an HTML attribute, and again as no separator, as a Markdown renderer escapes it (`%5C`); dot segments resolved first (either dot perhaps `%2e`, an empty segment counted); then escapes decoded, a run of slashes read as one and the owner and name matched in any case. So a dot segment that forms the path (`PolicyEngine/x/../microcosm/blob/<sha>/LICENSE#L3`) is refused even where the file's name is not one a reading cites. So a ref holding punctuation (`blob/feature+foo/tools/x.py#L12`), or whitespace before the host where markup has taken an attribute's opening quote (`href="https://a b@%67ithub.com/..."`), no longer hides a link of this repository. No resource holds such a link. A path here can hold an `=` (`docs/.../period=2025-01.json`), so the run of path characters before a cited name is tried as an in-repository path too, and a reverse locator reads a name after an `=` (`lines 3 of a/b=c.py`). An in-repository path's inner `..` segments are resolved (`tools/x/../y.py` is `tools/y.py`). In `us/ecps_parity_known_gaps.json`, the `asec_pool.py`, `tools/build_us_puf_support_base.py` and `experiments/build_j_recert/*` line citations now name `microcosm.build.us_runtime.asec_pool` symbols, the tool's functions, the `--asec-h5` arguments and `base_j.summary.json` keys. Where #720's Census person-column restore had made a sentence false, the sentence now says which columns the restore leaves out, and tests check those columns against `ASEC_CENSUS_PERSON_COLUMN_NAMES`. The us-data citations in `us/source_stages.json` and `us/target_parity_manifest.json` bind to archived commit `42ed5d45`. The PUF note now names that commit before its disclosure-record and disaggregator citations rather than after them. The manifest's ESI origin spells it `archived commit 42ed5d45 datasets/cps/cps.py` instead of leaving another commit in the string to pin it. The uk-data citations in `uk/uk_data_target_parity.json` name the register's incumbent commit `a43e8c5`, and those in `uk/uk_population_targets.json` name its registry ref `12a1e028`. Reading each cited range at its commit also corrected seven citations that pointed at the wrong code: SSI take-up (`cps.py` 584 / 1497-1499 labels swapped), SCF net worth (`source_impute.py` fit at 1227-1240), `hmrc_cgt.py` 195 to 196, `hmrc_spi.py` 230 to 229, `frs_only.py` to `services.py` 137-148, `_land.py` 52-73 (the file has 65 lines) to per-sector ranges, and the salary-sacrifice drop (`build_loss_matrix.py` 376-394 to the `except Exception` in `hmrc_salary_sacrifice.py`). Re-pinned: the `source_stages.json` raw hash (`sources.yaml` `stage_asset`, the generator's frozen hash, `us_spec_bundle.py`, `test_us_bundle_core_contracts.py`), the regenerated `us/spec/sources.yaml`, the US `spec_sha256` and `documentation_sha256`, three `inventory_coverage.EXPECTED_HASHES` (`source_manifest`, `late_resource_semantics`, `full_checkpoint`) and `docs/evidence/spec-engine/us-f0-coverage.json`. Existing stacked US pool stage checkpoints will be treated as stale and rebuilt. No behaviour or data changes.

docs/evidence/spec-engine/us-f0-coverage.json

@@ -4,7 +4,7 @@
     "version": 1
   },
   "country": "us",
-  "documentation_sha256": "801a99da54b8a4834a8c5e6c4b013fdae1e7576e0132de7e87cb1ef098f1ad8c",
+  "documentation_sha256": "811b4091f9419c8792e5cc7fcda30d54c5a7c6d35dd944c26c005e32cae06647",
   "field_usage": {
     "authored_normative_field_count": 32454,
     "claim_count": 49,
@@ -1361,12 +1361,12 @@
         ],
         "expected": {
           "producer_count": 38,
-          "sha256": "531b48cd1f9737c3ba04aaa430a1aaa25d035217b59be9da24844948687e78f1"
+          "sha256": "6634f7512bee554e291b635860e0efdb90b38647099be43e8e85ccfea3f9221d"
         },
         "failures": [],
         "observed": {
           "producer_count": 38,
-          "sha256": "531b48cd1f9737c3ba04aaa430a1aaa25d035217b59be9da24844948687e78f1"
+          "sha256": "6634f7512bee554e291b635860e0efdb90b38647099be43e8e85ccfea3f9221d"
         },
         "status": "covered"
       },
@@ -1775,12 +1775,12 @@
           "legacy_adapter.imputation.late_producer_resource_semantics"
         ],
         "expected": {
-          "sha256": "87b1e124969ec11b836740c027d28dc2a9d319d279ed55be1d9fc1523390f634",
+          "sha256": "10c05f0fd057f75536b47e3a0160ffb2b47cdec094b6694ebd62cc6bef50d9e8",
           "stages": 38
         },
         "failures": [],
         "observed": {
-          "sha256": "87b1e124969ec11b836740c027d28dc2a9d319d279ed55be1d9fc1523390f634",
+          "sha256": "10c05f0fd057f75536b47e3a0160ffb2b47cdec094b6694ebd62cc6bef50d9e8",
           "stages": 38
         },
         "status": "covered"
@@ -1887,7 +1887,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "030c842261a9f0b2378443633b92f96a876cdbc9cb9e75433967fbf97fdeec1c"
+          "sha256": "c88b0d7fce767e7415c10bfd003b6eb0508244775605a5adf27a430299ffbb4e"
         },
         "failures": [],
         "observed": {
@@ -1911,7 +1911,7 @@
             "alpha",
             "zeta"
           ],
-          "sha256": "030c842261a9f0b2378443633b92f96a876cdbc9cb9e75433967fbf97fdeec1c"
+          "sha256": "c88b0d7fce767e7415c10bfd003b6eb0508244775605a5adf27a430299ffbb4e"
         },
         "status": "covered"
       },
@@ -1975,7 +1975,7 @@
             "take_up_contract",
             "us_qbi_reconciliation_contract"
           ],
-          "sha256": "af44d50258bd8dc0fdc1c06e798b0a9f6f48d48d403c217072a9808d96b152f4"
+          "sha256": "bbd77902cc3254268df0cf138a0cfef7b2b0effb931747fd0ce92b3cbf403b0c"
         },
         "status": "covered"
       },
@@ -2599,7 +2599,7 @@
       "country": "us",
       "schema_id": "country_spec",
       "schema_version": 1,
-      "spec_sha256": "c677d50bd0bbcd76c8c141c2c281a7bd49145369f08da8d5db324f2febbada5f"
+      "spec_sha256": "b5c87a41b665901bdbd08a9cefd816b9c93b26c1928e841f99435ee895f24278"
     }
   },
   "report_schema_version": 3,
@@ -2609,7 +2609,7 @@
     "country": "us",
     "schema_id": "country_spec",
     "schema_version": 1,
-    "spec_sha256": "c677d50bd0bbcd76c8c141c2c281a7bd49145369f08da8d5db324f2febbada5f"
+    "spec_sha256": "b5c87a41b665901bdbd08a9cefd816b9c93b26c1928e841f99435ee895f24278"
   },
   "status": "pass"
 }

packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py

@@ -350,12 +350,12 @@
     "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25",
     "authority": "9d4a9672a0f03039b1fe874b9fe21ed575be0d29f14afc396d03cdf5c809bdd2",
     "early_families": "4aa9f736fd76e83955477ad1667e58f48f264783f05bdc7f0102cd32d61323bd",
-    "full_checkpoint": "030c842261a9f0b2378443633b92f96a876cdbc9cb9e75433967fbf97fdeec1c",
+    "full_checkpoint": "c88b0d7fce767e7415c10bfd003b6eb0508244775605a5adf27a430299ffbb4e",
     "gap_fill_schedule": "1c31f9868f7884347cc19cf1ff65da43f950b9114941a715bab168246db414a7",
     "graph_nodes": "40cd51ffdfe2e9d9d08d48c08e8ded9de1e4b134783bab05c4abc6ad5c72ca1e",
     "geography_assignment": "44e7f5f6cd7ceeef85d532c3d46306d03de4f4ee811376e6d29920742746b65c",
     "late_families": "d91f9ff0eb52f43e7b6eed3d5c58c37abe1620c3a11021da15dae9c10e16d382",
-    "late_resource_semantics": "531b48cd1f9737c3ba04aaa430a1aaa25d035217b59be9da24844948687e78f1",
+    "late_resource_semantics": "6634f7512bee554e291b635860e0efdb90b38647099be43e8e85ccfea3f9221d",
     "late_schedule": "e59c019d3d454eac99ac0ac209b6c5b6faaf9bdfcaeee18c36a25be19bf7da2f",
     "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356",
     "primary_tuples": "fdf23da429f8c501e198a2f5b719763d523a41565c2153fdf17b0344f21bcd7c",
@@ -373,7 +373,7 @@
     # self-employment component step changes acs_transfer's attested bytes.
     "seed_map": "eb6b685478a35cbbf3295773c8396527d70167a4675432afa7530b0b64b611b2",
     "seed_protocol": "961eb7d68291fb5ed45fa96f6949af91a315a0c1db6ced1f514898c65fab935f",
-    "source_manifest": "87b1e124969ec11b836740c027d28dc2a9d319d279ed55be1d9fc1523390f634",
+    "source_manifest": "10c05f0fd057f75536b47e3a0160ffb2b47cdec094b6694ebd62cc6bef50d9e8",
     "take_up": "9522ce40f7dea569312dd7a7beb474e5a5afadd2f1e10cb5876534c3ef623d35",
     "tail": "256947cd7c1bbd85f5d1fa3911a07a170ae87830f098110a40ed25d93f78347c",
 }