Amalgam

Amalgam is a biomedical metaontology: a cross-resource integration layer of entity identifiers, names, and typed cross-references organized by Biolink class. It integrates source ontologies rather than replacing their semantics; typed associations are not equivalence assertions.

The Spark/Scala pipeline turns BABEL outputs (plus your own ontologies and hand-curated manual entries) into a Hugging Face dataset with cliques and crossrefs subsets, with one hard invariant:

Within a given class x taxon scope, a synonym belongs to exactly one CURIE.

The dataset ships original strings with junk filtered out. Normalization (stemming, level keys) is deliberately NOT done here — downstream consumers (e.g. Tablassert’s fullmap DB) own the real NLP. This codebase contains no stemmer.

Source code: SkyeAv/Amalgam (public GitHub repository). The Scala package and Mill module remain biolinkamalgam; existing build commands are unchanged.

Latest published build: SkyeAv/amalgam (public HF dataset; snapshot 2026jul22, 138,885,412 cliques). Publish a fresh run with make publish.

Output

cliques/<class>.parquet, one row per surviving clique leader, one file/split per Biolink class. Split names use snake_case without biolink: (Gene becomes gene, SmallMolecule becomes small_molecule). All public column headers are uppercase:

Column	Type	Meaning
CURIE	string	clique leader identifier
PREFERRED_NAME	string	the clique’s preferred name, as written
CLASS	string	Biolink category, e.g. biolink:Gene
SYNONYMS	array<string>	synonyms as originally written, sorted
TAXA	array<string>	the clique’s NCBITaxon CURIEs, sorted; empty when taxonless

Taxa and the top-taxa cutoff: config/taxa.yaml taxon_allowlist holds the same top-100 experimental taxa as Tablassert’s fullmap. A non-biolink:OrganismTaxon clique that carries taxa ships only if at least one is in the list. Taxonless cliques ship (fullmap parity), but first config/curie_taxa.yaml gives taxonless cliques the taxon of single-species identifier namespaces (MGI, RGD, ZFIN, FB, WormBase, SGD, HGNC and the Ensembl stable-id species prefixes in config/curie_taxa.yaml), so e.g. pig ENSSSC genes no longer bypass the cutoff. Inferred taxa also move such cliques out of the taxonless merge/arbitration scope into their species scope, so e.g. human Ensembl-only genes arbitrate against BABEL-tagged human cliques (the namespace veto still guards merges). validate checks the cutoff on the shipped parquet.

Beside the data: run_manifest.json (local stage metrics), LICENSE.md (upstream data terms), and README.md — a generated Hugging Face dataset card with frontmatter, schema docs, filter summary, and run stats. Pass --audit to also emit conflicts.tsv (arbitration audit) and merges.tsv (clique merge audit); they are impractical at BABEL scale, so they default off.

CURIE-like or missing preferred names use the first sorted, gate-admitted non-CURIE synonym when available. Existing human-readable preferred names are preserved. This runs before merging and arbitration, after the ingest cache.

Build and test

make setup   # resolve deps, compile
make test    # offline test suite (fixtures only, no network)
make lint    # scalafix --check + scalafmt --check
make fix     # apply scalafix rewrites + reformat
make check   # lint + test (the quality gate)

Heavy runs (full BABEL snapshot) belong on a big box; the wenceslaus profile in config/profiles.yaml is sized for 80 cores / 187GB.

Run

# Download a whole BABEL snapshot at once (bulk aria2c, resumable), then build:
./mill biolinkamalgam.run run --download 2026jul22 --profile wenceslaus \
  --out out/run1 --snapshot-dir /data/snapshots/2026jul22

# Against an already-downloaded snapshot:
./mill biolinkamalgam.run run --babel /path/to/snapshot --out out/run1

# Fast iteration: cap raw rows read per source (smoke-test config changes in seconds):
./mill biolinkamalgam.run run --babel /path/to/snapshot --sample 200000 --out out/sample1

# Benchmark the download engines against one URL (evidence for download.yaml):
./mill biolinkamalgam.run benchDownload --url <big-file-url> --out /tmp/dlbench

The downloader is pluggable in config/download.yaml (engine: aria2 | jdk-chunked | jdk), one bulk invocation for the whole snapshot, resume/skip via HEAD size checks. Split compendium parts (Gene.txt.00…​) are preferred over their whole-file twins automatically (never both — no double ingest).

Configuration (all YAML, all commented)

File	Decides
config/taxa.yaml	major-taxon bucketing + the top-100 taxon_allowlist (fullmap parity; empty disables)
config/filters.yaml	category drops (biolink:Publication, biolink:Agent), per-category CURIE prefix drops, banned substrings (language, uncharacterized, hypothetical), dead-term stopwords, numeric-junk regex (incl. decimals like 0.5), CURIE regex drops
config/promote_prefixes.yaml	identifier promotion: quoted '*' indexes every leader/equivalent CURIE, matching fullmap
config/download.yaml	download engine + tuning, snapshot base URL, file exclusions (Publication parts skipped)
config/profiles.yaml	named Spark profiles (cores/memory/master/shuffle), cluster targeting as pure YAML

Adding your own data

Drop files into a directory and pass --custom <dir>:

• .tsv — curie\tlabel\tsynonyms\tcategory\ttaxa (pipe-separated lists)

• .jsonl — rows matching the custom schema (curie, label, synonyms, category, taxa)

• .ndjson — manual entries (BABEL PR #887 shape): curie, type, preferred, alternatives, optional equivalents (multi-CURIE cliques; curie stays the leader), optional taxa. Priority 0: manual entries override — they win preferred names and arbitration tiebreaks against every other source.

{"curie": "EUPATH:0009259", "type": "biolink:ClinicalFinding", "preferred": "Shannon-indexed alpha diversity data", "alternatives": ["Shannon alpha diversity", "Alpha diversity"]}

For anything more complex, subclass SynonymSource and add one line to Registry.forConfig.

Additional sources and typed crossrefs

--clinvar <variant_summary.txt.gz> adds NCBI ClinVar variations and the rsIDs referenced by those variations. It does not claim exhaustive dbSNP coverage. Each CLINVAR: variation and each DBSNP:rs…​ locus stays a separate biolink:SequenceVariant clique: one rsID can reference several alleles. Gene and disease annotations never become variant synonyms.

scripts/fetch-obo.sh <dir> downloads the reviewed config/obo_sources.yaml list. Pass --obo <dir> [--obo-config <yaml>] to ingest it: MP, ZP, FYPO, WBPhenotype, APO, TO, MAXO, HsapDv, MmusDv, ZFS, FBdv, WBls, MOD, TAXRANK, and FAO. The list records the Biolink class, taxa, PURL and OBO Foundry license. Only active own-prefix terms, names, EXACT synonyms and alt_ids enter cliques; BROAD/NARROW/RELATED synonyms and imported foreign-prefix terms are excluded. Malformed upstream alt_ids are logged and skipped (for example APO’s alt_id: curators), not inserted as names or identifiers. Valid terms remain available. The config comments document gaps, including mixed-category ontologies and resources without a current OBO export. This is not complete Biolink coverage.

When either source is enabled, typed edges are written beside the main parquet, one file per source: crossrefs/<source>.parquet, columns SUBJECT, PREDICATE, OBJECT. Source is encoded in the split name, not a column: clinvar, obo_mp, etc. Predicates remain typed associations, not equivalence:

• dbsnp_reference: ClinVar variation to dbSNP locus (many-to-many).

• gene_association: ClinVar variation to NCBIGene.

• xref: OBO cross-reference (relevance only, not equivalence).

• replaced_by: obsolete OBO term to its successor.

Edges retain source CURIEs, not post-merge leaders; an obsolete term may appear only in crossrefs/. No crossref is used as a synonym or merge key in the main parquet. There are exactly two HF subsets: cliques (default) and crossrefs. If no source contributes edges, crossrefs has a schema-bearing zero-row empty split. Use streaming=True for that split; datasets rejects zero-row splits in materialized mode:

load_dataset("SkyeAv/amalgam", "cliques", split="gene")
load_dataset("SkyeAv/amalgam", "crossrefs", split="clinvar")

publish-hf.sh validates the card and uploads only its exact declared files, README.md and LICENSE.md. The same commit removes obsolete clique/crossref parquet files and the old single-file layout. Audit files, local metrics and interrupted-write debris stay local. Existing runs must be regenerated. Run upload-contract tests offline with uv run --with pyyaml python -m unittest discover -s scripts -p 'test_publish_hf.py'.

Amalgam code and original compilation rights (selection and arrangement, to the extent protectable) are licensed under PolyForm Noncommercial 1.0.0. See LICENSE for the unmodified license and LICENSING.md for scope. Upstream data is not relicensed; independently available upstream rights remain unchanged. The generated LICENSE.md preserves upstream terms, including attribution/share-alike and restricted-use resources. Previously distributed Apache-2.0 versions retain their existing grants. A license notice grants no redistribution rights; confirm authorization before public publication. Per-synonym provenance is not exported, so dropping restricted-prefix leaders alone cannot establish compliance.

make run ARGS="crossrefs --out <run> --top 200 --profile wenceslaus" writes crossref-report/summary.tsv (edge and distinct-node counts by predicate, source and namespace pair) and degree.tsv (bounded inbound/outbound high-degree nodes per predicate). Degree counts distinct neighbors, not independent evidence or error probabilities. Predicates never mix; no components are merged. This first report does not compute statistical confidence, source independence or precision.

The existing BABEL/custom/GTDB ingest cache stays reusable. Additional sources are loaded as a small fresh layer each run. A prefix scan rejects overlapping core leaders rather than silently creating duplicate CURIEs. Use a new output directory for each run; sidecars from old runs are not a source of truth.

Merge execution and benchmarking

Merge computation materializes narrow contested rows, mergeable rows and the final loser-to-winner mapping as parquet before reusing them. Wide name payloads are fetched only after the mapping is settled. The merged table is also materialized before arbitration. Temporary stages live beside the output and are removed on normal completion or a caught failure; a killed JVM can leave .amalgam-stages-* directories that require manual cleanup after confirming the run has stopped. This bounds recomputation, not total input size. Provision scratch for shuffles and materialized data and measure representative samples before full builds.

make run ARGS="benchMerge --input <canonical-clique.parquet> --out <new-dir> --master local[8]" exercises the same merge boundaries. It prints input/output/absorbed counts and elapsed milliseconds; the output retains stages for inspection. Sample whole preferred-name groups, not isolated rows, to preserve candidate density.

Conservative preferred-name merging

Equal labels do not prove identity. biolink:SequenceVariant cliques never merge by preferred name. Other preferred-name groups stay separate when members use the same identifier prefix but share no identifier under it (for example, two PubChem CIDs with the label "Cobalt;erbium"). An ambiguous group does not choose an arbitrary target for a third clique. Distinct CURIEs still arbitrate shared names within class and taxon scope, and rows with empty synonyms survive.

Synonym diagnostics (report only)

make run ARGS="scrub --out <finished-run> --profile wenceslaus" writes <finished-run>/scrub-report/ counts and evidence samples. It never modifies or writes a replacement dataset. A diagnostic row is a signal, not a proven error; valid abbreviations, translations, chemical names and cross-taxon names can look like outliers.

config/synonym_scrub.yaml controls the bounds. Cohesion scores only cliques between min_clique_members and max_clique_members names. Larger cliques are skipped, not certified clean. Preferred names and CURIE-shaped aliases (including INCHIKEY: and GTDB:) are protected from all signals. The cross signal checks only cohesion outliers against exact stripped preferred-name keys, skipping keys with more than cross_key_max_targets targets. It does not search fuzzy near matches. This intentionally trades recall for bounded work; the previous global MinHash join exhausted scratch space on the full snapshot.

Hub counts are same-class recurrence among non-preferred, non-identifier aliases, not taxon-scoped evidence of an error. Signal counts overlap and are not unique synonym counts. No flags are applied automatically.

Rules that will not change

• No conflation, ever: synonyms-conflated/ and *Conflated* files are never read.

• Taxon bucketing: only major_taxa get their own scope; the long tail shares OTHER_TAXA, taxonless cliques share NO_TAXON. The same name MAY recur across categories or taxa.

• Winner order (merge and arbitration): preferred-name match, most synonyms, identifier count, source priority (lower wins; manual/custom 0, BABEL 10), smallest CURIE. Total order — reruns reproduce.

• Tests are offline by construction: FixtureSnapshotProvider throws on non-local paths.

Full dbSNP and biomedical Wikidata bulk work

The Ralph bulk-source implementation is in progress. The current published/run coverage remains unchanged until full-source ingestion and validation finish.

config/dbsnp_bulk.yaml pins all 30 RefSNP partitions in NCBI human dbSNP b157: 25 chromosome partitions plus other, nosnppos, unsupported, merged and withdrawn. The sample rs268 file is excluded because it duplicates real data. The inventory totals 485,701,336,617 compressed bytes. A GRCh38 VCF alone is not this scope. Historical nonhuman dbSNP/EVA releases are not part of the human pin.

config/wikidata_bulk.yaml pins the official 2026-10-19 complete entity JSON bzip2 dump (103,488,697,311 bytes). Biomedical inclusion rules will be explicit and versioned; this pin is the input scan, not an assertion that every item is medical. Wikidata structured data are CC0; Wikipedia article text is not imported.

The inventory implementation validates HTTPS release pins, exact sizes and MD5 checksums. Projection receipts require committed parquet parts and match the entire manifest fingerprint and a required projection fingerprint (code version and scope config). A receipt permits releasing a processed source shard while retaining resumability. Receipts track projected file names/sizes, not cryptographic hashes of projected data; they do not replace final dataset validation. No large bulk download is started by tests.

RefSNP projection

src/biolinkamalgam/source/DbSnpSource.scala projects every lifecycle partition into one compact identity row per rsID plus typed association edges:

• One status per record from exactly one snapshot block: active (primary), merged, withdrawn, unsupported, locationless (nosnppos). Contradictory or missing lifecycle fails validation.

• Output keeps only identity names: CURIE DBSNP:rs<id>, preferred rs<id>. No allele, gene symbol or placement string ever becomes a locus alias. Live partitions (active/unsupported/locationless) emit rows; merged and withdrawn remain typed history, not synonym rows.

• Typed crossref edges: gene_association to NCBIGene ids, cites to PMIDs, merged_into to DBSNP:rs ids. Duplicated ids collapse per record.

• Partial Spark schema reads only identifier/lifecycle fields; the huge placements/frequencies payloads are never materialized. Splittable bzip2 keeps shard reads parallel without staging decompressed JSON.

• Fixtures in test/resources/dbsnp/ are reduced-column records from the official b157 sample and first records of each exceptional partition; identifiers and lifecycle fields are verbatim, other payloads dropped.