[AdaAibaby]

Problem

Deleting a template soft-deletes the DB row but left the build artifacts (rootfs, kernel, FC snapshots) in GCS/S3 forever. GetExclusiveBuildsForTemplateDeletion existed with zero callers. Fixes #3239.

Why DB-exclusive is not enough (this is the important part)

A template build stores only the bytes it changed vs its ancestors; the full image is reconstructed by following the header BuildMap into parent/grandparent builds. Those intermediate layers are never written to the DB (a cache hit reuses prior layers under a new build ID). So a build that is DB-exclusive to template T can still be the header-mapping ancestor of another live template's build. Deleting its {buildID}/ prefix orphans that ancestor — the DB looks consistent, the other sandbox even starts, and it only surfaces as a silent, unrecoverable Input/output error when a missing block is read. No versioning ⇒ permanent.

What this PR does

Reachability guard (folded into processExclusiveBuilds): a DB-exclusive candidate is deleted only if it is not reachable from any live root's memfile/rootfs header closure. Fails closed (deletes nothing) if the ancestor reader is unconfigured, the live-root set can't be queried, or any live root's header can't be read — strictly no worse than leaving artifacts in place.

Live-root set completion: GetLiveBuildRootsForTeamExcludingTemplate now UNIONs the team's in-progress/pending builds (status_group) on top of other active templates' assigned builds — closing the gap where a concurrent build reusing the doomed template's exclusive ancestor has no env_build_assignments row yet.

Reconciliation metrics so orphaned storage is derivable, not guessed:
candidates == deleted + delete_failed + skipped_nil_node + guard_kept (success), or fail_closed > 0 when nothing was deleted. fail_closed carries a low-cardinality reason (reader_unconfigured|live_roots_query|header_unreadable) and is the alert signal for "storage not being reclaimed". deleted/failed carry a cluster attribute. team/build IDs stay on logs/trace, not metric attributes (cardinality).

Background hardening kept: panic-safe, deadline-bounded (2m), bounded fan-out (8), best-effort; storage errors never affect the already-committed 204.

Supersedes #3740

#3740 had the reachability guard but on the pre-hardening base (raw goroutine, no recover/timeout/limit, no metrics). This PR unifies both: guard and hardening and reconciliation metrics. #3740 is closed in favor of this.

Tests

processExclusiveBuilds guard/skip/fail paths, fail-closed classification, and DB-backed live-root tests (in-progress-without-assignment is a root, soft-deleted/ready-unassigned excluded, team-scoped). go build/vet/gofmt clean; unit tests green; DB tests green against a real migrated Postgres.

---

Update — checkpoint corruption guard + online grace window + pluggable live-root sources

Three follow-up commits hardening the guard, all driven by the checkpoint/snapshot corruption vector and the live-root/TOCTOU gaps around it.

1) Snapshot/checkpoint live-root source (corruption fix). A snapshot (paused sandbox / checkpoint) is a COW diff layer on top of a base template, so its header mapping references the base template's builds — but that edge lives only in object-storage headers, never in env_build_assignments, so the base build looks DB-exclusive and deletable. GetLiveBuildRootsForTeamExcludingTemplate now UNIONs a third source: the team's snapshot builds, joined via the snapshots table and deliberately not gated on active_envs. This keeps a base build in the keep set even after the snapshot's own env is soft-deleted but its diff layer still lives in storage — exactly the window ExistsTemplateSnapshots (which joins active_envs) can let a base deletion slip through.

2) Online deletion grace window (narrows TOCTOU). A DB-exclusive candidate younger than TemplateArtifactMinAgeSeconds (feature flag, fallback 24h; <=0 disables) is not deleted online; it is counted skipped_too_new and left to the offline sweeper. This narrows the window between the live-root query and DeleteBuild, where a fresh build may have just reused the candidate as a cache-hit ancestor before its protecting row/header landed. The candidate query now selects created_at.

Reconciliation identity extended:

candidates == deleted + delete_failed + skipped_nil_node + guard_kept + skipped_too_new   (success)
candidates with fail_closed > 0                                                            (nothing deleted)

New meter api.template.artifacts.skipped_too_new.

3) Pluggable live-root sources, fail-closed on any source error. The keep-set build is refactored behind a liveRootSource interface (dbLiveRootSource is the sole production impl today; orchestrator/Redis-pinned roots can be added later). collectLiveRoots unions + dedups all sources and fails closed on the FIRST source error (classified live_roots_query), so a partial root set can never expand into an under-protected keep set.

Tests

DB-backed: snapshot build is a live root when its base template is deleted, and stays one after the snapshot's env is soft-deleted (the checkpoint guard); another team's snapshot is never a root. Unit: grace window defers a too-new candidate and deletes an old one; collectLiveRoots unions/dedups and fails closed on any source error.

Note: the handlers-package unit tests require #3800 (an unrelated pre-existing rls.proto double-registration panic) to run in CI. The DB-backed tests here pass independently.

/cc @ryan-sana Looking forward to your code review.