Skip to content

feat: add leaderless signaling and parallel full-stack deployment - #3

Merged
h4x3rotab merged 5 commits into
mainfrom
leaderless-signaling
Oct 26, 2026
Merged

h4x3rotab merged 5 commits into
mainfrom
leaderless-signaling

Conversation

@h4x3rotab

@h4x3rotab h4x3rotab commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Replace the external signaling broker with authenticated, leaderless snapshot signaling hosted by every coordinator CVM. No separate Vultr signaling server or leader election is required.
  • Use direct ICE/QUIC connections only; remove TURN/relay support. STUN is configurable and defaults to Cloudflare's public STUN service.
  • Bind answers to exact offers, recover publication clocks after restart, isolate candidate replay by attempt, and defer ICE gathering until both peers' credentials are available.
  • Provision independent coordinator/workload CVMs concurrently through one ordinary Terraform apply. Allocate coordinator identities, derive broker URLs, then configure coordinator values through provider-owned phala_cvm_env resources. Remove the shell driver, targeted phases, and generated roster inputs. Parent apps no longer overwrite instance-owned slot environments.
  • Wrap coordinator resources in repo-local modules/mesh-coordinators: callers put shared values in env and per-instance values in instances[name].env; the module derives the signaling roster without an input/output dependency cycle. Include state moves for the former flat resources and preserve legacy app-only placement.
  • Update examples, pinned images, operator docs, changelog, and implementation regression tests.

Dependencies / merge order

The examples require the provider's new phala_cvm_env, allowlist-only env_keys, env_update_scope, wait_for_identity, and readiness semantics. Those are not available in published 0.3.0-beta.4, which remains the current version constraint. Validation used a locally built provider via Terraform development overrides, as documented. A provider release and example version-constraint update are needed before registry-only deployment is supported.

The pinned verifier image includes the linked verifier fix. No phala-cloud-monorepo source changes are included.

Prior live validation: prod14 + prod15

Fresh eight-CVM deployment: three coordinators, three PostgreSQL nodes, and two webdemo nodes, split evenly across the hosts.

  • The previous staged deployment driver completed in 2m53s, versus 10m39s for the serial baseline driver. This is not a benchmark of the new single-apply resource flow.
  • Both web gateways were healthy by 3m24s; both PostgreSQL replicas were streaming by 3m29s. These application-readiness timings are separate from driver completion.
  • All 28 direct links, including 16 cross-host links, connected. Across 56 directed first attempts: zero ICE failures and maximum checking-to-connected time of 3 seconds.
  • 128 /all requests exercised 1,024 inner calls with zero errors.
  • Eight v3 manifests matched their compose hashes and expected KMS/image identities; attestation admission remained enabled.
  • Final Terraform plan exited 0 with no changes. PostgreSQL replication was checked through runtime state/logs; no independent SQL write/read client test was performed.

Identity allocation remains a genuine dependency, now expressed in Terraform's graph; independent boots and updates overlap. Cloud create/reboot time remains and no global latency bound is claimed.

Native single-apply validation

  • Both examples validate with the new local provider and generate acyclic Terraform graphs. Workload allocation does not depend on coordinator environment-update readiness.
  • The provider's Terraform Core integration test uses a local mock Cloud API: one apply allocates three identities, then a three-request barrier proves concurrent environment updates. A sensitive-value rotation updates in place; both no-op plans exit 0.
  • A read-only migration plan against a copy of the live eight-CVM state verifies the module state moves, three new env bindings, and in-place ownership changes: no deletions, workload changes, slot identity changes, compose measurement changes, or allowlist changes. No native-path production migration or fresh deployment has been applied yet.
  • Eleven Terraform mock-provider module tests pass, covering env merging, per-slot targeting, generated rosters, measured allowlists, explicit bootstrap selection, placement compatibility, invalid inputs, and unknown sensitive values. They do not create Cloud resources.
  • Existing staged deployments must remove their generated signaling.auto.tfvars.json and legacy driver inputs after backing up state. Keep the supplied moved blocks, preserve names and placement, and initialize the local module. The guide documents ownership handover and readiness versus application-health checks.

Automated checks and PR split

Local credentials, Terraform state, generated deployment files, and temporary overrides are excluded.

Signed-off-by: Hang Yin <h4x3rotab@gmail.com>
Signed-off-by: Hang Yin <h4x3rotab@gmail.com>
@h4x3rotab
h4x3rotab merged commit 62a1553 into main Oct 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant