feat: add leaderless signaling and parallel full-stack deployment

.dockerignore

@@ -4,8 +4,11 @@
 snapshots
 
 clusters/*/.terraform
+modules/**/.terraform
 clusters/*/terraform.tfstate*
 clusters/*/terraform.tfvars
+clusters/*/signaling.auto.tfvars.json
+clusters/*/.signaling-roster.*
 clusters/*/*.backup
 clusters/*/*.bak
 

.gitignore

@@ -18,8 +18,11 @@ workloads/webdemo/webdemo
 
 # Terraform local state. Keep .terraform.lock.hcl tracked.
 clusters/*/.terraform/
+modules/**/.terraform/
 clusters/*/terraform.tfstate*
 clusters/*/terraform.tfvars
+clusters/*/signaling.auto.tfvars.json
+clusters/*/.signaling-roster.*
 clusters/*/*.backup
 clusters/*/*.bak
 

AGENTS.md

@@ -0,0 +1,19 @@
+# Repository Instructions
+
+- Avoid degradation handling, fallbacks, hacks, heuristics, local
+  stabilizations, or post-processing bandages that are not faithful
+  general algorithms.
+- When output is structured with multiple levels of bullet points, use
+  multiple levels when the content needs them. The single-level rule
+  only applies when the content can be structured in one level.
+
+## Changelog
+
+- Maintain `CHANGELOG.md`.
+- Add an `Unreleased` entry for every merged change that affects user
+  behavior, deployment flow, trust or security model, public
+  configuration, examples, or user-facing documentation.
+- Keep entries short and concrete. State what changed and why it
+  matters to an operator or developer.
+- When cutting a release, move relevant `Unreleased` entries under a
+  dated version heading and leave a new empty `Unreleased` section.

CHANGELOG.md

@@ -0,0 +1,124 @@
+# Changelog
+
+This changelog tracks `service-mesh` releases.
+
+## Unreleased
+
+### Changed
+
+- Add `modules/mesh-coordinators` so examples keep env values beside named
+  instances while the module handles broker discovery and separate value
+  resources. Include state moves from the existing flat coordinator resources.
+
+- Replace the staged deployment shell driver with `phala_cvm_env` resources.
+  Allocate coordinator identities, derive broker URLs, and configure values
+  in one ordinary apply, without targeted phases or a generated roster file.
+  The app owns measured env keys; each environment binding owns only values.
+
+- Run independent deployment operations concurrently using Terraform's
+  parallelism (10 by default), without a workload boot barrier behind
+  coordinator environment updates. Document per-slot environment ownership.
+- Pin the verifier with coalesced AMD endorsement fetches and HTTP 503 for
+  collateral outages, so parallel attestation does not misclassify a rate
+  limit as an invalid quote and restart otherwise healthy workloads.
+- Pin both examples to the tested leaderless sidecar image and pin the
+  Postgres demo workloads, replacing stale or floating image references.
+
+- Deferred ICE candidate gathering until matching peer credentials arrive,
+  preventing idle NAT mappings from expiring while a peer is still booting.
+- Moved signaling onto every coordinator CVM with leaderless, authenticated
+  snapshot replication; broker loss no longer requires an external server
+  or leader election. Derive endpoint wiring from allocated slot identities.
+- Made public STUN configurable (Cloudflare by default) and removed TURN
+  relay support from the mesh transport so application traffic stays direct.
+- Bound ICE answers to explicit offer IDs, replacing timing-based retry
+  supersession and preventing duplicate/stale broker deliveries from
+  restarting healthy links.
+- Switched admission evidence generation from TDX-only `GetQuote` to
+  the platform-neutral `Attest` API so admission can support SEV-SNP.
+- Added optional `node_id` placement for pinning every HA Postgres demo
+  CVM to a specific Phala Cloud node.
+- Reworked README flow around demo, concepts, verification, and starter.
+- Moved image references into compose files. Terraform no longer
+  replaces image placeholders.
+- Added Terraform checks that reject `${...}` in compose image values.
+- Replaced workload `secrets` config with `compose_env`.
+- Clarified the remaining workload-aware rollout gap.
+
+### Security
+
+- Patroni passwords now come from in-TEE derivation, not Terraform
+  state.
+- Validated the experimental KMS key-provider setting on AMD SEV-SNP so
+  direct app deployments bind the KMS public-key identity into MRConfig v3.
+
+### Validation
+
+- Verified the full eight-CVM Postgres example across prod14 and prod15,
+  including direct cross-host links, workload admission, streaming replicas,
+  and MRConfig v3/KMS identity preservation on bootstrap and named CVMs.
+
+### Documentation
+
+- Added this changelog and changelog maintenance instructions.
+- Added the starter architecture SVG and updated the HA Postgres SVG.
+- Updated attestation and publishing docs for deterministic compose
+  images.
+
+## service-mesh v0.1.1, 2026-06-04
+
+### Security
+
+- Consul ACL management token now comes from in-TEE derivation, not
+  Terraform state.
+- The token is no longer a Terraform input or deploy-host secret.
+
+### Validation
+
+- Verified on real Phala Cloud with the 8-CVM HA Postgres demo.
+- Verified admission, mesh traffic, Patroni HA, and no management token
+  in Terraform state.
+
+### Images
+
+- Published GHCR images for `mesh-sidecar`, `webdemo`, `patroni`, and
+  `signaling`.
+- Tags: `latest`, `sha-532df17b61e90f79d17f4d3b9d98cf8b90b80917`.
+
+### Known Limits
+
+- Use `terraform apply -parallelism=1` until the upstream KMS nonce race
+  is fixed.
+- Patroni passwords and the cross-role gossip key still came from
+  Terraform state at this release.
+
+## service-mesh v0.1.0, 2026-06-01
+
+### Added
+
+- First tagged release of the dstack service mesh example.
+- Added the HA Postgres demo.
+- Added the `clusters/starter` API and worker demo.
+- Reorganized the repo into `components`, `workloads`, `clusters`,
+  `support`, and `docs`.
+
+### Changed
+
+- Pinned the Phala Terraform provider to `0.3.0-beta.4`.
+- Fixed unintended instance replacement for env and compose updates.
+
+### Validation
+
+- Verified end to end on real Phala Cloud with the 8-CVM HA Postgres
+  demo.
+
+### Images
+
+- Published attested GHCR images for `mesh-sidecar`, `webdemo`,
+  `patroni`, and `signaling`.
+- Tags: `latest`, `sha-daf8dce4d2e8e9e47960df039114fcc330ef07bf`.
+
+### Known Limits
+
+- Use `terraform apply -parallelism=1` until the upstream KMS nonce race
+  is fixed.

README.md

@@ -95,8 +95,8 @@ service-to-service paths.
    ```
 
 3. Edit `terraform.tfvars` if you need a different cluster name,
-   replica count, bootstrap endpoint, KMS backend, or attestation
-   setting.
+   replica count, STUN provider, KMS backend, attestation setting,
+   or an explicit Phala Cloud `node_id` for pinned placement.
 
 4. Set your Phala Cloud API key.
 
@@ -107,13 +107,24 @@ service-to-service paths.
 5. Deploy the cluster.
 
    ```bash
-   terraform init -upgrade
-   terraform apply -parallelism=1
+   terraform init
+   terraform apply
    ```
 
 > [!IMPORTANT]
-> Use `-parallelism=1`. This serializes app creation for the current
-> Phala provider app and slot creation path.
+> The local `mesh-coordinators` module keeps env values beside each named
+> instance and handles identity allocation, broker URLs, and configuration.
+> Independent boots and updates overlap in one apply, including after
+> scaling or replacement.
+> No shell driver or generated roster is needed. See
+> [leaderless signaling](docs/signaling.md).
+>
+> The examples pin the tested leaderless sidecar image. After changing its
+> code, build/publish the image and update every compose digest. Old images
+> do not implement the new broker endpoint. Check broker compatibility and
+> application health after applying. The examples require an unreleased
+> provider with `phala_cvm_env`, allowlist-only `env_keys`, `wait_for_identity`,
+> and `env_update_scope`. Select a local build until a release is available.
 
 ## Verify The HA Postgres Demo
 
@@ -299,8 +310,8 @@ group. It has no database and no Patroni role tags.
 4. Deploy the starter cluster.
 
    ```bash
-   terraform init -upgrade
-   terraform apply -parallelism=1
+   terraform init
+   terraform apply
    ```
 
 5. Test the public API and private worker route.
@@ -566,6 +577,8 @@ service design.
 - `clusters/patroni-demo`: full demo cluster, including Terraform and
   compose files.
 - `clusters/starter`: minimal API + worker cluster template.
+- `modules/mesh-coordinators`: named coordinator groups with inline env inputs
+  and internally derived broker rosters.
 - `components`: reusable platform components built into the sidecar or
   support images.
 - `workloads`: demo workload images used by the HA Postgres demo.
@@ -574,6 +587,14 @@ service design.
 
 ## Development
 
+With the unreleased local provider selected, run the coordinator module's
+mock-provider tests (no Cloud resources are created):
+
+```bash
+terraform -chdir=modules/mesh-coordinators get
+terraform -chdir=modules/mesh-coordinators test
+```
+
 Validate Terraform:
 
 ```bash
@@ -608,13 +629,17 @@ docker build -t service-mesh-mesh-sidecar-check -f components/mesh-sidecar/Docke
 
 ## Current Limits
 
-- The public connection-bootstrap endpoint is for demos and development.
-  Production deployments should use a managed or tenant-owned endpoint.
+- Signaling runs on the coordinator CVMs with no elected leader or external
+  server. Peers need one commonly reachable broker for new connections.
+  Public STUN discovers addresses; networks that prevent direct ICE
+  connections fail explicitly rather than relaying.
 - Terraform can add or remove replicas. Measured compose or image
   updates still need a workload-aware rollout procedure with health
   checks, drain order, and admission policy handling.
-- Run one Terraform apply at a time with `-parallelism=1` for the
-  current Phala provider app and slot creation path.
+- Run one Terraform apply at a time per state file. Terraform defaults to
+  parallelism 10; use `terraform apply -parallelism=1` for older cloud
+  backends without the KMS nonce-allocation fix. See the
+  [parallel bootstrap requirements](docs/signaling.md#parallel-bootstrap).
 - The cluster-wide service-layer gossip key is still generated by
   Terraform and stored in the Terraform state file. The planned
   replacement is secret generation and distribution inside attested
@@ -624,6 +649,8 @@ docker build -t service-mesh-mesh-sidecar-check -f components/mesh-sidecar/Docke
 
 ## More Docs
 
+- [`CHANGELOG.md`](CHANGELOG.md): release history and unreleased
+  changes.
 - [`docs/attestation.md`](docs/attestation.md): trust flow and evidence
   checks.
 - [`docs/architecture.md`](docs/architecture.md): lower-level network