Swartzn/feat/cli managed remote storage targets by swartzn · Pull Request #396 · ThinkParQ/beegfs-go

swartzn · 2026-11-03T18:16:58Z

What does this PR do / why do we need it?

Remote storage targets (RSTs) could only be defined in the beegfs-remote TOML file. Changing one
needed a restart of Remote and every sync node. Targets that shared a store had to repeat its
endpoint and credentials. Buckets had to be made with another tool, on a host with a route to the
store and a copy of its keys.

This PR adds:

beegfs remote target create|modify|delete|list. Remote stores the definitions in a new
BadgerDB (job.rst-db) and applies each change live, to itself and to every sync node. On first
start it imports the TOML targets. After that the database is the only source.
Backends, managed with beegfs remote backend create|modify|delete|list|verify. A backend is
the named connection to one remote store, such as an S3 endpoint and its credentials. A target
of the same type names it with --backend and inherits every setting it leaves unset. A change
to a backend, such as a credential rotation, reaches all its targets live.
beegfs remote bucket list|create|modify|delete, with the alias container (second
commit). Remote runs each request against the store with a backend's settings. So the CLI host
needs no route to the store and no keys.
rst.TargetType. Each RST type describes its own configuration, including which settings a
backend must and must not set. The CLI, Remote and rst.New use it instead of switching on the
type. A new RST type needs no CLI or Remote code.
One shared HTTP client for all S3 providers (third commit). Live changes rebuild providers,
and each rebuild used to drop the warm connections to the endpoint.

Behaviour:

Every target passes one check, however it arrives. Its type must be supported (Mock is refused),
it must have every required setting, and a provider must build from it. The name is optional.
modify changes only the flags given, and is refused if the target changed since it was read.
--backend= detaches a target and keeps what it inherited as its own values.
Moving a target's objects (endpoint, region, bucket, partition) while unfinished jobs use it
needs --force. This includes a backend change that moves its targets.
delete needs --force. Unfinished jobs are cancelled and undone as Remote next meets them. A
deleted ID is reused only with create --id=<id> --force.
A backend cannot be deleted while a target names it.
backend create|modify --verify checks a new definition from Remote before storing it.
backend verify checks a stored backend from Remote and from every sync node.
bucket delete refuses a bucket that a live target uses, even with --force. Targets of every
S3-based type count. Two endpoints that differ only in host case, a default port, a trailing
slash or a virtual-hosted bucket count as the same store. A populated bucket needs --force,
which empties it first.
bucket create|modify succeed when the change succeeded, even if Remote cannot read the bucket
back afterwards. Details Remote cannot read show as -, and Remote logs why.
A sync node that rejects a pushed set goes offline and starts no work until it accepts one.
remote target list --deleted shows deleted targets. remote job cancel gains --message.
remote list keeps the output of earlier releases, because scripts parse it.
remote target list has the new layout: one column per setting, and what each target inherits.

Compatibility:

Sync nodes must be upgraded before Remote. Remote requires rst-live-update from every
node. The CLI needs rst-management from Remote to change targets, and says so against an older
Remote. remote list still works there.
Every S3 and xtreemstore target must set endpoint-url. Before, each node could pick its own
endpoint from its environment or the region, so nodes could disagree. A TOML file that relied on
this stops Remote at the upgrade, naming the target. The shipped TOML explains the fix.
job.rst-db defaults to rst.badger next to job.path-db. The shipped TOML documents rollback.

Related Issue(s)

Protobuf: feat(remote): add RPCs to manage remote storage targets protobuf#82 (pinned at e5223df)
Builds on Swartzn/fix/remote node connection handler #333 (remote node connection handler), which live updates rely on.

Where should the reviewer(s) start reviewing this?

common/rst/targettype.go, then s3type.go and xtreemstoretype.go.
common/rst/backend.go: resolving a target from its backend, and what a backend must set.
common/rst/store.go: ValidateTarget, and ClientStore.UpdateConfig applying a diff.
rst/remote/internal/job/rstconfig.go: create, update, delete and verify for targets and
backends.
rst/remote/internal/rstdb/: the database and the TOML import.
rst/remote/internal/workermgr/rsts.go: applying a set and pushing it to sync nodes.
rst/remote/internal/job/rstlookup.go: jobs of a deleted target.
rst/sync/internal/workmgr/: the pause after a rejected set.
common/rst/bucket.go and s3bucket.go, then rst/remote/internal/job/rstbucket.go: bucket
management and the delete rules.
ctl/internal/cmd/rst/, ctl/pkg/ctl/rst/target.go and bucket.go: the CLI.
common/rst/s3.go: the shared HTTP client.

Are there any specific topics we should discuss before merging?

New work can reach a sync node before the change does. Remote applies a change to itself
first, then pushes it to each node, usually within milliseconds. A job in that window can run
with the old set. A new target fails as unknown. After a bucket change, the object may land in
the old bucket, as if the job ran just before the change. No data is lost. A per-node gate in
Remote would close the window. Is it worth it now?
The Remote and sync gRPC servers are not authenticated. That was already true of the existing
RPCs, but this PR raises the stakes. Any host that can reach Remote can change targets and
backends, read backend secrets, or delete a bucket with its objects. Any host that can reach a
sync node can replace its targets, or make it contact any address through VerifyBackend.
Until that is fixed, these ports must be limited to admin and BeeGFS hosts.
A pull that finishes after its target is deleted is still undone. Should it complete through
the retired provider instead?
Counting unfinished jobs scans the whole path DB. It runs only when a target moves, is
deleted or has its ID reused. A job submitted during the scan is not counted.
Retired providers keep their credentials from delete time. If the key is revoked first, the
undo fails.
bucket delete does not check deleted targets. Their stubs may still name the bucket, and
deleting it strands them.
delete confirms with --force, while some other commands use --yes.

What are the next steps after this PR?

Authentication on the Remote and sync gRPC servers.
A maintenance or deprecation state for targets.
An index of jobs by target, if the full scan becomes a problem.
Finding the stubs that name a deleted target, which bucket delete could then check too.
Letting a target clear a value its backend sets, and backends in the TOML file.

Checklist before merging:

Documentation:
- Developer documentation: code comments, CLAUDE.md, and beegfs-remote.toml.
- User documentation: the target, backend and bucket commands, the TOML import, the
  endpoint-url requirement and the upgrade order.
Testing:
- Unit tests cover every rule above, the push to a real gRPC sync node, and two providers
  sharing one connection. Backend verify runs against a local HTTP server that answers like
  S3. Bucket management runs against a fake S3 API.
- Integration tests: not yet run end to end. Still to do: a delete with queued work and a
  Remote restart, a delete during a builder walk, an offline node reconnecting, a backend
  change while jobs run, and the bucket commands against a real store.
Git Hygiene:
- Conventional Commits.
- Three commits on top of the Swartzn/fix/remote node connection handler #333 commits: targets and backends, buckets, and the shared S3
  HTTP client.

🤖 Generated with Claude Code

swartzn · 2026-11-03T18:18:21Z

Here's some useful diagrams: https://claude.ai/artifact/MFc9D3SkkE2Y8uTsFUrT2f?sk=LwuBqkn1c776XsbMZy5FuA

Remote storage targets could only be defined in the beegfs-remote TOML file, and changing one needed a restart of Remote and every sync node. Targets that shared a store had to repeat its endpoint and credentials. Add `beegfs remote target create|modify|delete|list`. Remote stores the definitions in a new BadgerDB (job.rst-db) and applies each change live, to itself and to every sync node. On first start it imports the TOML targets. After that the database is the only source. Add backends, managed with `beegfs remote backend create|modify|delete|list|verify`. A backend is the named connection to one remote store. A target of the same type names it with --backend and inherits every setting it leaves unset. Remote stores only what each one sets, so a change to a backend reaches every target that names it. Each RST type describes its configuration through rst.TargetType, which the CLI, Remote and rst.New use instead of switching on the type. The type also marks which settings a backend must set, and which it must not set. For S3 and xtreemstore, a backend must set endpoint-url, region, access-key and secret-key, and must not set bucket. Rules: - Every target passes one check, however it arrives: create, modify, a backend change, or the TOML import. Its type must be one a client supports, so Mock is refused. It must have every setting its type requires, and a provider must build from it. - A target's name is optional, as it was in the TOML file. Jobs, stubs and every command name a target by its ID. A name that is given must be at most 256 bytes, valid UTF-8 and free of control characters. A backend's name follows the same rule and is required, because targets refer to the backend by it. - An S3 or xtreemstore target must set endpoint-url. Before, an empty endpoint-url let the S3 library pick one on each node, from that node's environment, its shared config file or the region. So Remote and the sync nodes could disagree. A TOML file that relied on this stops Remote at the upgrade. The shipped TOML says how to fix it. - A backend must set region for the same reason. With no region, the S3 library takes one from each host's environment, and fails without one. - Nothing checks the form of a value, such as the endpoint URL. Only contacting the store proves a value works. - A target may set any setting itself, including one its backend must set, and its own value wins. - A target's type must match its backend's type. The CLI checks this before Remote does. - An invalid TOML file imports nothing and stops Remote with the target named. Fixing the file and restarting imports again. A TOML target cannot name a backend. - Modify changes only the flags given, and is refused if the target changed since it was read. - Changing one field of an inherited list entry, such as a storage class, keeps its other fields. - --backend= keeps the inherited values as the target's own. Remote fills them in from the backend's definition it holds when it applies the change, not from what the CLI read. A value the same modify clears falls back to the backend's value, as it would with the backend kept. - Moving a target's objects while unfinished jobs use it needs --force. - Delete needs --force. Unfinished jobs are cancelled and undone as Remote next meets them. The delete reason is at most 1024 bytes, with no control characters, as is a job cancel message. - A deleted ID is reused only with `create --id --force`. When the highest ID is taken, the next free ID is the lowest one never used. - A list entry is inherited whole. A backend's entries must be complete. - Deleting a backend is refused while a target names it. So a delete never changes a target. - `backend create|modify --verify` checks the new definition from Remote before storing it, and stores nothing when the store cannot be reached or rejects the keys. - `backend verify` checks a stored backend from Remote and from every sync node, and exits non-zero when a connected host fails. For S3 it lists at most one bucket. Keys that may not list buckets still pass, because the store checked them before it refused. A sync node that Remote still counts as connected but cannot reach is reported as not connected, not as a failed check. - job.rst-db defaults to rst.badger in the directory of job.path-db. The shipped TOML no longer sets it, so an older Remote still starts with that file, and it documents how to roll back. - A sync node that rejects a change starts no work until it accepts one. That includes a change it could not apply because it could not reach Remote. Sync nodes must be upgraded before Remote. - A change waits for a sync node that is connecting, so its push reaches the node. - `list --deleted` and `job cancel --message` require a Remote with target management, rather than silently doing nothing against an older one. - Every new command honors --output. - `beegfs remote list` keeps the output of earlier releases, because scripts parse it. It has the same columns, and policies stay text in JSON. `beegfs remote target list` has the new layout. Live changes also needed these fixes. A builder job whose target is deleted while it runs fails as a precondition instead of crashing sync. A sync node rebuilds its connection to Remote after a failed attempt. A deleted target stays findable through its retired provider while its delete applies. A target delete counts its unfinished jobs before it takes the lock that orders target changes. The protobuf dependency moves to 2c1f61d, which adds the RPCs for targets and backends. This does not: - Find stubs that name a deleted target. - Let a target clear a value its backend sets. - Let TOML define backends. - Require region on a target that names no backend. Such a target still depends on each host's environment, as before. - Keep a backend change made between a modify's read and its write, when the modify changes an inherited list entry. The modify copies the entry the CLI read. - Order job submission with a change. Remote applies a change to itself before it pushes it to each sync node. Work submitted in between can run on a node that still holds the old set. A new target then fails there as unknown. After a bucket or endpoint change, that node uses the old location, as a job run just before the change would. Tests cover each rule above in common/rst, rstdb, the job manager, the worker manager, the sync work manager and server, and the CLI backend and output. Verify is tested against a local HTTP server that answers like an S3 store. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

A backend holds the connection to a remote store, but its buckets still had to be made with another tool. That tool needed its own copy of the endpoint and the keys, on a host that can reach the store. Add `beegfs remote bucket list|create|modify|delete`, with the alias `container`. Each command names a backend with --backend. Remote runs every request against the store with that backend's settings. So the CLI host needs no route to the store and no keys. Each RST type manages its buckets through an rst.BucketManager, returned by TargetType.Buckets. S3 and xtreemstore share the S3 one. Rules: - create and modify set versioning (--versioning) and tags (--tag). modify can also remove tags (--remove-tag, --clear-tags). A store cannot turn versioning off once it was enabled. - create --create-target also creates a target that names the backend and sets only the bucket. If the bucket is created but the target is not, the bucket stays and the command fails. - list shows the buckets of every backend, or of one with --backend. --verbose adds versioning and tags. A backend whose store cannot be listed makes the command exit non-zero. A bucket whose versioning or tags cannot be read is still listed, with "-" for them, and Remote logs why. - create and modify do not fail when the bucket cannot be read back after the change. A retry of the create would only answer AlreadyExists. The details then show "-", and Remote logs why. - A store that does not implement bucket tagging has buckets without tags. - delete refuses a bucket that a live target uses, with or without --force. A target uses a bucket when its effective configuration names that bucket in the backend's store. Targets of every type with an S3 part count, so an xtreemstore target can use an S3 backend's bucket. - Two endpoints reach the same store when they differ only in host case, a default port, a trailing slash, or the bucket as the first label of the host. An endpoint that cannot be parsed counts as the same store, so the delete is refused. - delete refuses a bucket that holds objects, old versions or unfinished uploads, unless --force is given. With --force, Remote empties the bucket first and reports what it removed. - Emptying runs without the lock that orders target changes, because it can take long. While a delete runs, Remote refuses any target change that would point a target at that bucket. The final delete takes the lock and checks the targets again. - Every new command honors --output. This does not: - Check deleted targets. Their stubs may still name the bucket, and deleting it strands them. - Match one store reached through two different host names, such as an alias in DNS. - Store anything in rstdb, or push anything to sync nodes. The protobuf dependency moves to e5223df, which adds the bucket RPCs. Tests cover the S3 bucket manager against a fake S3 API, including endpoint spellings and details that cannot be read. They also cover the delete rules and the change lock in the job manager, a read back that fails after a change, and the CLI flag parsing. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

Every S3 provider built its own HTTP client, and so its own connection pool. Remote and the sync nodes now rebuild a target's provider whenever the target changes, and a change to a backend rebuilds the provider of every target that names it. Each rebuild dropped the warm connections to the endpoint, so the next requests paid for new TCP and TLS handshakes. A credential rotation on a backend did that for every bucket that shares it. All S3 clients now use one HTTP client, built once per process. Sharing is safe for any mix of targets: the transport keys idle connections by scheme, host, port and proxy, so different endpoints never share a connection, and the SDK signs each request, so credentials are not tied to one. - The shared client is built through the same SDK steps each provider's client went through, so it gets the root CAs from AWS_CA_BUNDLE and the timeouts from AWS_DEFAULTS_MODE. - It is assigned in an s3.New option, in newS3AWSClient, which providers and backend verify share. In a defaults mode other than "legacy", s3.New copies a client passed through aws.Config, and each copy builds its own transport. - Idle connections are no longer capped across endpoints. Per endpoint they are capped at the same GOMAXPROCS-scaled number as before, now for all targets on that endpoint together. The environment is read once, so a change to AWS_CA_BUNDLE or AWS_DEFAULTS_MODE needs a restart. With AWS_DEFAULTS_MODE=auto, every target gets the "standard" timeouts, because the shared client has no region to compare with the host's. A test rebuilds a provider with rotated credentials against a local server, under the defaults mode that would copy the client, and checks that both providers used one connection. Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>

swartzn requested a review from a team as a code owner November 3, 2026 18:17

swartzn requested a review from iamjoemccormick November 3, 2026 18:17

swartzn self-assigned this Nov 3, 2026

swartzn force-pushed the swartzn/feat/cli-managed-remote-storage-targets branch 4 times, most recently from 468731f to 51af297 Compare November 9, 2026 13:01

swartzn changed the base branch from main to swartzn/fix/remote-node-connection-handler November 9, 2026 13:02

swartzn force-pushed the swartzn/feat/cli-managed-remote-storage-targets branch 2 times, most recently from 32259a5 to 78d2dbe Compare November 9, 2026 13:12

swartzn added 2 commits November 9, 2026 13:27

swartzn force-pushed the swartzn/feat/cli-managed-remote-storage-targets branch from 78d2dbe to 5e149ed Compare November 9, 2026 19:28

Navigation Menu

Repository navigation

Swartzn/feat/cli managed remote storage targets - #396

Swartzn/feat/cli managed remote storage targets#396
swartzn wants to merge 3 commits into
swartzn/fix/remote-node-connection-handlerfrom
swartzn/feat/cli-managed-remote-storage-targets

swartzn commented Nov 3, 2026 •

edited

Loading

Uh oh!

swartzn commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

swartzn commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do / why do we need it?

Related Issue(s)

Where should the reviewer(s) start reviewing this?

Are there any specific topics we should discuss before merging?

What are the next steps after this PR?

Checklist before merging:

Uh oh!

swartzn commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

swartzn commented Nov 3, 2026 •

edited

Loading