Skip to content

feat(rst,watch): require connection authentication on Remote and Sync - #398

Open
swartzn wants to merge 6 commits into
mainfrom
swartzn/feat/add-connection-authentication-support
Open

swartzn wants to merge 6 commits into
mainfrom
swartzn/feat/add-connection-authentication-support

Conversation

@swartzn

@swartzn swartzn commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

What does this PR do / why do we need it?

Remote and Sync never checked their gRPC callers. Any host that could reach them could manage jobs
and RSTs, or make Sync, which skips file permission checks, read or write any file in BeeGFS.
Remote also sent its raw auth secret to every Sync node.

Both servers now require the connection authentication hash from every caller when the BeeGFS
client of their mount uses connection authentication. Each service reads the client's
connDisableAuthentication and connAuthFile from /proc/fs/beegfs. If it cannot read them, it
refuses to start unless the optional [auth] section, with auth-file or auth-disable, says what
to do. No setting can turn the check off while the client can be read.

  • Remote no longer sends the secret to Sync. Each node reads its own file and sends only the hash.
  • Remote's management.auth-file and management.auth-disable are deprecated and ignored.
  • beegfs-watch sends the hash only to subscribers that set grpc-send-auth-secret.

We gain authenticated Remote and Sync endpoints, and the raw secret no longer leaves a host. We
lose mixed-version operation: Remote, Sync and Watch must be upgraded together.

Related Issue(s)

https://github.com/ThinkParQ/beegfs-core/issues/4813

Where should the reviewer(s) start reviewing this?

  1. rst/internal/connauth/connauth.go: the package comment states the rules, and resolve applies them.
  2. common/beegfs/beegrpc/auth.go: AuthServerOptions, the server-side check.
  3. rst/remote/cmd/beegfs-remote/main.go and rst/sync/cmd/beegfs-sync/main.go: where each
    service resolves its state and wires it into its server, CTL and outgoing calls.
  4. rst/remote/internal/worker/beesync.go and rst/sync/internal/beeremote/grpc.go: how Remote
    and Sync authenticate to each other.
  5. watch/internal/subscriber/: the grpc-send-auth-secret opt-in.

Are there any specific topics we should discuss before merging?

  • Containers. A container running Remote or Sync must be able to read /proc/fs/beegfs, and
    the auth file at the path the client reports. Otherwise [auth] must be set. Mounting the client
    inside the container meets both requirements.
  • Old clients. A client that does not report connDisableAuthentication counts as unreadable,
    so it also needs [auth]. We have not checked which client version added the key.
  • Watch upgrade. An existing Watch subscriber entry for Remote must add
    grpc-send-auth-secret = true. Without it, events stop reaching Remote once Remote checks, and
    nothing fails at startup.
  • Scope of the check. It proves membership of the file system, not admin rights. Every host
    that holds the secret can still use the full Remote API.

What are the next steps after this PR?

  • Deprecate the unused auth_secret and auth_disable fields in flex.proto.
  • Release notes: the lockstep upgrade, the Watch opt-in, the deprecated management.auth-*
    settings, and the container requirements.

Checklist before merging:

  • Documentation:
    • Developer documentation: code comments and the shipped TOML files are updated.
    • User documentation: still needs the items listed under next steps.
  • Testing:
    • Unit tests added for the client and [auth] rules, decoding of unset [auth] keys, the
      server check, and authentication between Remote and Sync.
    • Integration testing: not yet run on a cluster with connection authentication.
  • Git Hygiene:
    • Do commits adhere to Conventional Commits?
    • Is the commit history squashed down reasonably?

🤖 Generated with Claude Code

Remote and Sync never checked their gRPC callers. Any host that could reach them could manage
jobs and RSTs, or make Sync, which skips file permission checks, read or write any file in BeeGFS.
Remote also sent its raw auth secret to every Sync node.

Both servers now require the connection authentication hash from every caller when the BeeGFS
client of their mount uses connection authentication. Each service reads the client's
connDisableAuthentication and connAuthFile from /proc/fs/beegfs. If it cannot read them, it refuses
to start unless the optional [auth] section, with auth-file or auth-disable, says what to do. No
setting can turn the check off while the client can be read.

- Remote no longer sends the secret to Sync. Each node reads its own file and sends only the hash.
- Remote's management.auth-file and management.auth-disable are deprecated and ignored.
- beegfs-watch sends the hash only to subscribers that set grpc-send-auth-secret.

Remote, Sync and Watch must be upgraded together. The unused auth_secret and auth_disable proto
fields are left for a separate change.

Tests cover the client and [auth] rules, the server check, and authentication between Remote and
Sync.

Assisted-by: Claude:claude-opus-5-5
@swartzn swartzn self-assigned this Nov 10, 2026
@swartzn
swartzn requested a review from a team as a code owner November 10, 2026 15:52
Remote and Sync rejected callers that failed connection authentication, but logged nothing. The
caller saw Unauthenticated, while the operator of the server saw no sign of it. Watch had the
opposite gap. A subscriber that rejected the event stream looked connected. Watch only saw a bare
io.EOF when it next sent an event, so the reason was lost.

AuthServerOptions now takes a logger and warns about rejected calls. The warning names the caller
address, the method and the reason. It never holds the secret or the hash the caller sent. A caller
with the wrong secret usually retries in a loop, so the server logs the first rejection and then at
most one a minute. Each log counts the rejections it skipped.

Watch now probes each event stream right after it opens it. Headers mean the subscriber accepted.
A stream that ends without headers is a rejection, so Connect fails with its status and the handler
retries with backoff. A subscriber that sends nothing for two seconds is treated as accepted, as
before. The subscriber library sends headers as soon as it accepts a stream, so it never waits.
The error that ends the receive goroutine is now kept, and the handler logs it. An Unauthenticated
status gets a hint to set grpc-send-auth-secret.

This change does not alter which calls are accepted or rejected.

Tests cover the rate limit, the log fields and that no secret is logged. They also cover the
probe for accepted, rejected, silent and authenticated subscribers, and the kept receive error.

Assisted-by: Claude:claude-opus-5-5
The connection authentication change tested its rules and the shared
interceptor. Its wiring had no tests: FromMount, Remote's startup, both
gRPC servers, the legacy auth file cleanup and a Sync node rejected by
Remote. A wiring mistake there could leave a server open, and no test
would catch it.

The change adds those tests. Three small refactors let them run without
a BeeGFS mount or root:

- connauth reads procfs through a getClients variable that tests stub.
- legacySyncAuthFile is a variable, so tests can point it at a temporary
  directory.
- Remote's startup moves into ctlAuth, setWorkerAuth and
  newBeeRemoteNode, so each step can be tested on its own.

Behaviour does not change.

Tests check:
- FromMount filters by the mount point and rejects two clients.
- FromMount logs the source, and never logs the secret.
- Both servers reject a missing or wrong hash before the handler runs,
  stay open with a nil secret, and rate-limit rejection logs.
- The description Remote pushes to Sync carries no secret.
- Sync answers UpdateConfig with a failure when Remote rejects its
  secret, and Remote treats the node as failed.
- Deprecated management.auth-* settings still parse.
- The [auth] settings decode as unset when absent, in Remote and Sync.

Assisted-by: Claude:claude-opus-5-5
govulncheck fails CI with GO-2026-6865. Version v1.18.7 of the compress module panics when it
decodes a frame that holds two blocks. Badger reaches the affected fse and huff0 code through
kvstore.NewMapStore, and Sync reaches it through workmgr.Manager.UpdateConfig.

The module is an indirect dependency of badger. This change raises it to v1.20.2, the first fixed
release. No code changes are needed. The NOTICE files stay the same because the license text did
not change.

Tests: go build ./... and make test-unit pass. govulncheck no longer reports GO-2026-6865.

Assisted-by: Claude:claude-opus-5-5
govulncheck fails CI with GO-2026-6859. In Go 1.27.3, MaxBytesReader in net/http races on the
request header, and the race can crash an HTTP/1 server. Every binary reaches net/http, for
example through the S3 client, the OTel exporters and the beegfs-remote pprof listener.

The fix is in the standard library of Go 1.27.4. This change raises the go directive, which CI
uses to pick the toolchain. No code or dependency changes are needed.

Tests: under Go 1.27.4, check-go-version, check-gofmt, check-linters, check-go-tidy, test-unit and
check-licenses pass. govulncheck reports 0 vulnerabilities that the code reaches.

Assisted-by: Claude:claude-opus-5-5
CLAUDE.md said the toolchain was Go 1.26.4. go.mod has since moved to 1.27.4, so the text was two
bumps out of date. The sentence now points to go.mod, which is the one place the version is set.

Assisted-by: Claude:claude-opus-5-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant