Skip to content

Keep the backend running when Redis drops - #1315

Open
bfliao wants to merge 1 commit into
mainfrom
leo/redis-error-handler
Open

bfliao wants to merge 1 commit into
mainfrom
leo/redis-error-handler

Conversation

@bfliao

@bfliao bfliao commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Problem

When Redis goes down, the backend crashes with it.

  • The trigger: the Redis client (apps/backend/src/bootstrap/loaders/redis.ts) has no error listener, and the backend has no uncaughtException handler. node-redis emits error when the connection drops and on every failed reconnect attempt. With no listener, Node treats that as an uncaught exception and the process exits.
  • The effect: when Redis restarts (it was OOM-killed on Oct 22), every backend pod exits at the same moment and the API is down. It stays down until Redis is back and the pods restart and pass their health checks.
  • Reproduced locally with the backend's client setup and version (node-redis 5.8.3): when the connection drops, the process exits immediately with Unhandled 'error' event: SocketClosedUnexpectedlyError.

Change

  • New apps/backend/src/lib/redis-client.ts: creates the client with an error listener that logs and lets node-redis reconnect on its own.
    • Errors are logged at most once every 10 seconds, with a count of the ones skipped, because failed reconnects can repeat several times a second.
    • It also logs when the connection is ready, so recovery shows up in the logs.
  • bootstrap/loaders/redis.ts now uses it. It lives in lib/ because config.ts requires about 20 env vars, which would make the loader hard to test.
  • While Redis is down, commands wait in node-redis's offline queue, so requests that need Redis are delayed rather than crashing the process. The API recovers by itself when Redis returns.

Testing

  • New test src/lib/redis-client.test.ts: starts a fake Redis, connects, drops the connection and closes the port (as an OOM kill would), then brings the server back on the same port. The process survives and the client reconnects.
    • With the fix: passes. The logs show connected → one "Redis error, reconnecting" → connected.
    • With the listener removed: fails in under a second with failureType: 'uncaughtException' (Socket closed unexpectedly), and the test run exits instead of hanging.
  • npm test in apps/backend: 9/9 pass.
  • ESLint and Prettier pass on the changed files.
  • tsc reports the same 250 errors as main, none in these files. They come from GraphQL types that weren't generated locally.

Not covered

  • A long Redis outage: requests that need Redis wait in the offline queue until it reconnects, rather than failing fast.
  • What Redis loses on restart: sessions and caches are still lost (no persistence). Making Redis restarts rare is Deploy bt-base, Mongo and Redis from the repository #1312's job (maxmemory and a 512Mi limit for bt-prod-redis).

Generated by Claude Code

The Redis client had no "error" listener and the backend has no
uncaughtException handler. node-redis emits "error" when the connection
drops and on every failed reconnect, so when Redis restarted (it was
OOM-killed on Oct 22) Node treated it as an uncaught exception and every
backend pod exited at once, taking the API down until Redis and the pods
came back.

Create the client in lib/redis-client.ts with an "error" listener that
logs (at most every 10 seconds, with a count of skipped errors) and lets
node-redis reconnect; commands wait in its offline queue meanwhile. Also
log when the connection is ready.

The new test drops a fake Redis server and brings it back on the same
port: the process survives and the client reconnects. Without the
listener the test fails with the same unhandled 'error' event.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132twi4SNP1UzsCZvrPLhy6
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-08T10:05:49.309609Z
We couldn’t complete this review. Comment @codex review to try again.
ad01aa2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bfliao
bfliao requested a review from sashakmurray November 11, 2026 00:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants