Repository navigation
fix(replays): Keep replay recordings in /data/replays and prune them - #4607
Draft
sentry-junior[bot] wants to merge 2 commits into
Draft
sentry-junior[bot] wants to merge 2 commits into
sentry-junior[bot] wants to merge 2 commits into
Conversation
…olicy Replay recording segments were written to the default filestore on the sentry-data volume. Sentry writes them without File rows and relies on a bucket lifecycle policy to expire them, so nothing ever deleted them. Store them in a 'replays' SeaweedFS bucket that expires objects after SENTRY_EVENT_RETENTION_DAYS, mirroring the profiles bucket setup. Refs getsentry/sentry#121278 Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
Coverage Results 📊✅ 22 passed | Total: 22 | Pass Rate: 100% | Execution Time: 9m 40s 📊 Comparison with Base Branch
✨ Test counts unchanged from base. All tests are passing successfully. ✅ Patch coverage is 100.00% (no changed executable lines found; target 50%). Coverage diff@@ Coverage Diff @@
## master #4607 +/-##
==========================================
Coverage 95.58% 95.58% —%
==========================================
Files 5 5 —
Tracked lines 339 339 —
Branches 0 0 —
==========================================
Hits 324 324 —
Misses 15 15 —
Partials 0 0 —Generated by Coverage Action |
Collaborator
|
@DominikB2014 I would avoid using SeaweedFS here. Can we just use |
Drop the SeaweedFS bucket in favor of the filesystem backend with a dedicated /data/replays directory. A daily sentry-cleanup cron job deletes recordings older than SENTRY_EVENT_RETENTION_DAYS. On upgrade, install points replay.storage.* at /data/replays and moves existing recordings out of /data/files so they keep playing back and expire like new ones. Co-Authored-By: Dominik Buszowiecki <44422760+DominikB2014@users.noreply.github.com>
sentry-junior
Bot
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft description. The author should rewrite it in their own words before marking this ready.
Replay recording segments are written to the default filestore (
/data/filesonsentry-data) becausereplay.storage.backendis not set. Sentry writes these blobs straight to storage, with noFilerows, sosentry cleanupnever deletes them and they pile up forever on both fresh installs and upgrades. See getsentry/sentry#121278.This keeps the
filesystembackend but pointsreplay.storage.*at a dedicated/data/replaysdirectory. A second daily cron entry insentry-cleanupdeletes anything there older thanSENTRY_EVENT_RETENTION_DAYS. The newinstall/migrate-replay-storage.shcreates the directory. On upgrades, ifAPPLY_AUTOMATIC_CONFIG_UPDATESallows it, it appends the config and moves existing recordings out of/data/files. It only moves<retention>/<project_id>/<32-hex replay_id>directories, so regular file blobs (<2hex>/<4hex>/<26hex>files) are never touched. Moved recordings keep their mtimes, so ones already past retention are removed on the next cleanup run.Open question:
APPLY_AUTOMATIC_CONFIG_UPDATES=1switch the path and move recordings without a prompt.Legal Boilerplate
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
via @DominikB2014.
--
View Session [Sentry]