Skip to content

Better detect when to drop late frames with EFFECTS_ATLAS - #96

Draft
moi15moi wants to merge 1 commit into
peerless2012:masterfrom
moi15moi:Handle-better-frame-drops
Draft

moi15moi wants to merge 1 commit into
peerless2012:masterfrom
moi15moi:Handle-better-frame-drops

Conversation

@moi15moi

@moi15moi moi15moi commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

With this sample (https://files.catbox.moe/8u4w9r.mkv), I was getting around 60 frames dropped. Now, with these changes, I get around 7 to 10 frames.

With EFFECTS_ATLAS, libass renders the subtitles inside media3's video effects. When rendering is slower than the frame rate (heavy typesetting, animated karaoke), media3's two ways of dropping late frames work badly:

  • After the effects, media3 drops the frames that turned out late. By then the subtitles are already rendered, so dropping the frame saves no time: the work is thrown away and the next frame is just as late.
  • Before the effects, media3 drops frames it predicts will be late, based on how late the frames it showed were. That information is a couple of frames old, and dropped frames don't count toward it, so while libass stays slower than the frame rate the prediction keeps saying "late". It ends up dropping up to 2 frames out of 3, when dropping about 1 out of 10 would be enough to keep up.

The result is very choppy video where a few dropped frames would do.

Add AssMediaCodecVideoRenderer, right before a frame enters the video effects, drop it if its display time is already over (pts + duration < now). Never drop if no frame was shown for 100ms, so playback slows to about 10fps instead of freezing. Only frames that can no longer be shown on time are dropped, and they are dropped before libass spends any time on them.

AssRenderersFactory takes an optional Context to create the renderer and buildWithAssSupport() passes it. withAssSupport() gets the same optional parameter. Without a Context, or when the video renderer is a subclass of MediaCodecVideoRenderer (which may have behavior of its own), the renderer is kept and media3's dropping is used, with a warning in the log.

@ethaldeman @damontecres

With EFFECTS_ATLAS, libass renders the subtitles inside media3's video effects.
When rendering is slower than the frame rate (heavy typesetting, animated karaoke), media3's two ways of dropping late frames work badly:

- After the effects, media3 drops the frames that turned out late.
  By then the subtitles are already rendered, so dropping the frame saves no time:
  the work is thrown away and the next frame is just as late.
- Before the effects, media3 drops frames it *predicts* will be late, based on how late the frames it showed were.
  That information is a couple of frames old, and dropped frames don't count toward it,
  so while libass stays slower than the frame rate the prediction keeps saying "late".
  It ends up dropping up to 2 frames out of 3, when dropping about 1 out of 10 would be enough to keep up.

The result is very choppy video where a few dropped frames would do.

Add AssMediaCodecVideoRenderer, right before a frame enters the video effects, drop it if its display time is already over (pts + duration < now).
Never drop if no frame was shown for 100ms, so playback slows to about 10fps instead of freezing.
Only frames that can no longer be shown on time are dropped, and they are dropped before libass spends any time on them.

AssRenderersFactory takes an optional Context to create the renderer and buildWithAssSupport() passes it.
withAssSupport() gets the same optional parameter.
Without a Context, or when the video renderer is a subclass of MediaCodecVideoRenderer (which may have behavior of its own),
the renderer is kept and media3's dropping is used, with a warning in the log.
@ethaldeman

ethaldeman commented Nov 14, 2026 •

Copy link
Copy Markdown

I tested this pull request using the sample. To the human eye, playback looked similar to my previous test mentioned in damontecres/Wholphin#2227. However the dropped frames did improve from 124 to 97.

I would love to test this build integrated with Wholphin and see if the ~30 seconds video and audio desycn is improved, but that is beyond my current skill to setup and build. If that apk was provided, I would be happy to test it though.

@moi15moi

moi15moi commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

However the dropped frames did improve from 124 to 97.

I am surprised that you only get this kind of amelioration.
If you use mpv-android, how much frame does it drop with the sample I provided (https://files.catbox.moe/8u4w9r.mkv)?

For me, mpv drops around 20 frames. Before this PR, I was getting around 60 frame dropped and now I get 7 to 10, so a pretty big improvement.

I would love to test this build integrated with Wholphin and see if the ~30 seconds video and audio desycn is improved,

Probably not. This PR doesn't do anything related to that.
It's probably a bug in wolphin, but I am not too sure about it.

@ethaldeman

Copy link
Copy Markdown

However the dropped frames did improve from 124 to 97.

I am surprised that you only get this kind of amelioration. If you use mpv-android, how much frame does it drop with the sample I provided (https://files.catbox.moe/8u4w9r.mkv)?

For the original sample ioxho8.mkv and pure mpv-android, i had 89 dropped frames, pretty comparable to this pull requests 97 dropped frames. 8u4w9r.mkv had 227 dropped frames in pure mpv-android and 235 dropped frames for this pull requests EFFECTS_ATLAS

This all suggests this pull request brings EFFECTS_ATLAS performance pretty close to mpv!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants