Bug 255869

Summary:	[WebCodecs] Poor video frame rate exporting from Construct Animate
Product:	WebKit	Reporter:	Ashley Gullen <ashley>
Component:	Media	Assignee:	Kimmo Kinnunen <kkinnunen>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	eric.carlson, jean-yves.avenard, jer.noble, webkit-bug-importer, youennf
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified

Description Ashley Gullen 2023-04-24 07:15:42 PDT

Our animation software Construct Animate (https://animate.construct.net) uses WebCodecs to export animations to videos. It does this faster than realtime by rendering each frame to a canvas as fast as possible, and encoding each frame on the canvas with WebCodecs. Then it uses a library like mp4box.js to mux the resulting video to a playable media file which the user can preview or download.

This works fine in Chrome but in Safari 16.4 the resulting video has a poor framerate. It's inconsistent but seems to be around 1 frame per second, which is not enough to be useful. I also observed a tab crash in Safari following these steps once or twice but was not able to reliably reproduce that.

Steps to reproduce:
1. Open https://animate.construct.net/#open=make-animations-with-construct
2. Choose Menu > Project > Export...
3. Double-click "Video"
4. Click OK with default settings (should specify MP4 format with H.264 codec)
5. Approve any popup prompt (it is easier to allow popups for the site in advance)
6. The video export process should complete after a moment and show the result in a dialog. There the video can be previewed or downloaded.

Observed result:

In Safari the resulting video has a poor frame rate.

Expected result:

A smooth 30 FPS frame rate, as Chrome is able to produce.

Comment 1 Radar WebKit Bug Importer 2023-04-24 10:09:50 PDT

<rdar://problem/108459224>

Comment 2 youenn fablet 2023-05-09 05:48:13 PDT

Looking at the provided test, the encoding part seems fine, it encodes around 150 video frames.
Looking at WebCodecs input, it seems most VideoFrames have the same content as the previous ones.
The issue might be in the generation of the VideoFrames from canvas, or maybe the canvas drawing.

@Ashley, I have difficulties debugging your canvas -> VideoFrame code.
It might help if you could you take a look at your canvas -> VideoFrame code path, maybe isolate what you are doing in a separate page that I could play with.

Comment 3 Ashley Gullen 2023-05-22 06:42:49 PDT

I tried making a minimal reproduction but it worked fine and did not demonstrate the problem. So I'm afraid I'm not sure what it is about what Construct Animate is doing specifically that is resulting in this problem with Safari only.

Comment 4 Kimmo Kinnunen 2023-10-23 05:50:24 PDT

Pull request: https://github.com/WebKit/WebKit/pull/19413

Comment 5 EWS 2023-10-25 03:02:48 PDT

Committed 269757@main (3cf847f4c0b4): <https://commits.webkit.org/269757@main>

Reviewed commits have been landed. Closing PR #19413 and removing active labels.