Created attachment 467354 [details] gdb (bt full; c) output.txt On Nvidia RTX 4070 (driver version 535) 1. Set WEBKIT_DMABUF_RENDERER_DISABLE_GBM=1 2. Set WEBKIT_GST_DMABUF_SINK_DISABLED=1 3. Visit apple.com/apple-watch-series-8 4. Crash
Created attachment 467384 [details] gdb (bt full; c) 2.txt Here's a similar one encountered on https://www.apple.com/apple-watch-ultra/
*** Bug 260984 has been marked as a duplicate of this bug. ***
*** Bug 261872 has been marked as a duplicate of this bug. ***
This isn't an NVIDIA-related issue, because in my bug #260984 I hit the same crash with AMD graphics.
This seems to be a regression. This website has always worked before this bug was reported and Apple didn't change the website design
Are you able to reproduce the crash reliably? I'm not able to trigger it by visiting those apple.com websites.
Yes I am able to trigger it reliably
If you could figure out which WebKitGTK release it broke in, that would help. If you could bisect it, that would help even more (but if you're not familiar with building WebKit, this may not be easy).
I do know how to build WebKit, but as I've said on the Matrix channel, all downloads from the Igalia repo fail so it's not convenient (I have to use system libraries in a toolbox), and the resultant MiniBrowser can't get GPU acceleration (I am unable to work around this issue). Beyond that, I also don't know how to use git bisect so instructions are welcome
(In reply to Kdwk from comment #9) > I do know how to build WebKit, but as I've said on the Matrix channel, all > downloads from the Igalia repo fail so it's not convenient (I have to use > system libraries in a toolbox), System libraries in a toolbox is the way to go. That said, is there a bug report for the problem with the Igalia repo? We need to either fix it or change build-webkit to stop depending on it. > and the resultant MiniBrowser can't get GPU > acceleration (I am unable to work around this issue). Beyond that, I also > don't know how to use git bisect so instructions are welcome Is that GPU acceleration problem caused by toolbx, perhaps? I wonder if GPU acceleration is required for you to reproduce this bug reliably? git bisect is really easy to use, if you have a regression you can confidently reproduce to determine whether a particular commit is bad or good. Example tutorial: https://stackoverflow.com/a/37306623/1120203
*** Bug 263509 has been marked as a duplicate of this bug. ***
*** Bug 266535 has been marked as a duplicate of this bug. ***
Here with Canary and WEBKIT_GST_DMABUF_SINK_DISABLED=1 and va decoders ranked to 0 I get scrambled output.
Created attachment 469480 [details] screenshot
With default dmabuf sink enabled and va decoders up-ranked, all is fine here... Anyway, is this crash in webKitMediaSrcStreamFlush still happening?
It is still reliably happening in WebKitGTK 2.42.4. The original pages no longer exist so here's another one: apple.com/apple-watch-ultra-2
I got a critical warning here in Ephy TP Commit: 0e3a544be3ab1e038c36260379e91163f7f02a7d3eb8d62d3a4ae9ff3a91d626 Parent: b0d6f546c77add052c453376afa9afacb0f0ba77ce055b57bf7644015564c650 Subject: Export org.gnome.Epiphany.Devel Date: 2024-01-21 05:07:56 +0000 (WebKitWebProcess:2): GStreamer-CRITICAL **: 17:44:13.183: gst_caps_remove_structure: assertion 'IS_WRITABLE (caps)' failed (gdb) bt #0 g_logv (log_domain=0x7f490fb229ae "GStreamer", log_level=G_LOG_LEVEL_CRITICAL, format=<optimized out>, args=args@entry=0x7f483bdd8fb0) at ../glib/gmessages.c:1277 #1 0x00007f490eec2233 in g_log (log_domain=<optimized out>, log_level=<optimized out>, format=<optimized out>) at ../glib/gmessages.c:1315 #2 0x00007f4843ea5f14 in gst_vp9_parse_negotiate (in_align=<optimized out>, in_caps=0x7f483000aae0 [GstCaps], self=0x559120b5b320 [GstVp9Parse|V0_parser]) at ../gst/videoparsers/gstvp9parse.c:299 #3 gst_vp9_parse_set_sink_caps (parse=0x559120b5b320 [GstVp9Parse|V0_parser], caps=<optimized out>) at ../gst/videoparsers/gstvp9parse.c:816 #4 0x00007f490fbaa7f8 in gst_base_parse_sink_event_default (parse=0x559120b5b320 [GstVp9Parse|V0_parser], event=0x7f4830007580 [GstEvent]) at ../libs/gst/base/gstbaseparse.c:1244 #5 0x00007f490fac466d in gst_pad_send_event_unchecked (pad=pad@entry=0x559120b4ded0 [GstPad|sink], event=event@entry=0x7f4830007580 [GstEvent], type=<optimized out>, type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM) at ../gst/gstpad.c:5939 #6 0x00007f490fac4d53 in gst_pad_push_event_unchecked (pad=pad@entry=0x7f483000dde0 [GstPad|video_0], event=0x7f4830007580 [GstEvent], type=<optimized out>, type@entry=GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM) at ../gst/gstpad.c:5572 #7 0x00007f490fac5518 in push_sticky (pad=pad@entry=0x7f483000dde0 [GstPad|video_0], ev=ev@entry=0x7f483bdd9410, user_data=user_data@entry=0x7f483bdd9480) at ../gst/gstpad.c:4057 #8 0x00007f490fab9c85 in events_foreach (pad=0x7f483000dde0 [GstPad|video_0], func=0x7f490fac5470 <push_sticky>, user_data=0x7f483bdd9480) at ../gst/gstpad.c:613 #9 0x00007f490fac85f1 in check_sticky (event=0x7f483000baa0 [GstEvent], pad=0x7f483000dde0 [GstPad|video_0]) at ../gst/gstpad.c:4116 #10 gst_pad_push_event (pad=0x7f483000dde0 [GstPad|video_0], event=0x7f483000baa0 [GstEvent]) at ../gst/gstpad.c:5705 #11 0x00007f488c0d4a0e in gst_matroska_demux_send_tags (demux=demux@entry=0x559120b54320 [GstMatroskaDemux|matroskademux0]) at ../gst/matroska/matroska-demux.c:1993 #12 0x00007f488c0dfef1 in gst_matroska_demux_parse_id (demux=0x559120b54320 [GstMatroskaDemux|matroskademux0], id=<optimized out>, length=<optimized out>, needed=6) at ../gst/matroska/matroska-demux.c:5655 #13 0x00007f488c0e7df4 in gst_matroska_demux_chain (pad=pad@entry=0x559120b54920 [GstPad|sink], parent=parent@entry=0x559120b54320 [GstMatroskaDemux|matroskademux0], buffer=<optimized out>, buffer@entry=0x5591209dbf30 [GstBuffer]) at ../gst/matroska/matroska-demux.c:6202 #14 0x00007f490fac2eec in gst_pad_chain_data_unchecked (pad=pad@entry=0x559120b54920 [GstPad|sink], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4463 #15 0x00007f490fac628e in gst_pad_push_data (pad=pad@entry=0x559120b59800 [GstPad|src], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4739 #16 0x00007f490fac68c4 in gst_pad_push (pad=0x559120b59800 [GstPad|src], buffer=0x5591209dbf30 [GstBuffer]) at ../gst/gstpad.c:4858 #17 0x00007f490fbbda5c in gst_base_transform_chain (pad=pad@entry=0x559120b56520 [GstPad|sink], parent=parent@entry=0x559120b56110 [GstIdentity|identity0], buffer=buffer@entry=0x5591209dbf30 [GstBuffer]) at ../libs/gst/base/gstbasetransform.c:2391 #18 0x00007f490fac2eec in gst_pad_chain_data_unchecked (pad=pad@entry=0x559120b56520 [GstPad|sink], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4463 #19 0x00007f490fac628e in gst_pad_push_data (pad=pad@entry=0x559120b56aa0 [GstPad|src], type=type@entry=4112, data=data@entry=0x5591209dbf30) at ../gst/gstpad.c:4739 #20 0x00007f490fac68c4 in gst_pad_push (pad=pad@entry=0x559120b56aa0 [GstPad|src], buffer=0x5591209dbf30 [GstBuffer]) at ../gst/gstpad.c:4858 #21 0x00007f490fbc1efb in gst_base_src_loop (pad=0x559120b56aa0 [GstPad|src]) at ../libs/gst/base/gstbasesrc.c:3035 #22 0x00007f490faf3204 in gst_task_func (task=0x559120b5ae00 [GstTask|appsrc0:src]) at ../gst/gsttask.c:384 #23 0x00007f490eee92c2 in g_thread_pool_thread_proxy (data=<optimized out>) at ../glib/gthreadpool.c:336 #24 0x00007f490eee86c9 in g_thread_proxy (data=0x7f48fc0019d0) at ../glib/gthread.c:821 #25 0x00007f49132a1e39 in start_thread (arg=<optimized out>) at pthread_create.c:444 #26 0x00007f4913329904 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Apart from that, unable to reproduce the issue (but I'm on Intel and AMD).
I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting apple.com/apple-watch-ultra-2 with software decoding and scrolling all the way down crashes the WebProcess.
(In reply to Kdwk from comment #19) > I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting > apple.com/apple-watch-ultra-2 with software decoding and scrolling all the > way down crashes the WebProcess. Can you share the backtrace? The one I shared earlier results from a warning, so it shouldn't trigger crashes, unless you set this env var G_DEBUG=fatal-criticals
(In reply to Philippe Normand from comment #17) > I got a critical warning here in Ephy TP > > (WebKitWebProcess:2): GStreamer-CRITICAL **: 17:44:13.183: > gst_caps_remove_structure: assertion 'IS_WRITABLE (caps)' failed > https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/5975
Created attachment 469531 [details] gdb (MacBook; software decoding).txt (In reply to Philippe Normand from comment #20) > (In reply to Kdwk from comment #19) > > I am able to reproduce this on non-Nvidia hardware. On my MacBook, visiting > > apple.com/apple-watch-ultra-2 with software decoding and scrolling all the > > way down crashes the WebProcess. > > Can you share the backtrace? > > The one I shared earlier results from a warning, so it shouldn't trigger > crashes, unless you set this env var G_DEBUG=fatal-criticals
Can you collect gst logs? Do you remember how?
I haven't been able to reproduce the crash on apple.com, but I hit this crash 100% of the time in Ephy Tech Preview (WebKitGTK 2.43.3, GStreamer 1.22.5) when loading https://www.newsweek.com/missouri-republican-senators-duel-nick-schroer-1863838 and scrolling down the page. Backtrace is basically the same as what I posted in bug #260984. I will attach a gst.log following the instructions https://trac.webkit.org/wiki/WebKitGTK/Debugging#Debuggingmultimediastuff. We should really move all the instructions you care about to https://docs.webkit.org/ so we can have some updated link to point to instructions.
Created attachment 469550 [details] Debug log BTW, since streamByName may return nullptr, I suggest webKitMediaSrcFlush should either handle that case or assert that it returns non-null.
That log looks incomplete. I can't reproduce this issue here in Ephy TP.
(In reply to Michael Catanzaro from comment #24) > I will attach a gst.log following the instructions > https://trac.webkit.org/wiki/WebKitGTK/Debugging#Debuggingmultimediastuff. > We should really move all the instructions you care about to > https://docs.webkit.org/ so we can have some updated link to point to > instructions. https://github.com/WebKit/Documentation/pull/78
(In reply to Philippe Normand from comment #26) > That log looks incomplete. I reproduced the issue today and got a second log. It looks the same as the first. That's really all there is before the crash occurs. I'll attach my second debug log and the dots requested by your new documentation.
Created attachment 469566 [details] Debug
BTW to reproduce in Tech Preview, just keep scrolling up and down the page. It seems to crash about 70% of the time, not the 100% that I claimed earlier. If it refuses to crash, then I press Ctrl+R and try again and it will probably crash.
I can reproduce the crash now, for the record, you need to start playing the video (auto play doesn't kick in here) and scroll down until the player moves to PiP state, then scroll up until it goes back to non-PiP state, and scroll down again and so on... So it seems the MSE src element tears down its streams and later on a seek triggers a flush on the same src element... One of the issues is that m_hasAllTracks in MediaSourcePrivateGStreamer doesn't seem to be set back to false after the streams have been removed...
Created attachment 470445 [details] region in the page where the crash happens I can reproduce the crash when scrolling down on https://apple.com/apple-watch-ultra-2 until it hits this area where the watch side view is a video element that is played and seeked as you scroll through it (you may need to scroll past it and return). Bisected it down to https://commits.webkit.org/265206@main. Before the user agent quirk, the video element played this MP4 file, without MSE: https://www.apple.com/105/media/us/apple-watch-ultra-2/2023/4d9e62e1-fe94-4bb9-abbe-0b8c9626a304/anim/schematic_rotation-2/large.mp4 After the user agent quirk, it now plays this WebM file instead, that has alpha channel signaled in the WebM container, and it's played via MSE: https://www.apple.com/105/media/us/apple-watch-ultra-2/2023/4d9e62e1-fe94-4bb9-abbe-0b8c9626a304/anim/schematic_rotation-2/large.webm (In reply to Michael Catanzaro from comment #25) > Created attachment 469550 [details] > Debug log > > BTW, since streamByName may return nullptr, I suggest webKitMediaSrcFlush > should either handle that case or assert that it returns non-null. Indeed. Though checking for a null Stream pointer and bailing early doesn't seem sufficient, as it does fixes the crash in webKitMediaSrcFlush, but then then video element is broken, not displaying anything. I'm investing this further.
(In reply to Carlos Bentzen from comment #32) > > I'm investing this further. investigating*, obviously.
(In reply to Carlos Bentzen from comment #32) > Bisected it down to https://commits.webkit.org/265206@main. Good job!
Found another reproducer for this crash. Try to play this video on nbcnews.com: https://www.nbcnews.com/news/us-news/toddler-dies-pinned-tire-uber-suv-dropped-houston-rcna144187
(Um, although maybe I should have picked a different video for bug report purposes. Presumably that one contains disturbing content if it doesn't trigger the crash.)
(In reply to Michael Catanzaro from comment #34) > (In reply to Carlos Bentzen from comment #32) > > Bisected it down to https://commits.webkit.org/265206@main. > > Good job! Well, I'm not sure this can be flagged as regression, unless we update the bug title again to be specific to Apple website.
(In reply to Philippe Normand from comment #37) > (In reply to Michael Catanzaro from comment #34) > > (In reply to Carlos Bentzen from comment #32) > > > Bisected it down to https://commits.webkit.org/265206@main. > > > > Good job! > > Well, I'm not sure this can be flagged as regression, unless we update the > bug title again to be specific to Apple website. Yeah, we get different content served with the user-agent quirk, but the new content seems valid on Firefox and Chrome, so the GStreamer MSE code is the one broken IMO, and was broken before already. Had the same page been served with WebM + MSE before r265206, it would also crash (I checked with the test below). Reduced the test case down to https://people.igalia.com/cadubentzen/webkit/bug260455. Scrolling past the video area and back, I get a crash reliably. (the video area is blank in webkitgtk, it doesn't play). In https://people.igalia.com/cadubentzen/webkit/bug260455_2, on the other hand, the video plays and I get no crash anymore. The only difference is the web page starts with the video in the viewport. Philippe pointed out to me that we have a setting via the environment variable WEBKIT_GST_ALLOW_PLAYBACK_OF_INVISIBLE_VIDEOS. Setting that to 1, the video plays and I get no crashes, so it's definitely related. Continuing to investigate...
Pull request: https://github.com/WebKit/WebKit/pull/26472
Committed 276798@main (f91aeb92bd8e): <https://commits.webkit.org/276798@main> Reviewed commits have been landed. Closing PR #26472 and removing active labels.
Reopened Bugzilla. Causes excessive CPU usage of cached web process and web process failure to render web content after cache restore, tracking revert in https://bugs.webkit.org/show_bug.cgi?id=274329.
I wound up using bug #274261 to track the revert. This will need a second try, sorry. :(
Fortunately it looks like this was not backported to 2.44 since nobody requested it.
(In reply to Michael Catanzaro from comment #43) > Fortunately it looks like this was not backported to 2.44 since nobody > requested it. It was backported to 2.44. See https://github.com/WebKit/WebKit/commit/30ad9a720e6b12a6c958fcef0d7dd3f52da485bd
OK, will revert there too. I must have gotten very confused when I checked for the backport....
Found yet another reproducer: visit https://www.msnbc.com/opinion/msnbc-opinion/trump-hush-money-verdict-biden-campaign-reaction-rcna154560 and just scroll down the page