Bug 260654

*** Bug 260456 has been marked as a duplicate of this bug. ***

Based on the discussion from https://bugs.webkit.org/show_bug.cgi?id=261874 at least part of the problem is that current versions of the NVIDIA driver do not support allocating GBM buffers with the R8 format. However, the next release (550) will support this. So it may be worth re-testing when that goes public later this year.

Created attachment 470157 [details]
Videos are now green

Erik, I have retested this with the 550.54.14 driver and videos became green instead

It would be interesting to know which format the video decoder outputs and which format the video sink negotiates... Can you share pipeline dumps?

How can I get that information?

https://docs.webkit.org/Ports/WebKitGTK%20and%20WPE%20WebKit/Multimedia.html

Created attachment 470179 [details]
Compressed dots folder

Thanks. Here's the compressed dot directory after running the command with youtube.com and playing a few videos there.

So the decoder outputs raw NV12 and for rendering we convert that to DMABufs within WebKit. That might be where the problem is.

> With WEBKIT_GST_DMABUF_SINK_DISABLED=1, sites don't crash but videos are black or refuse to be played.

Is this still happening?

Currently, with WEBKIT_GST_DMABUF_SINK_DISABLED=1 videos aren't black but simply don't play. Without this environment variable, videos seemingly can play, but they are green.

Erik, this issue still persists with Nvidia driver 550. Though GBM buffers with R8 formats are now supported, there remains a problem with converting raw NV12 to DMABufs. Nvidia users cannot view any video in WebKitGTK apps until this is resolved.

> there remains a problem with converting raw NV12 to DMABufs

Could you clarify what you mean by this? With the 550 driver it should be possible to allocate an NV12 GBM buffer and export it as a pair of DMA-BUFs. Note, though, that support for this format in our EGL driver is "external only". So if an NV12 buffer is imported as an EGLImage, that image can only be bound to GL_TEXTURE_EXTERNAL_OES (not, for example, GL_TEXTURE_2D). Could that maybe be the problem?

I used flatpak run --command=/bin/bash com.github.kdwk.Spidey to enter the Flatpak runtime then gst-play-1.0 to play local video files. This works fine. 

WebKitGTK 2.45.4/ GStreamer 1.24/

(In reply to Kdwk from comment #10)
> Currently, with WEBKIT_GST_DMABUF_SINK_DISABLED=1 videos aren't black but
> simply don't play. Without this environment variable, videos seemingly can
> play, but they are green.

If that still happens we'd need gst logs.

Created attachment 471804 [details]
Compressed dots folder (DMABuf renderer)

I have the new dots folder when playing videos with the DMABuf renderer. It now shows a black screen instead of green after the update of Nvidia drivers from 550 to 555. Sound works as usual (playback works except video).

That's not with nvidia decoder, but openh264... try with GST_PLUGIN_FEATURE_RANK=openh264dec:0 ?

Sorry i was looking at the wrong pipeline :)
What's the output of gst-inspect-1.0 nvvp9dec ?

Having pipeline dump for gst-play would be nice too.

Created attachment 471805 [details]
gst-inspect-1.0 nvvp9dec.txt

Here's the output of gst-inspect-1.0 nvvp9dec

GST_DEBUG="nv*:6,videodec*:6" logs too please

(In reply to Philippe Normand from comment #8)
> So the decoder outputs raw NV12 and for rendering we convert that to DMABufs
> within WebKit. That might be where the problem is.

Still same issue. Once we understand why the decoder negotiates raw NV12 instead of GLMemories with the sink, the issue will be closer to resolution.

(In reply to Philippe Normand from comment #17)

> Having pipeline dump for gst-play would be nice too.

I am unable to get this dump because it keeps insisting the files and directories don't exist.

However, gst-play-1.0 keep complaining about 

WARNING A lot of buffers are being dropped.
WARNING debug information: ../libs/gst/base/gstbasesink.c(3147): gst_base_sink_is_too_late (): /GstPlayBin3:playbin/GstPlaySink:playsink/GstBin:vbin/GstAutoVideoSink:videosink/GstGLImageSinkBin:videosink-actual-sink-glimage/GstGLImageSink:sink:
There may be a timestamping problem, or this computer is too slow.

(In reply to Kdwk from comment #21)
> (In reply to Philippe Normand from comment #17)
> 
> > Having pipeline dump for gst-play would be nice too.
> 
> I am unable to get this dump because it keeps insisting the files and
> directories don't exist.

Are you setting the dangerous env var? See https://docs.webkit.org/Ports/WebKitGTK%20and%20WPE%20WebKit/Multimedia.html#flatpak-apps

(In reply to Philippe Normand from comment #20)
> (In reply to Philippe Normand from comment #8)
> > So the decoder outputs raw NV12 and for rendering we convert that to DMABufs
> > within WebKit. That might be where the problem is.
> 
> Still same issue. Once we understand why the decoder negotiates raw NV12
> instead of GLMemories with the sink, the issue will be closer to resolution.

Because we always claim to support NV12?

Yes...

There's two different issues I think.

- with our GL sink we should ideally behave almost the same as gst-play (with glimagesink) and negotiate GLMemories all the way from decoder to sink.

- with our DMABuf sink I'm not sure how to proceed, but video/x-raw negotiation should be avoided... we'd need a dmabufupload (or maybe glupload can handle GLMemory conversion to DMABuf now) that would convert CUDA or GLMemories from the decoder to DMABufs.

(In reply to Philippe Normand from comment #25)
> Yes...
> 
> There's two different issues I think.
> 
> - with our GL sink we should ideally behave almost the same as gst-play
> (with glimagesink) and negotiate GLMemories all the way from decoder to sink.

If you know how to do that...

> - with our DMABuf sink I'm not sure how to proceed, but video/x-raw
> negotiation should be avoided... we'd need a dmabufupload (or maybe glupload
> can handle GLMemory conversion to DMABuf now) that would convert CUDA or
> GLMemories from the decoder to DMABufs.

This is how I think the DMABuf sink works:

We claim to support a fixed list of formats, which are the ones that can be handled by the texture mapper. Then two things can happen:

 1) gst can provide one of the formats as DMABuf: In this case we just import the DMABUf as textures and render them into the texture mapper.

 2) gst can't provide the formats as DMABuf: in this case we receive a raw video frame in one the formats we claimed in caps. We allocate a GBM buffer and upload the video frame there, then we do as if the DMABuf was provided by gst and import it as textures to render them into the texture mapper.

So, in both cases, I wonder what happens if the buffer format (and modifiers) is not supported by the driver. I guess in the first case we will fail to import the DMABuf and in the second case we will fail to allocate the buffer. I guess we should claim to support only the formats supported by the driver, the same way we do in the renderer. Do we know where exactly is failing?

And in case of not getting a DMABuf from gst, is it really more efficient to allocate, upload and then import the DMABuf? Can't we just fallback to GL or non-accelerated video?

Having the decoder output raw video (in main memory, not GPU) should be avoided, otherwise CPU usage might increase significantly...

Our hand-rolled GBM to DMABuf upload in the player is meant for software decoders...

@Kdwk: Can you get gst logs please? `GST_DEBUG="3,nv*:6`

Also for testing purpose can you disable the WebProcess sandbox? I suspect it might block access from the nv decoder to the cuda device...