Bug 251771

Summary: Overlapping VTTCues carried by InbandTextTrack (LegibleOutput) does not handle correctly
Product: WebKit Reporter: Taichiro Ishikawa <taichiro.ishikawa>
Component: MediaAssignee: Eric Carlson <eric.carlson>
Status: RESOLVED FIXED    
Severity: Normal CC: eric.carlson, jer.noble, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Safari 15   
Hardware: All   
OS: All   
Attachments:
Description Flags
Sample Stream for Multiple Cue
none
patch for handle all vttc atom in nativeSamples
none
Sample Stream for Multiple Cue (5 cue overlapping) none

Taichiro Ishikawa
Reported 2023-02-05 19:36:01 PST
Created attachment 464857 [details] Sample Stream for Multiple Cue Background: LegibleOutput sends multiple vttc atom (VTTCue in isobmff) in one nativeSample. (nativeSamples length = 1, but multiple vttc in the nativeSample) But, InbandTextTrackPrivateAVF::processNativeSamples handles only first vttc each callback and rest vttc still in buffer. So, rest cues are not handles in same callback and process in next callback and use next timing information, that wrongly. Issue: Overlapping Timing VTTCues in InbandTextTrack does not handle correctly Reproduction steps: Play m3u8 in attachmend zipfile. Actual behavior: VTTCue[0]: startTime=0, endTIme=2 VTTCue[1]: startTime=0.5786666666666667, endTime=0.5786666666666667 Expected behavior: VTTCue[0]: startTime=0, endTIme=2 VTTCue[1]: startTime=0, endTime=2 Impact: Video Streaming Service not capable of using vertical splitted (top and bottom) cues.
Attachments
Sample Stream for Multiple Cue (196.57 KB, application/zip)
2023-02-05 19:36 PST, Taichiro Ishikawa 		no flags
Details
patch for handle all vttc atom in nativeSamples (5.37 KB, patch)
2023-02-05 19:49 PST, Taichiro Ishikawa 		no flags
DetailsFormatted DiffDiff
Sample Stream for Multiple Cue (5 cue overlapping) (196.71 KB, application/zip)
2023-02-14 15:37 PST, Taichiro Ishikawa 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Taichiro Ishikawa
Comment 1 2023-02-05 19:49:41 PST
Created attachment 464859 [details] patch for handle all vttc atom in nativeSamples I wrote a patch for handle all vttc atom in InbandTextTrackPrivateAVF::processNativeSamples (Simply, add while loop) It is an example, might be memory issue. This patch works as expected for me.
Taichiro Ishikawa
Comment 2 2023-02-06 16:14:00 PST
I've hosted same sample stream on https://taichiro-ishikawa-play.github.io/hls-sample-stream/multiple-cue-for-webkit/master.m3u8 Please check startTime and endTime for each cue.
Taichiro Ishikawa
Comment 3 2023-02-08 23:37:59 PST
I also raised feedback in Feedback Assistant. (https://feedbackassistant.apple.com/feedback/11985197). And, add another sample stream. describe current webkit behavior below. https://taichiro-ishikawa-play.github.io/hls-sample-stream/multiple-cue-for-webkit2/master.m3u8 Original VTT Content in sample HLS VTT Fragment: VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2 VTTCue[1]: text="line 2 (1)", startTime=0, endTime=2 VTTCue[2]: text="line 1 (2)", startTime=2, endTime=4 VTTCue[3]: text="line 2 (3)", startTime=2.5, endTime=3.5 VTTCue[4]: text="line 3 (4)", startTime=3, endTime=4 Current webkit HLS TextTrack VTTCue result: VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2 VTTCue[1]: text="line 2 (1)", startTime=1.9996666666666667, endTime=2.4996666666666667 VTTCue[2]: text="line 1 (2)", startTime=2.4996666666666667, endTime=2.9996666666666667 VTTCue[3]: text="line 1 (2)", startTime=2.9996666666666667, endTime=3.4996666666666667 VTTCue[4]: text="line 2 (3)", startTime=3.4996666666666667, endTime=3.9996666666666667 VTTCue[5]: text="line 1 (2)", startTime=3.9996666666666667, endTime=3.9996666666666667 It is not same as Original VTT Content. Because InbandTextTrackPrivateAVF::processNativeSamples handles only first vttc atom in legibleoutput callback. Patched webkit HLS TextTrack VTTCue result: VTTCue[0]: text="line 1 (0)", startTime=0, endTime=2 VTTCue[1]: text="line 2 (1)", startTime=0, endTime=2 VTTCue[2]: text="line 1 (2)", startTime=1.9996666666666667, endTime=2.4996666666666667 VTTCue[3]: text="line 1 (2)", startTime=2.4996666666666667, endTime=2.9996666666666667 VTTCue[4]: text="line 2 (3)", startTime=2.4996666666666667, endTime=2.9996666666666667 VTTCue[5]: text="line 1 (2)", startTime=2.9996666666666667, endTime=3.4996666666666667 VTTCue[6]: text="line 2 (3)", startTime=2.9996666666666667, endTime=3.4996666666666667 VTTCue[7]: text="line 3 (4)", startTime=2.9996666666666667, endTime=3.4996666666666667 VTTCue[8]: text="line 1 (2)", startTime=3.4996666666666667, endTime=3.9996666666666667 VTTCue[9]: text="line 3 (4)", startTime=3.4996666666666667, endTime=3.9996666666666667 It is fragmented, so concatenate, it same as Original VTT Content.
Radar WebKit Bug Importer
Comment 4 2023-02-12 19:36:32 PST
<rdar://problem/105372930>
Taichiro Ishikawa
Comment 5 2023-02-14 15:37:51 PST
Created attachment 464991 [details] Sample Stream for Multiple Cue (5 cue overlapping) I attach another sample stream in zip (same as above m3u8). This incorrect behavior since over 5 years ago. https://developer.apple.com/forums/thread/75821 I hope to fix this incorrect behavior and wish to use separate positioning cue in webkit.
Taichiro Ishikawa
Comment 6 2023-03-28 13:18:04 PDT
This issue is still reproducible on Version 16.4. How is the progress?
Eric Carlson
Comment 7 2023-07-21 06:33:02 PDT
Pull request: https://github.com/WebKit/WebKit/pull/15982
EWS
Comment 8 2023-07-24 11:46:11 PDT
Committed 266260@main (6f89f7891c5c): <https://commits.webkit.org/266260@main> Reviewed commits have been landed. Closing PR #15982 and removing active labels.
Note You need to log in before you can comment on or make changes to this bug.