Bug 263227

Summary:	REGRESSION(264111@main): imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is a flaky failure
Product:	WebKit	Reporter:	Ben Schwartz <ben_schwartz>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	RESOLVED CONFIGURATION CHANGED
Severity:	Normal	CC:	achristensen, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=256780 https://bugs.webkit.org/show_bug.cgi?id=263060 https://bugs.webkit.org/show_bug.cgi?id=263221

Description Ben Schwartz 2023-10-16 17:23:19 PDT

The following tests are consistently failing on macOS wk2:

imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html
imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html

HISTORY:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fsimulcast%2Fbasic.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fprotocol%2Fhandover-datachannel.html&recent=false&platform=mac

TEXT DIFFs:

-PASS Basic simulcast setup with two spatial layers
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Basic simulcast setup with two spatial layers Test timed out
+
_________

-PASS Handover with datachannel reinitiated from new callee completes
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Handover with datachannel reinitiated from new callee completes Test timed out
+

DIFF URLs:

https://ews-build.s3-us-west-2.amazonaws.com/macOS-Monterey-Release-WK2-Tests-EWS/d3496a07-20520/imported/w3c/web-platform-tests/webrtc/simulcast/basic.https-pretty-diff.html
https://ews-build.s3-us-west-2.amazonaws.com/macOS-Monterey-Release-WK2-Tests-EWS/d3496a07-20520/imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel-pretty-diff.html

REPRODUCIBILITY:

I was able to reproduce this issue on macOS Ventura (13.6, 22G120/arm64) ToT using the following command:

run-webkit-tests --debug --iterations 50 --child-processes 5 -f imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html

REGRESSION:

I was able to identify a potential initial regression point at 264111@main; however, it appears that the failures became more frequent at 269302@main.

Comment 1 Radar WebKit Bug Importer 2023-10-16 17:24:00 PDT

<rdar://problem/117045403>

Comment 2 EWS 2023-10-16 17:35:43 PDT

Test gardening commit 269391@main (c193b95a25cf): <https://commits.webkit.org/269391@main>

Reviewed commits have been landed. Closing PR #19141 and removing active labels.

Comment 3 Ryan Haddad 2023-12-12 12:42:23 PST

It looks like imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is still a flaky failure, but imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html appears to be passing now.

Comment 4 youenn fablet 2024-03-11 06:01:19 PDT

imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is flaky on Monterey x86_64 only.
imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html is no longer flaky.

Updating test expectations in https://bugs.webkit.org/show_bug.cgi?id=270791.

I filed https://bugs.webkit.org/show_bug.cgi?id=270790 to continue the investigation since the same bug appears in several tests.