Bug 263227 - REGRESSION(264111@main): imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is a flaky failure
Summary: REGRESSION(264111@main): imported/w3c/web-platform-tests/webrtc/simulcast/bas...
Status: RESOLVED CONFIGURATION CHANGED
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-10-16 17:23 PDT by Ben Schwartz
Modified: 2024-03-11 06:01 PDT (History)
4 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Ben Schwartz 2023-10-16 17:23:19 PDT 
The following tests are consistently failing on macOS wk2:

imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html
imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html

HISTORY:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fsimulcast%2Fbasic.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fwebrtc%2Fprotocol%2Fhandover-datachannel.html&recent=false&platform=mac

TEXT DIFFs:

-PASS Basic simulcast setup with two spatial layers
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Basic simulcast setup with two spatial layers Test timed out
+
_________

-PASS Handover with datachannel reinitiated from new callee completes
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Handover with datachannel reinitiated from new callee completes Test timed out
+

DIFF URLs:

https://ews-build.s3-us-west-2.amazonaws.com/macOS-Monterey-Release-WK2-Tests-EWS/d3496a07-20520/imported/w3c/web-platform-tests/webrtc/simulcast/basic.https-pretty-diff.html
https://ews-build.s3-us-west-2.amazonaws.com/macOS-Monterey-Release-WK2-Tests-EWS/d3496a07-20520/imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel-pretty-diff.html

REPRODUCIBILITY:

I was able to reproduce this issue on macOS Ventura (13.6, 22G120/arm64) ToT using the following command:

run-webkit-tests --debug --iterations 50 --child-processes 5 -f imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html

REGRESSION:

I was able to identify a potential initial regression point at 264111@main; however, it appears that the failures became more frequent at 269302@main.
Comment 1 Radar WebKit Bug Importer 2023-10-16 17:24:00 PDT 
<rdar://problem/117045403>
Comment 2 EWS 2023-10-16 17:35:43 PDT 
Test gardening commit 269391@main (c193b95a25cf): <https://commits.webkit.org/269391@main>

Reviewed commits have been landed. Closing PR #19141 and removing active labels.
Comment 3 Ryan Haddad 2023-12-12 12:42:23 PST 
It looks like imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is still a flaky failure, but imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html appears to be passing now.
Comment 4 youenn fablet 2024-03-11 06:01:19 PDT 
imported/w3c/web-platform-tests/webrtc/simulcast/basic.https.html is flaky on Monterey x86_64 only.
imported/w3c/web-platform-tests/webrtc/protocol/handover-datachannel.html is no longer flaky.

Updating test expectations in https://bugs.webkit.org/show_bug.cgi?id=270791.

I filed https://bugs.webkit.org/show_bug.cgi?id=270790 to continue the investigation since the same bug appears in several tests.