Bug 254706

Summary:	[ iOS, macOS ] imported/w3c/web-platform-tests/html/dom/idlharness.https.html near constant failure.
Product:	WebKit	Reporter:	Bri Harris <bharris9>
Component:	New Bugs	Assignee:	youenn fablet <youennf>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	ap, cdumez, webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority:	P2	Keywords:	InRadar
Version:	Other
Hardware:	Unspecified
OS:	Unspecified

Description Bri Harris 2023-03-29 17:02:00 PDT

imported/w3c/web-platform-tests/html/dom/idlharness.https.html

is a near constant failure. It is failing on iOS 16 and is a flaky time out / failure on macOS Debug.

HISTORY:

https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fhtml%2Fdom%2Fidlharness.https.html%3Fexclude%3D(Document%7CWindow%7CHTML.*)


DIFF TEXT:
@@ -1275,9 +1275,9 @@
 PASS Worklet interface object length
 PASS Worklet interface object name
 PASS Worklet interface: existence and properties of interface prototype object
-FAIL Worklet interface: existence and properties of interface prototype object's "constructor" property assert_own_property: Worklet.prototype does not have own property "constructor" expected property "constructor" missing
+PASS Worklet interface: existence and properties of interface prototype object's "constructor" property
 PASS Worklet interface: existence and properties of interface prototype object's @@unscopables property
-FAIL Worklet interface: operation addModule(USVString, optional WorkletOptions) assert_own_property: interface prototype object missing non-static operation expected property "addModule" missing
+PASS Worklet interface: operation addModule(USVString, optional WorkletOptions)
 PASS Storage interface: existence and properties of interface object
 PASS Storage interface object length
 PASS Storage interface object name

Comment 1 Bri Harris 2023-03-29 17:02:33 PDT

I was able to replicate this failure on iOS 16 ToT as follows:

run-webkit-tests --iOS-simulator imported/w3c/web-platform-tests/html/dom/idlharness.https.html

Comment 2 Radar WebKit Bug Importer 2023-03-29 17:03:12 PDT

<rdar://problem/107393338>

Comment 3 Alexey Proskuryakov 2023-04-02 19:22:52 PDT

There are many problems with this test, but AFAICT the diff posted above just means that iOS has incorrect expectations for it.

Other problems that I see is that the test is super slow in debug, frequently timing out, and diffs of at least one more kind:

-PASS MediaError must be primary interface of errorVideo.error
-PASS Stringification of errorVideo.error
-PASS MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_ABORTED" with the proper type
-PASS MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_NETWORK" with the proper type
-PASS MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_DECODE" with the proper type
-PASS MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_SRC_NOT_SUPPORTED" with the proper type
-PASS MediaError interface: errorVideo.error must inherit property "code" with the proper type
-PASS MediaError interface: errorVideo.error must inherit property "message" with the proper type
+FAIL Stringification of errorVideo.error assert_class_string: class string of errorVideo.error expected "[object MediaError]" but got "[object Null]"
+FAIL MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_ABORTED" with the proper type assert_inherits: provided value is not an object
+FAIL MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_NETWORK" with the proper type assert_inherits: provided value is not an object
+FAIL MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_DECODE" with the proper type assert_inherits: provided value is not an object
+FAIL MediaError interface: errorVideo.error must inherit property "MEDIA_ERR_SRC_NOT_SUPPORTED" with the proper type assert_inherits: provided value is not an object
+FAIL MediaError interface: errorVideo.error must inherit property "code" with the proper type assert_inherits: provided value is not an object
+FAIL MediaError interface: errorVideo.error must inherit property "message" with the proper type assert_inherits: provided value is not an object

Comment 4 EWS 2023-04-04 15:46:19 PDT

Test gardening commit 262595@main (4798b5317d9d): <https://commits.webkit.org/262595@main>

Reviewed commits have been landed. Closing PR #12396 and removing active labels.

Comment 5 EWS 2023-04-07 21:10:53 PDT

Test gardening commit 259548.602@safari-7615-branch (1fd08d3625dd): <https://commits.webkit.org/259548.602@safari-7615-branch>

Reviewed commits have been landed. Closing PR #537 and removing active labels.

Comment 6 youenn fablet 2023-06-06 10:36:14 PDT

Looking at the test results, on iOS it might require a specific expectation for iPad vs. iPhone due to MSE support.

For Debug in general, idlharness test should be marked as slow.

Comment 7 youenn fablet 2023-10-17 06:20:15 PDT

Pull request: https://github.com/WebKit/WebKit/pull/19164

Comment 8 EWS 2023-10-18 03:49:30 PDT

Committed 269461@main (2e494e5298f7): <https://commits.webkit.org/269461@main>

Reviewed commits have been landed. Closing PR #19164 and removing active labels.