Bug 261306

Summary:	REGRESSION(267516@main): [ macOS ] 25 tests are constant failures following Sonoma OpenSource expectations migration.
Product:	WebKit	Reporter:	Ben Schwartz <ben_schwartz>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	Morningstar, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=259373 https://bugs.webkit.org/show_bug.cgi?id=263982 https://bugs.webkit.org/show_bug.cgi?id=263987 https://bugs.webkit.org/show_bug.cgi?id=267439

Description Ben Schwartz 2023-09-07 17:50:44 PDT

The following tests are consistent/constant failures on macOS Monterey (both wk1 and 2):

http/tests/appcache/fail-on-update-2.html

The following tests are constant failures on macOS Monterey+, wk1:

fast/scrolling/mac/scrollbars/scrollbar-state.html
fast/scrolling/mac/scrollbars/overlay-scrollbar-state.html

The following tests are constant failures on macOS Ventura, wk1:

editing/caret/insert-paragraph-does-not-paint-stale-carets.html
editing/input/editable-container-with-word-wrap-normal.html
editing/inserting/caret-position.html
editing/mac/input/caret-primary-bidi.html
editing/mac/input/firstrectforcharacterrange-plain.html
editing/mac/input/firstrectforcharacterrange-styled.html
editing/pasteboard/interchange-newline-2.html
editing/selection/5825350-2.html
editing/selection/block-cursor-overtype-mode.html
editing/selection/caret-in-empty-inline-1.html
editing/selection/caret-in-empty-inline-2.html
editing/selection/internal-caret-rect.html
editing/selection/mixed-editability-10.html
editing/selection/simple-line-layout-caret-is-gone.html
fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html
imported/w3c/web-platform-tests/mathml/relations/css-styling/display-2.html
media/media-can-play-mpeg4-video.html 
imported/w3c/web-platform-tests/xhr/open-method-case-sensitive.htm
media/video-source-type-params.html

The following tests are constant failures on macOS Ventura (x86_64 only), wk2.

scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html (x86_64 only)

HISTORY:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&suite=layout-tests&test=editing%2Fcaret%2Finsert-paragraph-does-not-paint-stale-carets.html&test=editing%2Finput%2Feditable-container-with-word-wrap-normal.html&test=editing%2Finserting%2Fcaret-position.html&test=editing%2Fmac%2Finput%2Fcaret-primary-bidi.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-plain.html&test=editing%2Fmac%2Finput%2Ffirstrectforcharacterrange-styled.html&test=editing%2Fpasteboard%2Finterchange-newline-2.html&test=editing%2Fselection%2F5825350-2.html&test=editing%2Fselection%2Fblock-cursor-overtype-mode.html&test=editing%2Fselection%2Fcaret-in-empty-inline-1.html&test=editing%2Fselection%2Fcaret-in-empty-inline-2.html&test=editing%2Fselection%2Finternal-caret-rect.html&test=editing%2Fselection%2Fmixed-editability-10.html&test=editing%2Fselection%2Fsimple-line-layout-caret-is-gone.html&test=fast%2Fforms%2Fauto-fill-button%2Finput-credit-card-auto-fill-button.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Foverlay-scrollbar-state.html&test=fast%2Fscrolling%2Fmac%2Fscrollbars%2Fscrollbar-state.html&test=http%2Ftests%2Fappcache%2Ffail-on-update-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fmathml%2Frelations%2Fcss-styling%2Fdisplay-2.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fxhr%2Fopen-method-case-sensitive.htm&test=media%2Fmedia-can-play-mpeg4-video.html&test=media%2Fvideo-source-type-params.html&test=scrollingcoordinator%2Fmac%2Ffixed-backgrounds%2Ffixed-background-in-overflow-in-iframe.html&platform=mac

REPRODUCTION:

All tests can be reproduced on the reported platform by using the following command:

run-webkit-tests -1 --clobber-old-results --verbose --test-list run_tests.txt

...where run_tests.txt contains the affected tests listed above.

REGRESSION:

According to the flakiness dashboard and bisection workflow, these tests have a clear regression point at 267516@main.

Comment 1 Radar WebKit Bug Importer 2023-09-07 17:51:04 PDT

<rdar://problem/115146287>

Comment 2 Ben Schwartz 2023-09-07 18:10:09 PDT

Adding http/tests/appcache/remove-cache.html, which is a consistent/constant failure on Monterey wk1.

Comment 3 EWS 2023-09-07 20:18:08 PDT

Test gardening commit 267764@main (1c1d8e1af449): <https://commits.webkit.org/267764@main>

Reviewed commits have been landed. Closing PR #17569 and removing active labels.

Comment 4 EWS 2023-09-08 09:57:10 PDT

Test gardening commit 267798@main (af7d03822467): <https://commits.webkit.org/267798@main>

Reviewed commits have been landed. Closing PR #17596 and removing active labels.

Comment 5 Ben Schwartz 2023-09-27 14:13:58 PDT

Adding imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html, which is a constant failure on macOS but is missing exceptions for Monterey wk2.

Comment 6 EWS 2023-09-27 14:53:41 PDT

Test gardening commit 268540@main (0564f1d21bab): <https://commits.webkit.org/268540@main>

Reviewed commits have been landed. Closing PR #18309 and removing active labels.

Comment 7 Ryan Haddad 2023-10-20 15:47:52 PDT

It looks like all of these except for scrollingcoordinator/mac/fixed-backgrounds/fixed-background-in-overflow-in-iframe.html were fixed via rebaselines.

http/tests/appcache/fail-on-update-2.html is a flaky timeout, but it appears that we already have at least one bug tracking this.

Before closing this, we need to make sure we have removed any failure expectations added as part of this bug.

Comment 8 Ben Schwartz 2023-10-30 19:14:02 PDT

Issues have been resolved on most tests, but persist on the following.


This test is consistently timing out on macOS:

http/tests/appcache/fail-on-update-2.html
http/tests/appcache/remove-cache.html (Monterey wk1 only: when not timing out, failing)


This test is constantly failing on macOS:

imported/w3c/web-platform-tests/content-security-policy/reporting-api/report-to-directive-allowed-in-meta.https.sub.html


I will remove the expectations for the other tests.

Comment 9 EWS 2023-10-30 19:53:56 PDT

Test gardening commit 269991@main (4b5edf812039): <https://commits.webkit.org/269991@main>

Reviewed commits have been landed. Closing PR #19754 and removing active labels.

Comment 10 Ben Schwartz 2023-10-31 11:45:00 PDT

Filed new bugs for the tests still experiencing issues. Closing, as verified issues with other tests resolved.