Bug 262595

Summary:	REGRESSION(Sonoma): [ Sonoma ] 2 lookup menu-related tests are constant text failures.
Product:	WebKit	Reporter:	Ben Schwartz <ben_schwartz>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	jenner, thorton, webkit-bot-watchers-bugzilla, webkit-bug-importer, wenson_hsieh
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=262489

Description Ben Schwartz 2023-10-03 18:11:28 PDT

The following tests are constantly failing with text diffs on macOS Sonoma:

editing/mac/dictionary-lookup/dictionary-lookup.html
editing/selection/context-menu-text-selection-lookup.html

HISTORY:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=editing%2Fmac%2Fdictionary-lookup%2Fdictionary-lookup.html&test=editing%2Fselection%2Fcontext-menu-text-selection-lookup.html

TEXT DIFFs:


 Some text to look up.
 
 Lookup string for normal text: 'look'.
-Lookup string for title-case phrase: 'New York'.
+Lookup string for title-case phrase: 'New'.
 Lookup string for unselectable text: 'null'.
_____

 This test checks that triggering the context menu selects/doesn't select using surround text as context (as platform-appropriate).
 
 New York
+FAILURE: We expected New York, but we got New.
 SUCCESS
 SUCCESS
-SUCCESS


DIFF URLs:

https://build.webkit.org/results/Apple-Sonoma-Release-WK1-Tests/268810@main%20(91)/editing/mac/dictionary-lookup/dictionary-lookup-pretty-diff.html
https://build.webkit.org/results/Apple-Sonoma-Release-WK1-Tests/268810@main%20(91)/editing/selection/context-menu-text-selection-lookup-pretty-diff.html

REPRODUCIBILITY:

I was able to reproduce this issue on macOS Sonoma (14.0, 23A344/arm64) ToT using the following command:

run-webkit-tests --verbose --clobber-old-results editing/mac/dictionary-lookup/dictionary-lookup.html editing/selection/context-menu-text-selection-lookup.html

REGRESSION:

This test regressed with the bring-up of Sonoma testers (see bug #262489).

NOTES:

This seems to be an issue with how tests on Sonoma are run when the menus are open, which could be an OS regression.

Comment 1 Radar WebKit Bug Importer 2023-10-03 18:11:52 PDT

<rdar://problem/116437531>

Comment 2 EWS 2023-10-03 19:36:21 PDT

Test gardening commit 268820@main (f399a5ee439d): <https://commits.webkit.org/268820@main>

Reviewed commits have been landed. Closing PR #18602 and removing active labels.

Comment 3 EWS 2024-07-02 13:26:31 PDT

Test gardening commit 280590@main (e604d0614fb7): <https://commits.webkit.org/280590@main>

Reviewed commits have been landed. Closing PR #30403 and removing active labels.

Comment 4 Robert Jenner 2024-07-02 14:58:59 PDT

(In reply to EWS from comment #3)
> Test gardening commit 280590@main (e604d0614fb7):
> <https://commits.webkit.org/280590@main>
> 
> Reviewed commits have been landed. Closing PR #30403 and removing active
> labels.

I modified the existing expectations because these tests haven't been fixed, and it appears to impact both Sonoma and Sequoia.