We are seeing constant failures/crashes/timeouts on macOS Sonoma and iOS 17 following tester bring-up last week. We are looking to file more specific bugs for each; however, this bug serves as a tracking issue and allows us to batch set test expectations while the issues are being investigated. The following tests are constant text failures on both macOS Sonoma (wk1 and wk2) and iOS 17: fast/block/margin-collapse/103.html fast/css/rtl-ordering.html fast/forms/input-appearance-spinbutton.html fast/forms/input-baseline.html fast/forms/listbox-bidi-align.html fast/forms/placeholder-position.html fast/forms/search/search-size-with-decorations.html fast/forms/select-visual-hebrew.html fast/forms/time/time-input-rendering-basic.html fast/text/drawBidiText.html fast/text/glyph-display-lists/glyph-display-list-colr-unshared.html fast/text/international/bidi-listbox.html fast/text/international/bidi-menulist.html fast/text/international/pop-up-button-text-alignment-and-direction.html tables/mozilla/bugs/bug2479-2.html The following tests are constant text failures on macOS Sonoma (wk1 and wk2) only: editing/input/caret-at-the-edge-of-input.html editing/inserting/before-after-input-element.html editing/inserting/multiple-lines-selected.html editing/mac/dictionary-lookup/dictionary-lookup.html editing/pasteboard/4806874.html editing/pasteboard/input-field-1.html editing/selection/3690703-2.html editing/selection/3690703.html editing/selection/3690719.html editing/selection/4895428-3.html editing/selection/4975120.html editing/selection/context-menu-text-selection-lookup.html editing/selection/drag-select-1.html editing/selection/select-from-textfield-outwards.html fast/css/line-height.html fast/forms/auto-fill-button/hide-auto-fill-strong-password-viewable-treatment-when-form-is-reset.html fast/forms/auto-fill-button/input-auto-fill-button.html fast/forms/auto-fill-button/input-contacts-auto-fill-button.html fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html fast/forms/auto-fill-button/input-strong-password-viewable.html fast/forms/basic-inputs.html fast/forms/box-shadow-override.html fast/forms/button-sizes.html fast/forms/control-restrict-line-height.html fast/forms/datalist/datalist-textinput-appearance.html fast/forms/encoding-test.html fast/forms/fieldset-align.html fast/forms/form-element-geometry.html fast/forms/input-align.html fast/forms/input-appearance-bkcolor.html fast/forms/input-appearance-default-bkcolor.html fast/forms/input-appearance-focus.html fast/forms/input-appearance-height.html fast/forms/input-appearance-preventDefault.html fast/forms/input-appearance-selection.html fast/forms/input-appearance-spinbutton-up.html fast/forms/input-appearance-visibility.html fast/forms/input-appearance-width.html fast/forms/input-disabled-color.html fast/forms/input-double-click-selection-gap-bug.html fast/forms/input-placeholder-visibility-1.html fast/forms/input-placeholder-visibility-3.html fast/forms/input-spaces.html fast/forms/input-table.html fast/forms/input-text-click-inside.html fast/forms/input-text-click-outside.html fast/forms/input-text-double-click.html fast/forms/input-text-drag-down.html fast/forms/input-text-option-delete.html fast/forms/input-text-scroll-left-on-blur.html fast/forms/input-text-self-emptying-click.html fast/forms/input-text-word-wrap.html fast/forms/input-type-text-min-width.html fast/forms/input-value.html fast/forms/input-width.html fast/forms/minWidthPercent.html fast/forms/number/number-appearance-rtl.html fast/forms/number/number-appearance-spinbutton-disabled-readonly.html fast/forms/number/number-appearance-spinbutton-layer.html fast/forms/placeholder-pseudo-style.html fast/forms/search-cancel-button-style-sharing.html fast/forms/search-display-none-cancel-button.html fast/forms/search-input-rtl.html fast/forms/search-rtl.html fast/forms/search-styled.html fast/forms/search-vertical-alignment.html fast/forms/search/search-padding-cancel-results-buttons.html fast/forms/select-writing-direction-natural.html fast/forms/tabbing-input-iframe.html fast/forms/text-control-intrinsic-widths.html fast/forms/textfield-focus-ring.html fast/forms/textfield-outline.html fast/forms/textfield-overflow.html fast/forms/visual-hebrew-text-field.html fast/frames/take-focus-from-iframe.html fast/html/details-no-summary4.html fast/html/details-open-javascript.html fast/html/details-open2.html fast/html/details-open4.html fast/lists/dynamic-marker-crash.html fast/repaint/placeholder-after-caps-lock-hidden.html fast/repaint/renderer-destruction-by-invalidateSelection-crash.html fast/repaint/subtree-root-skipped.html fast/replaced/replaced-breaking-mixture.html fast/table/003.html fast/table/colspanMinWidth.html fast/table/spanOverlapRepaint.html fast/table/text-field-baseline.html fast/text/textIteratorNilRenderer.html fast/transforms/transformed-focused-text-input.html http/tests/navigation/javascriptlink-frames.html imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html imported/w3c/web-platform-tests/mathml/presentation-markup/mrow/mrow-fallback.html imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-002.html imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-003.html imported/w3c/web-platform-tests/mathml/relations/css-styling/floats/not-floating-001.html imported/w3c/web-platform-tests/mathml/relations/css-styling/ignored-properties-001.html imported/w3c/web-platform-tests/mathml/relations/css-styling/width-height-001.html svg/custom/inline-svg-in-xhtml.xml svg/hixie/mixed/003.xml tables/mozilla/bugs/bug1188.html tables/mozilla/bugs/bug12384.html tables/mozilla/bugs/bug18359.html tables/mozilla/bugs/bug24200.html tables/mozilla/bugs/bug2479-3.html tables/mozilla/bugs/bug2479-4.html tables/mozilla/bugs/bug28928.html tables/mozilla/bugs/bug4382.html tables/mozilla/bugs/bug4527.html tables/mozilla/bugs/bug46368-1.html tables/mozilla/bugs/bug46368-2.html tables/mozilla/bugs/bug51037.html tables/mozilla/bugs/bug55545.html tables/mozilla/bugs/bug59354.html tables/mozilla/bugs/bug7342.html tables/mozilla/bugs/bug96334.html tables/mozilla/bugs/bug99948.html tables/mozilla/dom/tableDom.html tables/mozilla/other/move_row.html tables/mozilla_expected_failures/bugs/bug2479-5.html tables/mozilla_expected_failures/bugs/bug92647-1.html transforms/3d/general/perspective-non-layer.html The following tests are constantly crashing on macOS Sonoma (wk1 and wk2) and iOS 17: imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.worker.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.html imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.worker.html http/wpt/mediarecorder/MediaRecorder-dataavailable.html http/wpt/mediarecorder/MediaRecorder-multiple-start-stop.html http/wpt/mediarecorder/MediaRecorder-requestData.html http/wpt/mediarecorder/pause-recording.html http/wpt/mediarecorder/record-96KHz-sources.html The following tests are constant text failures on macOS Sonoma (wk1 ONLY): editing/mac/attributed-string/anchor-element.html editing/mac/attributed-string/attrib-string-colors-with-color-filter.html editing/mac/attributed-string/attribute-string-for-copy-with-color-filter.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-1.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-2.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-3.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-4.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-5.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-with-style-1.html editing/mac/attributed-string/attributed-string-across-shadow-boundaries-with-style-2.html editing/mac/attributed-string/attributed-string-for-typing-with-color-filter.html editing/mac/attributed-string/attributed-string-for-typing.html editing/mac/attributed-string/basic.html editing/mac/attributed-string/comment-cdata-section.html editing/mac/attributed-string/font-size.html editing/mac/attributed-string/font-style-variant-effect.html editing/mac/attributed-string/font-weight.html editing/mac/attributed-string/letter-spacing.html editing/mac/attributed-string/text-decorations.html editing/mac/attributed-string/vertical-align.html The following test is a constant ImageOnlyFailure on macOS Sonoma (wk1 and wk2): fast/events/context-no-deselect.html The following test is constantly timing out on macOS Sonoma (wk1 and wk2): http/tests/media/video-webm-stall.html The following test, previously a constant text failure on both macOS Sonoma and iOS 17, is still consistently timing out on macOS x86_64 (wk1 and wk2) and flakily timing out on iOS 17 after rebaseline at 268612@main: http/tests/appcache/fail-on-update-2.html
<rdar://problem/116353628>
Correction: http/tests/appcache/fail-on-update-2.html is timing out on all versions of macOS and iOS, not just iOS17/Sonoma.
It appears the reported tests had these expectations set previously: LayoutTests/platform/ios-device/TestExpectations:27:fast/forms/input-text-word-wrap.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:28:fast/forms/search-rtl.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:29:fast/forms/search-vertical-alignment.html [ Failure ] LayoutTests/platform/ios-device/TestExpectations:31:fast/forms/visual-hebrew-text-field.html [ Failure ] LayoutTests/platform/ios-simulator-wk2/TestExpectations:169:tables/mozilla/bugs/bug2479-2.html [ Failure Pass ] LayoutTests/platform/ios-simulator/TestExpectations:14:fast/forms/input-text-word-wrap.html [ Skip ] LayoutTests/platform/ios-simulator/TestExpectations:15:fast/forms/search-vertical-alignment.html [ Skip ] LayoutTests/platform/ios-simulator/TestExpectations:17:fast/forms/visual-hebrew-text-field.html [ Skip ] LayoutTests/platform/ios-wk2/TestExpectations:1060:fast/forms/input-appearance-spinbutton.html [ Failure ] LayoutTests/platform/ios-wk2/TestExpectations:1107:webkit.org/b/225665 http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Pass ] LayoutTests/platform/ios-wk2/TestExpectations:2355:webkit.org/b/252866 [ Debug ] fast/text/glyph-display-lists/glyph-display-list-colr-unshared.html [ Pass Failure ] LayoutTests/platform/ios-wk2/TestExpectations:451:http/tests/navigation/javascriptlink-frames.html LayoutTests/platform/ios-wk2/TestExpectations:49:fast/repaint/placeholder-after-caps-lock-hidden.html [ Pass ] LayoutTests/platform/ios-wk2/TestExpectations:842:fast/text/drawBidiText.html [ Failure Pass ] LayoutTests/platform/ios/TestExpectations:1666:fast/forms/input-baseline.html [ Failure ] LayoutTests/platform/ios/TestExpectations:1678:fast/forms/placeholder-position.html [ Failure ] LayoutTests/platform/ios/TestExpectations:1681:fast/forms/search/search-size-with-decorations.html [ Failure ] LayoutTests/platform/ios/TestExpectations:2608:fast/table/003.html [ Skip ] LayoutTests/platform/ios/TestExpectations:3356:http/wpt/mediarecorder/pause-recording.html [ Pass Crash ] LayoutTests/platform/ios/TestExpectations:3400:http/tests/media/video-webm-stall.html [ Skip ] LayoutTests/platform/ios/TestExpectations:419:editing/selection/context-menu-text-selection-lookup.html [ Skip ] LayoutTests/platform/ios/TestExpectations:4527:fast/repaint/placeholder-after-caps-lock-hidden.html [ Timeout ] LayoutTests/platform/ios/TestExpectations:4535:fast/css/rtl-ordering.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4538:fast/forms/basic-inputs.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4539:fast/forms/input-appearance-height.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4540:fast/forms/input-value.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4541:fast/forms/listbox-bidi-align.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4542:fast/forms/placeholder-pseudo-style.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4543:fast/forms/select-visual-hebrew.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4544:fast/forms/time/time-input-rendering-basic.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4564:imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html [ Failure ] LayoutTests/platform/ios/TestExpectations:4575:tables/mozilla_expected_failures/bugs/bug92647-1.html [ Failure ] LayoutTests/platform/ios/TestExpectations:642:fast/forms/input-appearance-spinbutton-up.html [ Skip ] LayoutTests/platform/ios/TestExpectations:683:fast/forms/search-rtl.html [ Skip ] LayoutTests/platform/ios/TestExpectations:780:fast/text/textIteratorNilRenderer.html [ Skip ] LayoutTests/platform/mac-monterey/TestExpectations:9:http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Failure ] LayoutTests/platform/mac-ventura-wk2-lbse-text/TestExpectations:251:svg/custom/inline-svg-in-xhtml.xml [ Failure ] LayoutTests/platform/mac-ventura-wk2/TestExpectations:24:imported/w3c/web-platform-tests/fetch/range/non-matching-range-response.html [ Pass Failure ] LayoutTests/platform/mac-ventura/TestExpectations:16:fast/forms/auto-fill-button/input-credit-card-auto-fill-button.html [ Failure ] LayoutTests/platform/mac-ventura/TestExpectations:21:http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Timeout Failure ] LayoutTests/platform/mac-wk1/TestExpectations:1004:http/tests/media/video-webm-stall.html LayoutTests/platform/mac-wk2/TestExpectations:125:fast/repaint/placeholder-after-caps-lock-hidden.html [ Pass ] LayoutTests/platform/mac-wk2/TestExpectations:1370:webkit.org/b/225665 [ Release arm64 ] http/tests/appcache/fail-on-update-2.html [ DumpJSConsoleLogInStdErr Slow ] LayoutTests/platform/mac-wk2/TestExpectations:1695:[ Monterey+ ] http/tests/media/video-webm-stall.html [ Failure ] LayoutTests/platform/mac-wk2/TestExpectations:1700:[ Monterey Ventura x86_64 ] http/wpt/mediarecorder/pause-recording.html [ Pass Failure ] # change back to [ Monterey+ x86_64 ] when webkit.org/b/262425 is resolved. LayoutTests/platform/mac/TestExpectations:1980:[ BigSur ] http/tests/media/video-webm-stall.html [ Skip ] LayoutTests/platform/mac/TestExpectations:2746:[ Monterey ] fast/block/margin-collapse/103.html [ Pass Failure ImageOnlyFailure ] LayoutTests/platform/mac/TestExpectations:2753:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/mrow/mrow-fallback.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2754:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-002.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2755:[ Monterey ] imported/w3c/web-platform-tests/mathml/presentation-markup/tables/table-003.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2756:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/floats/not-floating-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2757:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/ignored-properties-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:2758:[ Monterey ] imported/w3c/web-platform-tests/mathml/relations/css-styling/width-height-001.html [ Pass Failure ] LayoutTests/platform/mac/TestExpectations:3034:http/tests/appcache/fail-on-update-2.html [ Pass Timeout ] LayoutTests/platform/mac/TestExpectations:61:fast/forms/search/search-padding-cancel-results-buttons.html [ Pass ] I will work around these when I file expectations.
--- SONOMA UPDATE --- After the changes landed at 268861@main, most of the macOS Sonoma expectations and baselines should be up-to-date. RESOLVED: Below are the tests were left out of the changes in 268861@main due to already being resolved. Bug #262210 (rdar://116139561) -- rebaseline landed in 268612@main: - http/tests/appcache/fail-on-update-2.html Bug #262425 (rdar://116267156) -- fixes landed in 268775@main and 268795@main: - http/wpt/mediarecorder/MediaRecorder-dataavailable.html - http/wpt/mediarecorder/MediaRecorder-multiple-start-stop.html - http/wpt/mediarecorder/MediaRecorder-requestData.html - http/wpt/mediarecorder/pause-recording.html - http/wpt/mediarecorder/record-96KHz-sources.html Bug #262427 (rdar://116268173) -- rebaselines landed in 268823@main: - fast/forms/basic-inputs.html - fast/forms/input-disabled-color.html - fast/forms/number/number-appearance-spinbutton-disabled-readonly.html - fast/forms/placeholder-pseudo-style.html OPEN: Below are the tests that still have outstanding bug reports. Bug #262595 (rdar://116437531): - editing/mac/dictionary-lookup/dictionary-lookup.html - editing/selection/context-menu-text-selection-lookup.html Bug #262604 (rdar://116443613): - http/tests/media/video-webm-stall.html Bug #262663 (rdar://116490690): - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/failures_X25519.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/generateKey/successes_X25519.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey.https.any.worker.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.html - imported/w3c/web-platform-tests/WebCryptoAPI/import_export/okp_importKey_failures_X25519.https.any.worker.html
Important note: in my previous comment, I listed a few bugs that are waiting on resolutions. These tests may require re-baselines, but we will only know after a resolution is reached.
Since I upgraded to macOS Sonoma, the following two SVG tests are constant text failures for me: svg/custom/inline-svg-in-xhtml.xml svg/hixie/mixed/003.xml Font related render tree text differences.. I tried to add Sonoma specific results here (https://ews-build.webkit.org/#/builders/50/builds/491), but the "macOS-AppleSilicon-Sonoma-Debug-WK2-Tests-EWS" results are actually identical to the ones obtained with Ventura/Monterey and not different, as on my machine. I was investigating the issue and came across this ticket, where you mentioned also these two tests as failing on Sonoma. Why don't we observe that on the aforementioned bot? I'm slightly confused.