Bug 251347

Summary:	[SOUP] Several WPT websockets tests are failing
Product:	WebKit	Reporter:	Diego Pino <dpino>
Component:	WebKitGTK	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	annevk, bugs-noreply, cgarcia, mcatanzaro, webkit-bug-importer, youennf
Priority:	P2
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified

Description Diego Pino 2023-01-29 23:28:44 PST

The following WPT websockets tests are failing for WebKitGTK:

  imported/w3c/web-platform-tests/websockets/constructor/010.html
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/017.html
  imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/018.html
  imported/w3c/web-platform-tests/websockets/opening-handshake/001.html
  imported/w3c/web-platform-tests/websockets/opening-handshake/003.html
  imported/w3c/web-platform-tests/websockets/security/001.html

Tests results:

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/constructor/010-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/constructor/010-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/constructor/010-actual.txt
@@ -1,4 +1,5 @@
-CONSOLE MESSAGE: WebSocket connection to 'ws://web-platform.test:49001/handshake_protocol' failed: The server did not accept the WebSocket handshake.
 
-PASS WebSockets: protocol in response but no requested protocol
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT WebSockets: protocol in response but no requested protocol Test timed out
+

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/017-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/017-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/017-actual.txt
@@ -1,4 +1,5 @@
-CONSOLE MESSAGE: WebSocket connection to 'ws://web-platform.test:49001/echo_raw' failed: Received invalid WebSocket response from the server
 
-PASS WebSockets: this, e.target, e.currentTarget, e.eventPhase
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT WebSockets: this, e.target, e.currentTarget, e.eventPhase Test timed out
+

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/018-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/018-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/interfaces/WebSocket/events/018-actual.txt
@@ -1,4 +1,5 @@
-CONSOLE MESSAGE: WebSocket connection to 'ws://web-platform.test:49001/echo_raw' failed: Received invalid WebSocket response from the server
 
-PASS open, message, error and close events
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT open, message, error and close events Test timed out
+

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/opening-handshake/001-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/opening-handshake/001-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/opening-handshake/001-actual.txt
@@ -1,4 +1,5 @@
-CONSOLE MESSAGE: WebSocket connection to 'ws://web-platform.test:49001/invalid' failed: The server did not accept the WebSocket handshake.
 
-PASS WebSockets: invalid handshake
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT WebSockets: invalid handshake Test timed out
+

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/opening-handshake/003-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/opening-handshake/003-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/opening-handshake/003-actual.txt
@@ -1,3 +1,5 @@
 
-PASS WebSockets: origin
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT WebSockets: origin Test timed out
+

https://build.webkit.org/results/GTK-Linux-64-bit-Release-Tests/259541%40main%20%2810392%29/imported/w3c/web-platform-tests/websockets/security/001-diff.txt

--- /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/security/001-expected.txt
+++ /home/buildbot/worker/GTK-Linux-64-bit-Release-Tests/build/layout-test-results/imported/w3c/web-platform-tests/websockets/security/001-actual.txt
@@ -1,4 +1,5 @@
-CONSOLE MESSAGE: WebSocket connection to 'ws://web-platform.test:49001/wrong_accept_key' failed: Server returned incorrect “Sec-WebSocket-Accept” key
 
-PASS WebSockets: wrong accept key
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT WebSockets: wrong accept key Test timed out
+

Comment 1 Carlos Garcia Campos 2023-02-13 01:51:40 PST

I can't reproduce these. I get failures because of the console message showing the error, but those could be just rebaselined.

Comment 2 Anne van Kesteren 2023-05-10 03:03:07 PDT

I looked into this a bit and I cannot figure out why macOS/iOS don't include console messages and GTK/WPE do. Note that it impacts both the API as well as the channel code so it's not really about macOS/iOS moving to a newer backend as far as I can tell. (Also discussed in #gtk on Slack.)

If someone knows why that is the case it seems like this would be a relatively straightforward fix.

Comment 3 Anne van Kesteren 2023-05-10 10:27:37 PDT

Additional information from Youenn:

> The console messages are put in stderr for some WPT folders
> since they were causing too much havoc. The code is 
> happening in shouldDumpJSConsoleLogInStdErr in 
> Tools/TestRunnerShared/TestFeatures.cpp
> I am not sure why GTK/WPE bots are not activating this option.

Seems like this is very much a GTK/WPE issue.