Bug 263260 - Move testharness.js tests to using a timeout_multiplier
Summary: Move testharness.js tests to using a timeout_multiplier
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-10-17 10:49 PDT by Sam Sneddon [:gsnedders]
Modified: 2023-10-17 10:50 PDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sam Sneddon [:gsnedders] 2023-10-17 10:49:54 PDT
Currently we set the internal testharness.js timeout in our testharnessreport.js, totally overriding testharness.js's default behaviours:

  setTimeout(timeout, testRunner.timeout * 0.9);

For Apple ports (though see also bug 263252), this gives for non-slow tests in most configurations a 27s timeout, and for slow tests in most configurations a 135s timeout.

Upstream wptrunner uses a 10s timeout for most tests, or a 60s timeout for "long" (i.e., slow) tests. That said, this is configurable in a bunch of ways; debug Firefox for example uses a 3x multiplier and therefore ends up with 30s/180s timeouts. (Within WebKit, the glib port uses double the timeout for debug, which would mean 20s/120s timeouts, still both shorter than what we currently use.)

In general, we should probably also be setting timeout_multiplier rather than overriding the timeout directly: this allows tests to use test.step_timeout() which scales its timeout based on the multiplier, which hopefully saves us some flakiness.

There's two main approaches we could go with here:

One is to set it based on the current TestRunner timeout and the default WPT timeout (something like testRunner.timeout * 0.9 / wptTimeout would maintain our current behaviour, but would always result in >1.0 multipliers, which is probably not necessary so we might want a smaller constant than 0.9).

The other is to actually change how we deal with timeouts throughout run-webkit-tests (which probably isn't a bad idea in general!) to have a base timeout and then just repeatedly increase the multiplier, and pass both into the test runner.
Comment 1 Radar WebKit Bug Importer 2023-10-17 10:50:02 PDT
<rdar://problem/117083423>