Bug 263111 - run-webkit-tests should make it obvious whether results are expected/unexpected
Summary: run-webkit-tests should make it obvious whether results are expected/unexpected
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Tools / Tests (show other bugs)
Version: WebKit Nightly Build
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2023-10-13 02:02 PDT by Sam Sneddon [:gsnedders]
Modified: 2023-10-13 02:09 PDT (History)
3 users (show)

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sam Sneddon [:gsnedders] 2023-10-13 02:02:57 PDT 
Running:

run-webkit-tests --no-build --no-show-results --release --repeat-each=100000 --exit-after-n-failures=1 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html

…did in fact not exit after 1 failure (which happened after a few repeats).

Presumably --repeat-each and --exit-after-n-failures don't compose as one might expect? Having some way to run a test repeatedly until it fails would be good when investigating flakiness, though.
Comment 1 Radar WebKit Bug Importer 2023-10-13 02:03:05 PDT 
<rdar://problem/116910267>
Comment 2 Sam Sneddon [:gsnedders] 2023-10-13 02:07:30 PDT 
Oh, maybe the problem is somewhere else:

```
[1/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[2/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[3/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[4/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[5/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[6/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[7/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[8/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[9/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[10/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
                                     
All 10 tests ran as expected.
```

Presumably because:

LayoutTests/platform/wk2/TestExpectations:876:webkit.org/b/259409 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html [ Pass Failure ]

I think my confusion here is because nowhere do we note whether the pass/fail is expected until the end of the test run?
Comment 3 Sam Sneddon [:gsnedders] 2023-10-13 02:09:49 PDT 
Let's change this bug to "make it obvious what is expected", because that's the real problem here.

Running with --expect-pass gets the behaviour I want.