Bug 263111

Summary:	run-webkit-tests should make it obvious whether results are expected/unexpected
Product:	WebKit	Reporter:	Sam Sneddon [:gsnedders] <gsnedders>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	NEW ---
Severity:	Normal	CC:	bfan2, jbedard, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified

Description Sam Sneddon [:gsnedders] 2023-10-13 02:02:57 PDT

Running:

run-webkit-tests --no-build --no-show-results --release --repeat-each=100000 --exit-after-n-failures=1 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html

…did in fact not exit after 1 failure (which happened after a few repeats).

Presumably --repeat-each and --exit-after-n-failures don't compose as one might expect? Having some way to run a test repeatedly until it fails would be good when investigating flakiness, though.

Comment 1 Radar WebKit Bug Importer 2023-10-13 02:03:05 PDT

<rdar://problem/116910267>

Comment 2 Sam Sneddon [:gsnedders] 2023-10-13 02:07:30 PDT

Oh, maybe the problem is somewhere else:

```
[1/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[2/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[3/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[4/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[5/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[6/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[7/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[8/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[9/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[10/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
                                     
All 10 tests ran as expected.
```

Presumably because:

LayoutTests/platform/wk2/TestExpectations:876:webkit.org/b/259409 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html [ Pass Failure ]

I think my confusion here is because nowhere do we note whether the pass/fail is expected until the end of the test run?

Comment 3 Sam Sneddon [:gsnedders] 2023-10-13 02:09:49 PDT

Let's change this bug to "make it obvious what is expected", because that's the real problem here.

Running with --expect-pass gets the behaviour I want.