Bug 263111
| Summary: | run-webkit-tests should make it obvious whether results are expected/unexpected | ||
|---|---|---|---|
| Product: | WebKit | Reporter: | Sam Sneddon [:gsnedders] <gsnedders> |
| Component: | Tools / Tests | Assignee: | Nobody <webkit-unassigned> |
| Status: | NEW | ||
| Severity: | Normal | CC: | bfan2, jbedard, webkit-bug-importer |
| Priority: | P2 | Keywords: | InRadar |
| Version: | WebKit Nightly Build | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
Sam Sneddon [:gsnedders]
Running:
run-webkit-tests --no-build --no-show-results --release --repeat-each=100000 --exit-after-n-failures=1 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html
…did in fact not exit after 1 failure (which happened after a few repeats).
Presumably --repeat-each and --exit-after-n-failures don't compose as one might expect? Having some way to run a test repeatedly until it fails would be good when investigating flakiness, though.
| Attachments | ||
|---|---|---|
| Add attachment proposed patch, testcase, etc. |
Radar WebKit Bug Importer
<rdar://problem/116910267>
Sam Sneddon [:gsnedders]
Oh, maybe the problem is somewhere else:
```
[1/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[2/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[3/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[4/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[5/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[6/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[7/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
[8/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[9/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html failed (text diff)
[10/10] imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html passed
All 10 tests ran as expected.
```
Presumably because:
LayoutTests/platform/wk2/TestExpectations:876:webkit.org/b/259409 imported/w3c/web-platform-tests/cookies/partitioned-cookies/partitioned-cookies.tentative.https.html [ Pass Failure ]
I think my confusion here is because nowhere do we note whether the pass/fail is expected until the end of the test run?
Sam Sneddon [:gsnedders]
Let's change this bug to "make it obvious what is expected", because that's the real problem here.
Running with --expect-pass gets the behaviour I want.