Bug 257393

Summary:	Speedometer's flight test has timeouts, which lead to non-determinism
Product:	WebKit	Reporter:	Scott Violet <sky>
Component:	Tools / Tests	Assignee:	Nobody <webkit-unassigned>
Status:	RESOLVED CONFIGURATION CHANGED
Severity:	Normal	CC:	ap, mark.lam, rniwa, ysuzuki
Priority:	P2
Version:	Other
Hardware:	Unspecified
OS:	Unspecified

Description Scott Violet 2023-05-26 14:12:05 PDT

See https://github.com/WebKit/WebKit/blob/main/PerformanceTests/Speedometer2.1/resources/todomvc/dependency-examples/flight/flight/app/js/main.js#L27 . The test does 6 timeouts, which can lead to non-determinism. If the test needs to wait until something is loaded, that should not be included in test time.

Comment 1 Ryosuke Niwa 2023-05-27 18:52:16 PDT

Speedometer 3 benchmark, which is being developed at https://github.com/WebKit/Speedometer/pull/144 does not have this workload.

Comment 2 Scott Violet 2023-06-05 09:00:24 PDT

Isn't that ~6 months out? Seems like it would be good to fix this before the next version of Speedometer.

Comment 3 Ryosuke Niwa 2023-06-05 09:02:51 PDT

(In reply to Scott Violet from comment #2)
> Isn't that ~6 months out? Seems like it would be good to fix this before the
> next version of Speedometer.

This workload is disabled by default anyway so it won't affect the official benchmark result.

Comment 4 Ryosuke Niwa 2023-06-05 09:03:43 PDT

(In reply to Ryosuke Niwa from comment #3)
> (In reply to Scott Violet from comment #2)
> > Isn't that ~6 months out? Seems like it would be good to fix this before the
> > next version of Speedometer.
> 
> This workload is disabled by default anyway so it won't affect the official
> benchmark result.

Actually, that's not true. This is about FlightJS's TodoMVC, which is included in the official result. Regardless, we're not going to fix this for 2.1.