Bug 260434

Summary: Make it possible to put fuzzy pixel matching thresholds in TestExpectations
Product: WebKit Reporter: Matthieu Dubet <m_dubet>
Component: Tools / TestsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: ap, jbedard, simon.fraser, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   

Description Matthieu Dubet 2023-08-19 11:51:42 PDT
We should have a new kind of test expectations to precisely defined how many pixels difference we have with the reference, to avoid any regression/show improvement.
Comment 1 Radar WebKit Bug Importer 2023-08-19 11:51:57 PDT
<rdar://problem/114138795>
Comment 2 Alexey Proskuryakov 2023-08-21 08:27:09 PDT
An older proposal was to add a mask image that precisely defines which pixels can be different. It's more complicated to implement of course, especially as we'd need tooling to create such masks.
Comment 3 Simon Fraser (smfr) 2023-08-21 09:54:44 PDT
Fuzzy matching can specify a range of pixels (but not which ones).
Comment 4 Simon Fraser (smfr) 2023-08-21 09:55:22 PDT
Oh, I think the request here is to put the fuzzy matching data in TestExpectations, rather than in the test.

Matthieu, could you clarify what you're asking for?
Comment 5 Matthieu Dubet 2024-03-20 11:00:32 PDT
@Simon,

Yes this is about our expectations/our current state, so it should be in the TestExpectations (like the textual expectations with a bunch of PASS and FAIL)

It's not about fuzzing per se (but it could reuse the fuzzing infrastructure to determine the pixel differences).