It happened in https://bugs.webkit.org/show_bug.cgi?id=248382 that the bots were reporting tests failures on a test that was no longer part of the sources. That binary was a leftover from previous builds.

So, to avoid ending up in situations like this (which actually happened recently again), the TestWebKitAPI directory should only contain binaries that are present in the codebase. This could be achieve by listing which Test binaries to run (this is actually how the script is working right now, but when the target is a dir the script simply pulls all contents in the dir):

https://searchfox.org/wubkat/source/Tools/Scripts/run-wpe-tests#35

Another alternative is to flush the TestWebKitAPI directory before building.