Blog · 2026-10-30
When the benchmark label is wrong
XXE in Nokogiri, CSRF without a cookie, and rand() as attacker input: three cases where we score lower on purpose, and how we decide.
Public benchmarks are the best tool we have for measuring a scanner honestly, and we publish our results on them. But a benchmark is a set of labels someone wrote, and some labels describe the benchmark's intent rather than the behaviour of the real library. Chasing those labels makes a scanner score better and work worse. Here are three places where we deliberately score lower than we could.
XXE in Nokogiri
One Ruby corpus marks Nokogiri::XML(input) as vulnerable to XML external entities and Nokogiri::XML(input) { |c| c.nonet.noent } as safe. In the real library it is the other way round. The default parse does not substitute external entities; noent is the option that turns substitution on. Flagging the default and trusting noent would teach every Ruby team the wrong lesson, so our Ruby XXE score stays at 0 on that corpus.
CSRF without a cookie
A PHP corpus labels any state-changing handler without a CSRF token as vulnerable — including API handlers that read a bearer token or take no credentials at all. Cross-site request forgery needs a credential the browser attaches by itself: a session cookie. We report CSRF only when a handler relies on the session or a cookie and has no token, nonce, Origin/Referer check or SameSite cookie. On DVWA that catches the real CSRF page and leaves its three defended versions alone. On the corpus it scores 4%.
rand() as attacker input
The Juliet C/C++ suite uses rand() as one of its sources of "untrusted" sizes. Treating a PRNG as attacker input would flag every malloc(rand() % n) in games and simulations. We don't, so those cases count as misses in our uncontrolled-allocation results.
How we decide
We take the corpus label at face value unless we can point to the library's documented behaviour, or to a real application where following the label would produce a false positive or a false negative. When we diverge, the benchmark report lists the case and the reason, so you can disagree with us.