Recall (TPR)
80.3%
32,571 of 40,553 labelled vulnerabilities, all corpora
False-positive rate
1.0%
411 of 40,771 safe samples flagged
Scan throughput
30 files/s
57,496 files across the timed runs. Measured in batch on a Windows developer workstation (single Node.js process, Jest harness) — a throughput figure, not end-to-end pull-request latency.
Pooled numbers weight large corpora more heavily; per-corpus and per-category results are below. Precision is not a single meaningful number across corpora with different positive/negative ratios, so each table lists TP, FN, FP and TN for you to compute it.
At a glance: it is not the same in every language
The pooled recall above averages very different results. It ranges from 46.6% (SAST Benchmark — Rust, Rust) to 100.0% (OWASP Benchmark (Python), Python). Read the row for the language you use, not only the headline.
| Corpus | Languages | Samples | Recall (TPR) | False-positive rate |
|---|---|---|---|---|
| OWASP Benchmark (Python) | Python | 1,230 | 100.0% | 0.8% |
| TrustLedger cross-file suite (internal) | TypeScript, Python, Java, C#, Go, PHP | 72 | 100.0% | 0.0% |
| OWASP Benchmark (Java) | Java | 2,740 | 94.0% | 1.4% |
| NIST Juliet (C#) | C# | 7,992 | 89.8% | 5.5% |
| NIST Juliet (C/C++) | C, C++ | 64,090 | 79.8% | 0.0% |
| SAST Benchmark — PHP | PHP | 1,250 | 67.0% | 6.6% |
| SAST Benchmark — Go | Go | 1,350 | 65.3% | 5.8% |
| SAST Benchmark — Ruby | Ruby | 1,350 | 63.4% | 6.8% |
| SAST Benchmark — Rust | Rust | 1,250 | 46.6% | 6.9% |
Results by corpus, language and category
SAST Benchmark — Go v0.5.1Go · 1,350 labelled samples · run 2026-11-16 · engine 7b67b01+score 59.6TPR 65.3%FPR 5.8%show categories ▾
Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 58.1.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| authnfailure | 11 | 14 | 1 | 24 | 44.0% | 4.0% | 40 |
| authzfailure | 5 | 20 | 2 | 23 | 20.0% | 8.0% | 12 |
| cmdi | 22 | 8 | 8 | 22 | 73.3% | 26.7% | 46.7 |
| codeinj | 20 | 5 | 0 | 25 | 80.0% | 0.0% | 80 |
| csrf | 10 | 15 | 0 | 25 | 40.0% | 0.0% | 40 |
| deserial | 8 | 17 | 2 | 23 | 32.0% | 8.0% | 24 |
| fileupload | 15 | 10 | 2 | 23 | 60.0% | 8.0% | 52 |
| hardcodedcreds | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| infodisclosure | 12 | 13 | 0 | 25 | 48.0% | 0.0% | 48 |
| inputval | 9 | 16 | 0 | 25 | 36.0% | 0.0% | 36 |
| ldapi | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| loginjection | 20 | 5 | 1 | 24 | 80.0% | 4.0% | 76 |
| nosql | 16 | 9 | 0 | 25 | 64.0% | 0.0% | 64 |
| pathtraver | 27 | 3 | 5 | 25 | 90.0% | 16.7% | 73.3 |
| race_condition | 8 | 17 | 0 | 25 | 32.0% | 0.0% | 32 |
| redirect | 17 | 8 | 2 | 23 | 68.0% | 8.0% | 60 |
| securecookie | 21 | 4 | 0 | 25 | 84.0% | 0.0% | 84 |
| sqli | 56 | 9 | 3 | 62 | 86.2% | 4.6% | 81.5 |
| ssrf | 17 | 8 | 3 | 22 | 68.0% | 12.0% | 56 |
| tlsverify | 16 | 9 | 2 | 23 | 64.0% | 8.0% | 56 |
| trustbound | 20 | 5 | 2 | 23 | 80.0% | 8.0% | 72 |
| weakcipher | 22 | 3 | 0 | 25 | 88.0% | 0.0% | 88 |
| weakhash | 22 | 3 | 2 | 23 | 88.0% | 8.0% | 80 |
| weakrand | 15 | 10 | 0 | 25 | 60.0% | 0.0% | 60 |
| xss | 18 | 7 | 4 | 21 | 72.0% | 16.0% | 56 |
SAST Benchmark — PHP v0.3.2PHP · 1,250 labelled samples · run 2026-11-16 · engine 7b67b01+score 60.5TPR 67.0%FPR 6.6%show categories ▾
Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 60.5.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| cmdi | 18 | 7 | 1 | 24 | 72.0% | 4.0% | 68 |
| codeinj | 21 | 4 | 1 | 24 | 84.0% | 4.0% | 80 |
| csrf | 1 | 24 | 0 | 25 | 4.0% | 0.0% | 4 |
| deserial | 18 | 7 | 2 | 23 | 72.0% | 8.0% | 64 |
| extract | 7 | 18 | 1 | 24 | 28.0% | 4.0% | 24 |
| fileinclusion | 22 | 3 | 2 | 23 | 88.0% | 8.0% | 80 |
| fileupload | 22 | 3 | 0 | 25 | 88.0% | 0.0% | 88 |
| hardcodedcreds | 16 | 9 | 0 | 25 | 64.0% | 0.0% | 64 |
| headerinj | 17 | 8 | 1 | 24 | 68.0% | 4.0% | 64 |
| ldapi | 23 | 2 | 1 | 24 | 92.0% | 4.0% | 88 |
| massassign | 14 | 11 | 2 | 23 | 56.0% | 8.0% | 48 |
| pathtraver | 21 | 4 | 5 | 20 | 84.0% | 20.0% | 64 |
| redirect | 19 | 6 | 4 | 21 | 76.0% | 16.0% | 60 |
| securecookie | 11 | 14 | 0 | 25 | 44.0% | 0.0% | 44 |
| sqli | 24 | 1 | 2 | 23 | 96.0% | 8.0% | 88 |
| ssrf | 20 | 5 | 10 | 15 | 80.0% | 40.0% | 40 |
| ssti | 12 | 13 | 1 | 24 | 48.0% | 4.0% | 44 |
| typejuggling | 6 | 19 | 0 | 25 | 24.0% | 0.0% | 24 |
| unsafereflect | 21 | 4 | 2 | 23 | 84.0% | 8.0% | 76 |
| variablevars | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| weakcipher | 22 | 3 | 1 | 24 | 88.0% | 4.0% | 84 |
| weakhash | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| weakrand | 21 | 4 | 0 | 25 | 84.0% | 0.0% | 84 |
| xss | 24 | 1 | 2 | 23 | 96.0% | 8.0% | 88 |
| xxe | 5 | 20 | 3 | 22 | 20.0% | 12.0% | 8 |
SAST Benchmark — Ruby v0.3.2Ruby · 1,350 labelled samples · run 2026-11-16 · engine 7b67b01+score 56.6TPR 63.4%FPR 6.8%show categories ▾
Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 56.6.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| authnfailure | 4 | 21 | 1 | 24 | 16.0% | 4.0% | 12 |
| authzfailure | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| cmdi | 17 | 8 | 2 | 23 | 68.0% | 8.0% | 60 |
| codeinj | 21 | 4 | 2 | 23 | 84.0% | 8.0% | 76 |
| csrf | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| deserial | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| dynmethod | 18 | 7 | 1 | 24 | 72.0% | 4.0% | 68 |
| fileinclusion | 23 | 2 | 2 | 23 | 92.0% | 8.0% | 84 |
| fileupload | 14 | 11 | 2 | 23 | 56.0% | 8.0% | 48 |
| hardcodedcreds | 15 | 10 | 0 | 25 | 60.0% | 0.0% | 60 |
| headerinj | 17 | 8 | 2 | 23 | 68.0% | 8.0% | 60 |
| ldapi | 21 | 4 | 0 | 25 | 84.0% | 0.0% | 84 |
| loginjection | 18 | 7 | 3 | 22 | 72.0% | 12.0% | 60 |
| massassign | 18 | 7 | 0 | 25 | 72.0% | 0.0% | 72 |
| pathtraver | 22 | 3 | 5 | 20 | 88.0% | 20.0% | 68 |
| redirect | 22 | 3 | 8 | 17 | 88.0% | 32.0% | 56 |
| regex | 15 | 10 | 3 | 22 | 60.0% | 12.0% | 48 |
| securecookie | 9 | 16 | 1 | 24 | 36.0% | 4.0% | 32 |
| sqli | 25 | 0 | 0 | 25 | 100.0% | 0.0% | 100 |
| ssrf | 19 | 6 | 5 | 20 | 76.0% | 20.0% | 56 |
| ssti | 14 | 11 | 0 | 25 | 56.0% | 0.0% | 56 |
| unsafereflect | 21 | 4 | 3 | 22 | 84.0% | 12.0% | 72 |
| weakcipher | 24 | 1 | 3 | 22 | 96.0% | 12.0% | 84 |
| weakhash | 18 | 7 | 0 | 25 | 72.0% | 0.0% | 72 |
| weakrand | 20 | 5 | 0 | 25 | 80.0% | 0.0% | 80 |
| xss | 16 | 9 | 3 | 22 | 64.0% | 12.0% | 52 |
| xxe | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
SAST Benchmark — Rust v0.5.2Rust · 1,250 labelled samples · run 2026-11-16 · engine 7b67b01+score 39.7TPR 46.6%FPR 6.9%show categories ▾
Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 39.7.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| authnfailure | 3 | 22 | 0 | 25 | 12.0% | 0.0% | 12 |
| authzfailure | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| cmdi | 23 | 2 | 6 | 19 | 92.0% | 24.0% | 68 |
| crypto | 12 | 13 | 5 | 20 | 48.0% | 20.0% | 28 |
| csrf | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| deserial | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| fileupload | 19 | 6 | 4 | 21 | 76.0% | 16.0% | 60 |
| hardcodedcreds | 15 | 10 | 0 | 25 | 60.0% | 0.0% | 60 |
| infodisclosure | 1 | 24 | 0 | 25 | 4.0% | 0.0% | 4 |
| inputval | 1 | 24 | 0 | 25 | 4.0% | 0.0% | 4 |
| intoverflow | 21 | 4 | 1 | 24 | 84.0% | 4.0% | 80 |
| ldapi | 19 | 6 | 6 | 19 | 76.0% | 24.0% | 52 |
| loginjection | 21 | 4 | 5 | 20 | 84.0% | 20.0% | 64 |
| memsafety | 17 | 8 | 0 | 25 | 68.0% | 0.0% | 68 |
| nosql | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| pathtraver | 24 | 1 | 0 | 25 | 96.0% | 0.0% | 96 |
| race_condition | 3 | 22 | 0 | 25 | 12.0% | 0.0% | 12 |
| redirect | 23 | 2 | 8 | 17 | 92.0% | 32.0% | 60 |
| redos | 17 | 8 | 1 | 24 | 68.0% | 4.0% | 64 |
| securecookie | 18 | 7 | 1 | 24 | 72.0% | 4.0% | 68 |
| sqli | 22 | 3 | 1 | 24 | 88.0% | 4.0% | 84 |
| ssrf | 2 | 23 | 0 | 25 | 8.0% | 0.0% | 8 |
| tlsverify | 6 | 19 | 0 | 25 | 24.0% | 0.0% | 24 |
| weakrand | 0 | 25 | 0 | 25 | 0.0% | 0.0% | 0 |
| xss | 24 | 1 | 5 | 20 | 96.0% | 20.0% | 76 |
OWASP Benchmark (Java) v1.2Java · 2,740 labelled samples · run 2026-11-03 · engine 0e73164score 92.6TPR 94.0%FPR 1.4%show categories ▾
Each test case is one servlet labelled vulnerable or safe for a category; helper classes are scanned alongside, as in a whole-repository scan. Source: https://github.com/OWASP-Benchmark/BenchmarkJava. Category-averaged score: 93.5.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| cmdi | 126 | 0 | 0 | 125 | 100.0% | 0.0% | 100 |
| crypto | 130 | 0 | 0 | 116 | 100.0% | 0.0% | 100 |
| hash | 89 | 40 | 0 | 107 | 69.0% | 0.0% | 69 |
| ldapi | 27 | 0 | 0 | 32 | 100.0% | 0.0% | 100 |
| pathtraver | 115 | 18 | 0 | 135 | 86.5% | 0.0% | 86.5 |
| securecookie | 36 | 0 | 0 | 31 | 100.0% | 0.0% | 100 |
| sqli | 272 | 0 | 18 | 214 | 100.0% | 7.8% | 92.2 |
| trustbound | 74 | 9 | 0 | 43 | 89.2% | 0.0% | 89.2 |
| weakrand | 200 | 18 | 0 | 275 | 91.7% | 0.0% | 91.7 |
| xpathi | 15 | 0 | 0 | 20 | 100.0% | 0.0% | 100 |
| xss | 246 | 0 | 0 | 209 | 100.0% | 0.0% | 100 |
OWASP Benchmark (Python) v0.1Python · 1,230 labelled samples · run 2026-11-02 · engine b4505ab+score 99.2TPR 100.0%FPR 0.8%show categories ▾
Each test case is one Flask view labelled vulnerable or safe for a category; helper modules are scanned alongside. Source: https://github.com/OWASP-Benchmark/BenchmarkPython. Category-averaged score: 98.8.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| cmdi | 13 | 0 | 0 | 7 | 100.0% | 0.0% | 100 |
| codeinj | 20 | 0 | 2 | 31 | 100.0% | 6.1% | 93.9 |
| deserialization | 18 | 0 | 3 | 33 | 100.0% | 8.3% | 91.7 |
| hash | 71 | 0 | 0 | 80 | 100.0% | 0.0% | 100 |
| ldapi | 16 | 0 | 0 | 13 | 100.0% | 0.0% | 100 |
| pathtraver | 65 | 0 | 0 | 103 | 100.0% | 0.0% | 100 |
| redirect | 13 | 0 | 0 | 21 | 100.0% | 0.0% | 100 |
| securecookie | 24 | 0 | 0 | 15 | 100.0% | 0.0% | 100 |
| sqli | 5 | 0 | 0 | 11 | 100.0% | 0.0% | 100 |
| trustbound | 18 | 0 | 0 | 19 | 100.0% | 0.0% | 100 |
| weakrand | 99 | 0 | 0 | 227 | 100.0% | 0.0% | 100 |
| xpathi | 51 | 0 | 0 | 135 | 100.0% | 0.0% | 100 |
| xss | 31 | 0 | 1 | 57 | 100.0% | 1.7% | 98.3 |
| xxe | 8 | 0 | 0 | 20 | 100.0% | 0.0% | 100 |
NIST Juliet (C#) vSARDC# · 7,992 labelled samples · run 2026-11-01 · engine 756c80dscore 84.4TPR 89.8%FPR 5.5%show categories ▾
Every case has a vulnerable (bad) and a safe (good) region; a case counts as detected when a finding of its CWE's rules lands in the bad region, and as a false positive when one lands in a good region. Cases with a web or network source. Source: https://samate.nist.gov/SARD/test-suites. Category-averaged score: 85.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| CWE89_SQL_Injection/s01 | 370 | 0 | 101 | 269 | 100.0% | 27.3% | 72.7 |
| CWE89_SQL_Injection/s02 | 296 | 0 | 81 | 215 | 100.0% | 27.4% | 72.6 |
| CWE78_OS_Command_Injection | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE80_XSS | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE81_XSS_Error_Message | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE83_XSS_Attribute | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE23_Relative_Path_Traversal | 197 | 25 | 0 | 222 | 88.7% | 0.0% | 88.7 |
| CWE36_Absolute_Path_Traversal | 197 | 25 | 0 | 222 | 88.7% | 0.0% | 88.7 |
| CWE601_Open_Redirect | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE113_HTTP_Response_Splitting/s01 | 328 | 42 | 10 | 360 | 88.6% | 2.7% | 85.9 |
| CWE113_HTTP_Response_Splitting/s02 | 263 | 33 | 8 | 288 | 88.9% | 2.7% | 86.1 |
| CWE90_LDAP_Injection | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE643_Xpath_Injection | 198 | 24 | 6 | 216 | 89.2% | 2.7% | 86.5 |
| CWE94_Improper_Control_of_Generation_of_Code | 198 | 24 | 6 | 216 | 89.2% | 2.7% | 86.5 |
| CWE470_Unsafe_Reflection | 192 | 30 | 0 | 222 | 86.5% | 0.0% | 86.5 |
| CWE117_Improper_Output_Neutralization_for_Logs | 198 | 24 | 6 | 216 | 89.2% | 2.7% | 86.5 |
NIST Juliet (C/C++) v1.3C, C++ · 64,090 labelled samples · run 2026-11-16 · engine 528472b+score 79.8TPR 79.8%FPR 0.0%show categories ▾
Every case has a vulnerable (bad) and a safe (good) function; a case counts as detected when a finding of its CWE's rules lands in a bad function, and as a false positive when one lands in a good function. Source: https://samate.nist.gov/SARD/test-suites. Category-averaged score: 84.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| CWE78_OS_Command_Injection | 4400 | 400 | 0 | 4800 | 91.7% | 0.0% | 91.7 |
| CWE134_Uncontrolled_Format_String | 2640 | 240 | 0 | 2880 | 91.7% | 0.0% | 91.7 |
| CWE23_Relative_Path_Traversal | 2100 | 300 | 0 | 2400 | 87.5% | 0.0% | 87.5 |
| CWE36_Absolute_Path_Traversal | 2100 | 300 | 0 | 2400 | 87.5% | 0.0% | 87.5 |
| CWE90_LDAP_Injection | 440 | 40 | 0 | 480 | 91.7% | 0.0% | 91.7 |
| CWE114_Process_Control | 440 | 136 | 0 | 576 | 76.4% | 0.0% | 76.4 |
| CWE242_Use_of_Inherently_Dangerous_Function | 18 | 0 | 0 | 18 | 100.0% | 0.0% | 100 |
| CWE377_Insecure_Temporary_File | 144 | 0 | 0 | 144 | 100.0% | 0.0% | 100 |
| CWE415_Double_Free | 880 | 82 | 0 | 962 | 91.5% | 0.0% | 91.5 |
| CWE416_Use_After_Free | 438 | 21 | 0 | 459 | 95.4% | 0.0% | 95.4 |
| CWE690_NULL_Deref_From_Return | 820 | 140 | 0 | 960 | 85.4% | 0.0% | 85.4 |
| CWE762_Mismatched_Memory_Management_Routines | 3182 | 382 | 0 | 3564 | 89.3% | 0.0% | 89.3 |
| CWE789_Uncontrolled_Mem_Alloc | 672 | 288 | 0 | 960 | 70.0% | 0.0% | 70 |
| CWE680_Integer_Overflow_to_Buffer_Overflow | 336 | 240 | 0 | 576 | 58.3% | 0.0% | 58.3 |
| CWE121_Stack_Based_Buffer_Overflow | 3210 | 1734 | 0 | 4944 | 64.9% | 0.0% | 64.9 |
| CWE122_Heap_Based_Buffer_Overflow | 3756 | 2166 | 0 | 5922 | 63.4% | 0.0% | 63.4 |
TrustLedger cross-file suite (internal) v756c80dTypeScript, Python, Java, C#, Go, PHP · 72 labelled samples · run 2026-11-01 · engine 756c80dscore 100TPR 100.0%FPR 0.0%show categories ▾
Our own suite: 72 small multi-file programs, each with a flow that crosses files (vulnerable) or a safe twin. Less independent than the public corpora, since we wrote it; included because no public corpus targets cross-file flows. Source: src/__tests__/benchmarks/crossFile in the TrustLedger repository. Category-averaged score: 100.
| Category | TP | FN | FP | TN | TPR | FPR | Score |
|---|---|---|---|---|---|---|---|
| csharp | 5 | 0 | 0 | 4 | 100.0% | 0.0% | 100 |
| go | 7 | 0 | 0 | 4 | 100.0% | 0.0% | 100 |
| java | 7 | 0 | 0 | 4 | 100.0% | 0.0% | 100 |
| php | 8 | 0 | 0 | 5 | 100.0% | 0.0% | 100 |
| python | 8 | 0 | 0 | 5 | 100.0% | 0.0% | 100 |
| typescript | 10 | 0 | 0 | 5 | 100.0% | 0.0% | 100 |
VulnBench 2.0 (Snyk): 108 attacker-reachable vulnerabilities in 20 real applications
Attacker-reachable recall
88 / 108
81.5% of the vulnerabilities; 80.1% averaged over the 20 applications, the metric the benchmark ranks by
Looser check
93 / 108
86.1% when any labelled location within 2 lines counts. A sensitivity check, not the headline
Precision lower bound
40.7%
88 of 216 findings of the scored types matched a labelled vulnerability; the rest may be real but unlabelled
Recall by vulnerability typeshow ▾
| Type | Reference | Matched | Recall | Looser |
|---|---|---|---|---|
| xss | 19 | 9 | 47.4% | 12 |
| path-traversal | 16 | 15 | 93.8% | 15 |
| command-injection | 13 | 12 | 92.3% | 12 |
| sql-injection | 13 | 12 | 92.3% | 13 |
| open-redirect | 11 | 9 | 81.8% | 9 |
| ssrf | 9 | 7 | 77.8% | 8 |
| prototype-pollution | 6 | 6 | 100.0% | 6 |
| insecure-deserialization | 5 | 5 | 100.0% | 5 |
| template-injection | 4 | 3 | 75.0% | 3 |
| code-injection | 4 | 4 | 100.0% | 4 |
| redos | 3 | 1 | 33.3% | 1 |
| mass-assignment | 2 | 2 | 100.0% | 2 |
| improper-type-validation | 2 | 2 | 100.0% | 2 |
| idor | 1 | 1 | 100.0% | 1 |
Proof: all 108 vulnerabilities, with the finding that matched eachshow ▾
Each row is one labelled vulnerability: the application, its type, the labelled source/sink location, and the rule, file and line of the TrustLedger finding that matched it. The same list is machine-readable at /benchmarks/vulnbench.json.
| App | Type | Labelled location | Result | Matched by |
|---|---|---|---|---|
| ardanservice | command-injection | app/domain/checkapp/diagnostics.go:10 | found | command-injection · app/domain/checkapp/checkapp.go:55 |
| ardanservice | sql-injection | app/domain/userapp/department_search.go:12 | found | sql-injection · app/domain/userapp/userapp.go:184 |
| ardanservice | ssrf | app/domain/userapp/webhook.go:9 | found | ssrf · app/domain/userapp/userapp.go:163 |
| ardanservice | open-redirect | app/domain/userapp/redirect.go:9 | found | open-redirect · app/domain/userapp/userapp.go:174 |
| ardanservice | path-traversal | app/domain/auditapp/reports.go:12 | found | path-traversal · app/domain/auditapp/auditapp.go:66 |
| blanq | xss | app/components/SharedPathTooltip.vue:24 | found | xss · app/components/SharedPathTooltip.vue:24 |
| blanq | xss | app/composables/useCampaignMessage.ts:16 | found | xss · app/composables/useCampaignMessage.ts:16 |
| blanq | xss | app/composables/useReferralSource.ts:16 | found | xss · app/composables/useReferralSource.ts:16 |
| blanq | prototype-pollution | app/composables/useSharedPreferences.ts:14 | found | prototype-pollution · app/composables/useSharedPreferences.ts:14 |
| blanq | prototype-pollution | app/composables/useSharedPreferences.ts:22 | found | prototype-pollution · app/composables/useSharedPreferences.ts:22 |
| blanq | redos | server/api/auth/email-policy.get.js:10 | missed | — |
| blanq | ssrf | server/api/integrations/logo-preview.get.js:15 | missed | — |
| boilwater | command-injection | config/dns-diagnostics.js:9 | found | command-injection · controllers/api.js:1751 |
| boilwater | open-redirect | config/redirect-handler.js:1 | found | open-redirect · controllers/api.js:1769 |
| boilwater | path-traversal | config/document-downloads.js:6 | found | path-traversal · controllers/api.js:1778 |
| boilwater | open-redirect | controllers/user.js:649 | found | open-redirect · controllers/user.js:649 |
| coffeeshop | sql-injection | src/main/java/nl/scottjohnson/coffeeshop/repository/SearchRepository.java:25 | found | sql-injection · src/main/java/nl/scottjohnson/coffeeshop/controller/HomeController.java:30 |
| coffeeshop | path-traversal | src/main/java/nl/scottjohnson/coffeeshop/controller/UploadController.java:34 | found | path-traversal · src/main/java/nl/scottjohnson/coffeeshop/controller/UploadController.java:34 |
| coffeeshop | xss | src/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:91 | found | xss · src/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:91 |
| coffeeshop | xss | src/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:101 | missed | — |
| coffeeshop | template-injection | src/main/resources/templates/fragments/productsTable.html:45 | missed | — |
| conduit | sql-injection | conduit/articles/search.py:19 | found | sql-injection · conduit/articles/views.py:183 |
| conduit | path-traversal | conduit/articles/export.py:19 | found | path-traversal · conduit/articles/views.py:197 |
| conduit | command-injection | conduit/articles/pdf.py:16 | found | command-injection · conduit/articles/views.py:208 |
| conduit | insecure-deserialization | conduit/articles/views.py:229 | found | insecure-deserialization · conduit/articles/views.py:229 |
| conduit | code-injection | conduit/articles/views.py:244 | found | eval-exec · conduit/articles/views.py:244 |
| conduit | ssrf | conduit/user/avatar.py:8 | found | ssrf · conduit/user/views.py:66 |
| flask-bones | open-redirect | app/auth/views.py:36 | found | open-redirect · app/auth/views.py:36 |
| flask-bones | open-redirect | app/auth/views.py:120 | found | open-redirect · app/auth/views.py:120 |
| flask-bones | command-injection | app/utils.py:18 | found | command-injection · app/user/views.py:62 |
| flask-bones | command-injection | app/utils.py:23 | found | command-injection · app/user/views.py:72 |
| flask-bones | insecure-deserialization | app/user/views.py:25 | found | insecure-deserialization · app/user/views.py:25 |
| flask-bones | insecure-deserialization | app/auth/views.py:67 | found | insecure-deserialization · app/auth/views.py:67 |
| flask-bones | path-traversal | app/user/reports.py:9 | found | path-traversal · app/user/views.py:82 |
| flask-bones | path-traversal | app/user/reports.py:16 | found | path-traversal · app/user/views.py:92 |
| flask-bones | mass-assignment | app/user/views.py:102 | found | mass-assignment · app/user/views.py:102 |
| flask-bones | idor | app/user/views.py:115 | found | bola-missing-ownership-check · app/user/views.py:114 |
| go-sample | sql-injection | delivery/http/search.go:23 | found | sql-injection · delivery/http/api.go:33 |
| go-sample | sql-injection | delivery/http/search.go:52 | found | sql-injection · delivery/http/api.go:51 |
| go-sample | sql-injection | delivery/http/search.go:85 | found | sql-injection · delivery/http/api.go:79 |
| go-sample | xss | delivery/http/report.go:28 | missed | — |
| go-sample | xss | delivery/http/report.go:39 | missed | — |
| gobase | command-injection | internal/gateways/web/controllers/apiv1/workspace/diagnostics.go:10 | found | command-injection · internal/gateways/web/controllers/apiv1/workspace/controller.go:37 |
| gobase | open-redirect | internal/gateways/web/controllers/apiv1/workspace/navigation.go:8 | found | open-redirect · internal/gateways/web/controllers/apiv1/workspace/controller.go:30 |
| gobase | path-traversal | internal/gateways/web/controllers/apiv1/workspace/reports.go:12 | found | path-traversal · internal/gateways/web/controllers/apiv1/workspace/controller.go:49 |
| gobase | ssrf | internal/gateways/web/controllers/apiv1/workspace/branding.go:9 | found | ssrf · internal/gateways/web/controllers/apiv1/workspace/controller.go:62 |
| gobase | template-injection | internal/gateways/web/controllers/apiv1/workspace/templates.go:9 | found | ssti · internal/gateways/web/controllers/apiv1/workspace/controller.go:75 |
| goxygen | command-injection | server/web/diagnostics.go:10 | found | command-injection · server/web/diagnostics_handler.go:7 |
| goxygen | open-redirect | server/web/navigation_redirect.go:6 | found | open-redirect · server/web/navigation.go:10 |
| goxygen | path-traversal | server/web/report_files.go:10 | found | path-traversal · server/web/reports.go:9 |
| goxygen | ssrf | server/web/branding_assets.go:9 | found | ssrf · server/web/branding.go:9 |
| goxygen | xss | webapp/src/App.vue:48 | found | xss · webapp/src/App.vue:48 |
| goxygen | prototype-pollution | webapp/src/workspacePreferences.js:14 | found | prototype-pollution · webapp/src/workspacePreferences.js:12 |
| halloween | improper-type-validation | src/api/user/userController.ts:17 | found | type-confusion · src/api/user/userController.ts:16 |
| halloween | improper-type-validation | src/api/user/userController.ts:95 | found | type-confusion · src/api/user/userController.ts:95 |
| halloween | mass-assignment | src/api/user/userService.ts:87 | found | mass-assignment · src/api/user/userController.ts:51 |
| iguana | command-injection | src/gitintegration/frontend.py:45 | found | command-injection · src/gitintegration/views.py:123 |
| iguana | redos | src/search/views.py:67 | found | redos · src/search/views.py:67 |
| iguana | redos | src/search/frontend.py:133 | missed | — |
| iguana | sql-injection | src/issue/search.py:19 | found | sql-injection · src/issue/views.py:423 |
| iguana | code-injection | src/issue/views.py:440 | found | eval-exec · src/issue/views.py:440 |
| iguana | path-traversal | src/project/export.py:21 | found | path-traversal · src/project/views.py:129 |
| keystonebank | path-traversal | backend/services/BankStatements.ts:14 | missed | — |
| keystonebank | prototype-pollution | backend/routes/memos.ts:52 | found | prototype-pollution · backend/routes/memos.ts:52 |
| keystonebank | sql-injection | backend/routes/finchat.ts:43 | partly (looser rule only) | sql-injection · backend/routes/finchat.ts:43 |
| keystonebank | xss | backend/routes/bank_profile_template.ts:27 | missed | — |
| lastsaas | xss | frontend/src/pages/public/CustomPage.tsx:65 | partly (looser rule only) | xss · frontend/src/pages/public/CustomPage.tsx:65 |
| lastsaas | open-redirect | frontend/src/pages/auth/LoginPage.tsx:29 | found | open-redirect · frontend/src/pages/auth/LoginPage.tsx:29 |
| lastsaas | ssrf | backend/internal/api/handlers/branding_assets.go:19 | found | ssrf · backend/internal/api/handlers/branding.go:160 |
| lastsaas | command-injection | backend/internal/health/diagnostics.go:11 | found | command-injection · backend/internal/api/handlers/health_diagnostics.go:17 |
| lastsaas | path-traversal | backend/internal/api/handlers/branding_assets.go:43 | found | path-traversal · backend/internal/api/handlers/branding.go:177 |
| pygmy | sql-injection | pygmy/model/search.py:20 | found | sql-injection · pygmy/rest/user.py:114 |
| pygmy | path-traversal | pygmy/utilities/export.py:20 | found | path-traversal · pygmy/rest/user.py:147 |
| pygmy | command-injection | pygmy/core/network.py:8 | found | command-injection · pygmy/rest/diagnostics.py:15 |
| pygmy | code-injection | pygmy/rest/user.py:135 | found | eval-exec · pygmy/rest/user.py:135 |
| pygmy | ssrf | pygmy/utilities/reachability.py:8 | partly (looser rule only) | ssrf · pygmy/rest/shorturl.py:61 |
| robs-awesome | sql-injection | backend/services/reports.py:9 | found | sql-injection · backend/www.py:56 |
| robs-awesome | path-traversal | backend/www.py:99 | found | path-traversal · backend/www.py:99 |
| robs-awesome | command-injection | backend/www.py:108 | found | command-injection · backend/www.py:108 |
| robs-awesome | ssrf | backend/services/integrations.py:6 | found | ssrf · backend/www.py:118 |
| robs-awesome | insecure-deserialization | backend/services/drafts.py:8 | found | insecure-deserialization · backend/www.py:80 |
| robs-awesome | open-redirect | backend/services/redirects.py:6 | missed | — |
| robs-awesome | code-injection | backend/www.py:70 | found | eval-exec · backend/www.py:70 |
| saas-starter-kit | command-injection | cmd/web-app/handlers/integration_diagnostics.go:11 | found | command-injection · cmd/web-app/handlers/integrations.go:20 |
| saas-starter-kit | open-redirect | cmd/web-app/handlers/user.go:108 | missed | — |
| saas-starter-kit | path-traversal | cmd/web-app/handlers/account_reports.go:10 | found | path-traversal · cmd/web-app/handlers/account.go:80 |
| saas-starter-kit | sql-injection | cmd/web-api/handlers/checklist.go:379 | found | sql-injection · cmd/web-api/handlers/checklist.go:379 |
| saas-starter-kit | ssrf | cmd/web-app/handlers/account_assets.go:10 | found | ssrf · cmd/web-app/handlers/account.go:65 |
| saas-starter-kit | template-injection | cmd/web-app/handlers/integrations.go:40 | found | ssti · cmd/web-app/handlers/integrations.go:40 |
| saas-starter-lite | open-redirect | apps/api/src/auth/auth.controller.ts:66 | found | open-redirect · apps/api/src/auth/auth.controller.ts:66 |
| saas-starter-lite | template-injection | apps/api/src/auth/auth.controller.ts:60 | found | ssti · apps/api/src/auth/auth.controller.ts:60 |
| saas-starter-lite | xss | apps/api/src/auth/auth.controller.ts:55 | found | xss · apps/api/src/auth/auth.controller.ts:55 |
| saas-starter-lite | xss | apps/web/src/app/pages/auth/register/register.ts:148 | partly (looser rule only) | xss · apps/web/src/app/pages/auth/register/register.ts:148 |
| saas-starter-lite | command-injection | apps/api/src/users/users.service.ts:39 | missed | — |
| saas-starter-lite | sql-injection | apps/api/src/users/users.service.ts:55 | found | sql-injection · apps/api/src/users/users.controller.ts:91 |
| sassyreg | xss | src/views/examples/PackageProfile.js:168 | found | xss · src/views/examples/PackageProfile.js:168 |
| sassyreg | xss | src/views/examples/PackageProfile.js:325 | partly (looser rule only) | xss · src/views/examples/PackageProfile.js:322 |
| slingshot | xss | src/components/AboutPage.js:14 | found | xss · src/components/AboutPage.js:14 |
| slingshot | xss | src/components/HomePage.js:13 | found | xss · src/components/HomePage.js:13 |
| slingshot | prototype-pollution | src/utils/sharedPreferences.js:10 | found | prototype-pollution · src/utils/sharedPreferences.js:10 |
| slingshot | prototype-pollution | src/utils/sharedPreferences.js:26 | found | prototype-pollution · src/utils/sharedPreferences.js:26 |
| vinyl-marketplace | sql-injection | src/main/java/io/acme/engineering/vynil_marketplace/service/VinylService.java:41 | found | sql-injection · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:42 |
| vinyl-marketplace | insecure-deserialization | src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:87 | found | insecure-deserialization · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:88 |
| vinyl-marketplace | path-traversal | src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:128 | found | path-traversal · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:128 |
| vinyl-marketplace | path-traversal | src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:167 | found | path-traversal · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:167 |
| vinyl-marketplace | xss | src/main/resources/static/index.html:297 | missed | — |
| vinyl-marketplace | xss | src/main/resources/static/index.html:298 | missed | — |
| vinyl-marketplace | xss | src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:118 | missed | — |
How this compares with what Snyk published
Attacker-reachable vulnerability recall as printed on the release page; we did not re-run these tools. Source: https://vulnbench.com/releases/2.0, copied 2026-11-12.
| Scanner | Matched | Recall (macro) | Precision lower bound |
|---|---|---|---|
| TrustLedger (this run) | 88 | 80.1% | 40.7% |
| Snyk Code | 89 | 82.4% | 20.9% |
| Claude Code /security-review, Opus 5 xhigh | 85 | 79.3% | 74.6% |
| Claude Code, Sonnet 5 xhigh | 73 | 67.0% | 74.5% |
| Codex Security, GPT-5.6 Luna | 45 | 41.2% | 20.8% |
Read this as a comparison of one run against published numbers, not a leaderboard entry. We scored with our own implementation of the published rule, not Snyk's harness; their results carry resampling intervals over the 20 applications that ours does not, and their precision figures are computed over a different set of reported findings. The scorer, the probe and the pinned corpus commit (385a375) are in the repository under benchmarks/, so the run can be repeated. Engine commit 23da128, measured 2026-11-12.
AI-attribution validation
Precision
100.0%
Recall
60.0%
Accuracy
80.0%
threshold 0.5
ROC-AUC
1.00
Tool attribution
100.0%
10 labelled samples
Read the method and its limits in the AI attribution documentation.
False positives and false negatives we know about
False positives — flagged, but safe
A switch on a constant character
The helper switches on "ABC".charAt(1), which is always 'B', and that arm assigns a constant; the arms that assign the request value never run. The engine does not evaluate the switch subject, so it keeps every arm and the line-level concatenation match stands. Engine gap; most of the remaining BenchmarkJava SQL false positives are constant-folding tricks of this kind (constant switches and if-conditions, fixed list and map positions).
A sanitizer written as a character filter
The username is cleaned with chars().filter(|c| !c.is_ascii_control()), which removes CR and LF, but the engine does not recognise a closure-based filter as a log sanitizer, so the log line is still reported. Engine gap.
Ownership proven by an HMAC-signed token
The handler verifies an HMAC-signed resource token that binds the caller to the resource id before the lookup. The engine recognises comparisons with the authenticated principal but not a signature check, so it reports the lookup. Engine gap.
An https-only check is labelled safe
The corpus treats a request URL as safe once its scheme is https. Restricting the scheme does not stop a request to an internal host (https://169.254.169.254/, an internal service), so we keep reporting it. A deliberate disagreement with the label.
A signing-method check does not make a literal key safe
The test checks that the token uses an HMAC method and then verifies it with the key []byte("supersecret"). The corpus labels it safe because the algorithm is pinned; we still report it, because a key written in the source lets anyone mint a token the parser accepts.
An instanceof check after unserialize is too late
A request value is passed to igbinary_unserialize and the result is then checked with instanceof. The corpus labels it safe; we report it, because the object is already built (and its destructors and wakeup methods run) before the check, so a gadget chain does not need the type to match.
False negatives — missed
A hash algorithm chosen in a properties file
The algorithm name is read with getProperty("hashAlg1", ...) from benchmark.properties, where it is set to a weak hash. The engine does not resolve values from .properties files, so every hash case configured this way is missed. Engine gap; on the roadmap.
A stub HTTP client
The request URL reaches simulated_reqwest_get(), a function the corpus defines to stand in for an HTTP client and which only formats a string. We report calls to real HTTP libraries, not functions that merely have a client-like name, so these cases count as misses. A deliberate choice.
Nokogiri's default parse is labelled vulnerable
Nokogiri::XML(input) does not substitute external entities by default; the corpus labels it vulnerable and labels the noent option — which turns substitution on — as safe. We follow the library's real behaviour, so Ruby XXE scores 0 on this corpus. A deliberate disagreement with the label.
CSRF without a session or cookie
The corpus labels a state-changing handler with no CSRF token as vulnerable even when it reads no session or cookie. Cross-site request forgery needs a credential the browser attaches by itself, so we report CSRF only for session- or cookie-authenticated handlers. A deliberate choice.
A weak password hash behind a helper in another file
models/user.ts stores security.hash(password), and the weak MD5 hash lives in lib/insecurity.ts. The weak hash itself is reported there, but not the use of it for passwords in the model. Engine gap.
yaml.Unmarshal into interface{} is labelled vulnerable
The corpus files decoding a request body with gopkg.in/yaml.v3 into an interface{} under deserialization. Go's YAML decoder builds plain maps, slices and scalars and runs no code, so there is no gadget to reach; we do not report it, and score it as a miss rather than change the label.
A same-file function named like a deserializer
The test calls deserialize_object(body.as_bytes()), a function defined a few lines below that returns a constant. Nothing in the file reads or builds an object, and we do not treat a locally defined function as a library sink because of its name, so the case counts as a miss.
Known limitations
- Benchmarks are small, synthetic programs. They measure whether known patterns are recognised, not how the engines behave on your codebase, and scores on different corpora are not comparable with each other.
- Trace accuracy is not measured: no public corpus labels the expected source-to-sink path.
- Kotlin has no public labelled corpus and is not in this report; it is covered by unit tests and checked against the AndroGoat and Goatlin apps by hand.
- Juliet cases are scored by region (a finding in the bad or good function); Juice Shop challenges count as detected when a finding lands within two lines of a marked line.
- The internal cross-file suite was written by us. It is included because no public corpus targets cross-file flows, but it is less independent than the public corpora.
- The AI-attribution check uses 10 labelled TypeScript/JavaScript files: a regression check, not a statistically strong evaluation.
- Throughput is measured in batch on a developer workstation, on the corpora scanned in a single pass. It is not end-to-end pull-request latency.
- Where we disagree with a corpus label we keep the label in the score and explain the disagreement in the examples above, rather than changing the label.
- Many Rust cases, and some PHP and Ruby ones, call a function defined in the same file that only imitates a library call (simulated_reqwest_get, aes_ecb_encrypt, native_tls_danger). We scan the code as written and do not treat a locally defined function as a library sink because of its name, so those cases count as misses; this is the main reason Rust's recall is low, and it says little about code that calls the real libraries.
- The sast-benchmark and Juliet C/C++ corpora were re-run on 2026-11-16. The OWASP Benchmark (Java, Python), Juliet C# and Juice Shop corpora were not available for that run, so each of them keeps the engine commit named against it rather than being presented as measured on today's engine.
- Juliet C/C++: cases that call virtual methods through a base class defined in another file, and the paired cases whose bad and good paths depend on two independent calls of the same random-looking condition, are not detected; we do not tune for the second kind because a path-insensitive analysis cannot tell it from its safe twin.
Comparison with other scanners
We have not yet run CodeQL or Semgrep on these corpora under identical conditions, so we publish no comparison numbers. We will add one when we have run both tools on the same corpus versions, with their default security rules, scored by the same scripts.
Planned method: CodeQL and Semgrep run on the same corpus versions with their default security query packs, scored by the same scripts and matching rules; we will publish categories where they beat us as prominently as those where we lead.
Methodology
What runs
The production scanner (runScan) — every language engine, cross-file analysis and the line-level detectors — on each corpus in batches, exactly as a pull request is scanned. No benchmark-specific configuration. Helper files a corpus ships next to its test cases are scanned with them, as a whole-repository scan would.
How it is scored
A labelled case is a true positive when a finding of that category's rules lands in the case (per file, per method for Juliet's bad/good regions); a safe case with such a finding is a false positive. Counts are of labelled samples, each either vulnerable or safe: a Juliet case has one of each, so it counts twice. The mapping from our rule ids to each corpus's categories is part of the published harness.
Reproducing it
The scorers and the probe harness are in the repository under benchmarks/, with the corpus versions and download links listed in its README. Engine commit 756c80d, 7b67b01+, 0e73164, b4505ab+, 528472b+ (per corpus above); report generated 2026-11-16.
What these numbers do not say
Benchmarks are small, synthetic programs. They measure whether known patterns are recognised, not how a scanner behaves on your codebase. Scores across different corpora are not comparable with each other.