Measured 2026-11-16 · engine 756c80d, 7b67b01+, 0e73164, b4505ab+, 528472b+

Benchmark report

How often the engines find real vulnerabilities, and how often they flag safe code, on public labelled benchmarks that anyone can download. Every number below comes from a run of the scorers against engine commit 756c80d, except where a corpus names the later engine it was re-measured on. Failures and limitations are listed alongside the wins. Download the machine-readable report.

Recall (TPR)

80.3%

32,571 of 40,553 labelled vulnerabilities, all corpora

False-positive rate

1.0%

411 of 40,771 safe samples flagged

Scan throughput

30 files/s

57,496 files across the timed runs. Measured in batch on a Windows developer workstation (single Node.js process, Jest harness) — a throughput figure, not end-to-end pull-request latency.

Pooled numbers weight large corpora more heavily; per-corpus and per-category results are below. Precision is not a single meaningful number across corpora with different positive/negative ratios, so each table lists TP, FN, FP and TN for you to compute it.

At a glance: it is not the same in every language

The pooled recall above averages very different results. It ranges from 46.6% (SAST Benchmark — Rust, Rust) to 100.0% (OWASP Benchmark (Python), Python). Read the row for the language you use, not only the headline.

CorpusLanguagesSamplesRecall (TPR)False-positive rate
OWASP Benchmark (Python)Python1,230100.0%0.8%
TrustLedger cross-file suite (internal)TypeScript, Python, Java, C#, Go, PHP72100.0%0.0%
OWASP Benchmark (Java)Java2,74094.0%1.4%
NIST Juliet (C#)C#7,99289.8%5.5%
NIST Juliet (C/C++)C, C++64,09079.8%0.0%
SAST Benchmark — PHPPHP1,25067.0%6.6%
SAST Benchmark — GoGo1,35065.3%5.8%
SAST Benchmark — RubyRuby1,35063.4%6.8%
SAST Benchmark — RustRust1,25046.6%6.9%

Results by corpus, language and category

Open a corpus to see every vulnerability category. A category with a low score is a real gap — the roadmap lists which ones we are working on.
SAST Benchmark — Go v0.5.1Go · 1,350 labelled samples · run 2026-11-16 · engine 7b67b01+score 59.6TPR 65.3%FPR 5.8%show categories ▾

Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 58.1.

CategoryTPFNFPTNTPRFPRScore
authnfailure111412444.0%4.0%40
authzfailure52022320.0%8.0%12
cmdi22882273.3%26.7%46.7
codeinj20502580.0%0.0%80
csrf101502540.0%0.0%40
deserial81722332.0%8.0%24
fileupload151022360.0%8.0%52
hardcodedcreds17802568.0%0.0%68
infodisclosure121302548.0%0.0%48
inputval91602536.0%0.0%36
ldapi17802568.0%0.0%68
loginjection20512480.0%4.0%76
nosql16902564.0%0.0%64
pathtraver27352590.0%16.7%73.3
race_condition81702532.0%0.0%32
redirect17822368.0%8.0%60
securecookie21402584.0%0.0%84
sqli56936286.2%4.6%81.5
ssrf17832268.0%12.0%56
tlsverify16922364.0%8.0%56
trustbound20522380.0%8.0%72
weakcipher22302588.0%0.0%88
weakhash22322388.0%8.0%80
weakrand151002560.0%0.0%60
xss18742172.0%16.0%56
SAST Benchmark — PHP v0.3.2PHP · 1,250 labelled samples · run 2026-11-16 · engine 7b67b01+score 60.5TPR 67.0%FPR 6.6%show categories ▾

Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 60.5.

CategoryTPFNFPTNTPRFPRScore
cmdi18712472.0%4.0%68
codeinj21412484.0%4.0%80
csrf1240254.0%0.0%4
deserial18722372.0%8.0%64
extract71812428.0%4.0%24
fileinclusion22322388.0%8.0%80
fileupload22302588.0%0.0%88
hardcodedcreds16902564.0%0.0%64
headerinj17812468.0%4.0%64
ldapi23212492.0%4.0%88
massassign141122356.0%8.0%48
pathtraver21452084.0%20.0%64
redirect19642176.0%16.0%60
securecookie111402544.0%0.0%44
sqli24122396.0%8.0%88
ssrf205101580.0%40.0%40
ssti121312448.0%4.0%44
typejuggling61902524.0%0.0%24
unsafereflect21422384.0%8.0%76
variablevars17802568.0%0.0%68
weakcipher22312488.0%4.0%84
weakhash17802568.0%0.0%68
weakrand21402584.0%0.0%84
xss24122396.0%8.0%88
xxe52032220.0%12.0%8
SAST Benchmark — Ruby v0.3.2Ruby · 1,350 labelled samples · run 2026-11-16 · engine 7b67b01+score 56.6TPR 63.4%FPR 6.8%show categories ▾

Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 56.6.

CategoryTPFNFPTNTPRFPRScore
authnfailure42112416.0%4.0%12
authzfailure0250250.0%0.0%0
cmdi17822368.0%8.0%60
codeinj21422384.0%8.0%76
csrf0250250.0%0.0%0
deserial17802568.0%0.0%68
dynmethod18712472.0%4.0%68
fileinclusion23222392.0%8.0%84
fileupload141122356.0%8.0%48
hardcodedcreds151002560.0%0.0%60
headerinj17822368.0%8.0%60
ldapi21402584.0%0.0%84
loginjection18732272.0%12.0%60
massassign18702572.0%0.0%72
pathtraver22352088.0%20.0%68
redirect22381788.0%32.0%56
regex151032260.0%12.0%48
securecookie91612436.0%4.0%32
sqli250025100.0%0.0%100
ssrf19652076.0%20.0%56
ssti141102556.0%0.0%56
unsafereflect21432284.0%12.0%72
weakcipher24132296.0%12.0%84
weakhash18702572.0%0.0%72
weakrand20502580.0%0.0%80
xss16932264.0%12.0%52
xxe0250250.0%0.0%0
SAST Benchmark — Rust v0.5.2Rust · 1,250 labelled samples · run 2026-11-16 · engine 7b67b01+score 39.7TPR 46.6%FPR 6.9%show categories ▾

Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file. Source: https://github.com/0llino/sast-benchmark. Category-averaged score: 39.7.

CategoryTPFNFPTNTPRFPRScore
authnfailure32202512.0%0.0%12
authzfailure0250250.0%0.0%0
cmdi23261992.0%24.0%68
crypto121352048.0%20.0%28
csrf0250250.0%0.0%0
deserial0250250.0%0.0%0
fileupload19642176.0%16.0%60
hardcodedcreds151002560.0%0.0%60
infodisclosure1240254.0%0.0%4
inputval1240254.0%0.0%4
intoverflow21412484.0%4.0%80
ldapi19661976.0%24.0%52
loginjection21452084.0%20.0%64
memsafety17802568.0%0.0%68
nosql0250250.0%0.0%0
pathtraver24102596.0%0.0%96
race_condition32202512.0%0.0%12
redirect23281792.0%32.0%60
redos17812468.0%4.0%64
securecookie18712472.0%4.0%68
sqli22312488.0%4.0%84
ssrf2230258.0%0.0%8
tlsverify61902524.0%0.0%24
weakrand0250250.0%0.0%0
xss24152096.0%20.0%76
OWASP Benchmark (Java) v1.2Java · 2,740 labelled samples · run 2026-11-03 · engine 0e73164score 92.6TPR 94.0%FPR 1.4%show categories ▾

Each test case is one servlet labelled vulnerable or safe for a category; helper classes are scanned alongside, as in a whole-repository scan. Source: https://github.com/OWASP-Benchmark/BenchmarkJava. Category-averaged score: 93.5.

CategoryTPFNFPTNTPRFPRScore
cmdi12600125100.0%0.0%100
crypto13000116100.0%0.0%100
hash8940010769.0%0.0%69
ldapi270032100.0%0.0%100
pathtraver11518013586.5%0.0%86.5
securecookie360031100.0%0.0%100
sqli272018214100.0%7.8%92.2
trustbound74904389.2%0.0%89.2
weakrand20018027591.7%0.0%91.7
xpathi150020100.0%0.0%100
xss24600209100.0%0.0%100
OWASP Benchmark (Python) v0.1Python · 1,230 labelled samples · run 2026-11-02 · engine b4505ab+score 99.2TPR 100.0%FPR 0.8%show categories ▾

Each test case is one Flask view labelled vulnerable or safe for a category; helper modules are scanned alongside. Source: https://github.com/OWASP-Benchmark/BenchmarkPython. Category-averaged score: 98.8.

CategoryTPFNFPTNTPRFPRScore
cmdi13007100.0%0.0%100
codeinj200231100.0%6.1%93.9
deserialization180333100.0%8.3%91.7
hash710080100.0%0.0%100
ldapi160013100.0%0.0%100
pathtraver6500103100.0%0.0%100
redirect130021100.0%0.0%100
securecookie240015100.0%0.0%100
sqli50011100.0%0.0%100
trustbound180019100.0%0.0%100
weakrand9900227100.0%0.0%100
xpathi5100135100.0%0.0%100
xss310157100.0%1.7%98.3
xxe80020100.0%0.0%100
NIST Juliet (C#) vSARDC# · 7,992 labelled samples · run 2026-11-01 · engine 756c80dscore 84.4TPR 89.8%FPR 5.5%show categories ▾

Every case has a vulnerable (bad) and a safe (good) region; a case counts as detected when a finding of its CWE's rules lands in the bad region, and as a false positive when one lands in a good region. Cases with a web or network source. Source: https://samate.nist.gov/SARD/test-suites. Category-averaged score: 85.

CategoryTPFNFPTNTPRFPRScore
CWE89_SQL_Injection/s013700101269100.0%27.3%72.7
CWE89_SQL_Injection/s02296081215100.0%27.4%72.6
CWE78_OS_Command_Injection19230022286.5%0.0%86.5
CWE80_XSS19230022286.5%0.0%86.5
CWE81_XSS_Error_Message19230022286.5%0.0%86.5
CWE83_XSS_Attribute19230022286.5%0.0%86.5
CWE23_Relative_Path_Traversal19725022288.7%0.0%88.7
CWE36_Absolute_Path_Traversal19725022288.7%0.0%88.7
CWE601_Open_Redirect19230022286.5%0.0%86.5
CWE113_HTTP_Response_Splitting/s01328421036088.6%2.7%85.9
CWE113_HTTP_Response_Splitting/s0226333828888.9%2.7%86.1
CWE90_LDAP_Injection19230022286.5%0.0%86.5
CWE643_Xpath_Injection19824621689.2%2.7%86.5
CWE94_Improper_Control_of_Generation_of_Code19824621689.2%2.7%86.5
CWE470_Unsafe_Reflection19230022286.5%0.0%86.5
CWE117_Improper_Output_Neutralization_for_Logs19824621689.2%2.7%86.5
NIST Juliet (C/C++) v1.3C, C++ · 64,090 labelled samples · run 2026-11-16 · engine 528472b+score 79.8TPR 79.8%FPR 0.0%show categories ▾

Every case has a vulnerable (bad) and a safe (good) function; a case counts as detected when a finding of its CWE's rules lands in a bad function, and as a false positive when one lands in a good function. Source: https://samate.nist.gov/SARD/test-suites. Category-averaged score: 84.

CategoryTPFNFPTNTPRFPRScore
CWE78_OS_Command_Injection44004000480091.7%0.0%91.7
CWE134_Uncontrolled_Format_String26402400288091.7%0.0%91.7
CWE23_Relative_Path_Traversal21003000240087.5%0.0%87.5
CWE36_Absolute_Path_Traversal21003000240087.5%0.0%87.5
CWE90_LDAP_Injection44040048091.7%0.0%91.7
CWE114_Process_Control440136057676.4%0.0%76.4
CWE242_Use_of_Inherently_Dangerous_Function180018100.0%0.0%100
CWE377_Insecure_Temporary_File14400144100.0%0.0%100
CWE415_Double_Free88082096291.5%0.0%91.5
CWE416_Use_After_Free43821045995.4%0.0%95.4
CWE690_NULL_Deref_From_Return820140096085.4%0.0%85.4
CWE762_Mismatched_Memory_Management_Routines31823820356489.3%0.0%89.3
CWE789_Uncontrolled_Mem_Alloc672288096070.0%0.0%70
CWE680_Integer_Overflow_to_Buffer_Overflow336240057658.3%0.0%58.3
CWE121_Stack_Based_Buffer_Overflow321017340494464.9%0.0%64.9
CWE122_Heap_Based_Buffer_Overflow375621660592263.4%0.0%63.4
TrustLedger cross-file suite (internal) v756c80dTypeScript, Python, Java, C#, Go, PHP · 72 labelled samples · run 2026-11-01 · engine 756c80dscore 100TPR 100.0%FPR 0.0%show categories ▾

Our own suite: 72 small multi-file programs, each with a flow that crosses files (vulnerable) or a safe twin. Less independent than the public corpora, since we wrote it; included because no public corpus targets cross-file flows. Source: src/__tests__/benchmarks/crossFile in the TrustLedger repository. Category-averaged score: 100.

CategoryTPFNFPTNTPRFPRScore
csharp5004100.0%0.0%100
go7004100.0%0.0%100
java7004100.0%0.0%100
php8005100.0%0.0%100
python8005100.0%0.0%100
typescript10005100.0%0.0%100

VulnBench 2.0 (Snyk): 108 attacker-reachable vulnerabilities in 20 real applications

VulnBench is Snyk's open benchmark of source-to-sink vulnerabilities in 20 open-source applications (Go, JavaScript/TypeScript, Python, Java). We ran the production scanner on all 20 applications and scored it with the benchmark's published matching rule: the finding must be the right type and land within 2 lines of the labelled source or sink. Every one of the 108 outcomes is listed below, including the 20 we missed.

Attacker-reachable recall

88 / 108

81.5% of the vulnerabilities; 80.1% averaged over the 20 applications, the metric the benchmark ranks by

Looser check

93 / 108

86.1% when any labelled location within 2 lines counts. A sensitivity check, not the headline

Precision lower bound

40.7%

88 of 216 findings of the scored types matched a labelled vulnerability; the rest may be real but unlabelled

Recall by vulnerability typeshow ▾
TypeReferenceMatchedRecallLooser
xss19947.4%12
path-traversal161593.8%15
command-injection131292.3%12
sql-injection131292.3%13
open-redirect11981.8%9
ssrf9777.8%8
prototype-pollution66100.0%6
insecure-deserialization55100.0%5
template-injection4375.0%3
code-injection44100.0%4
redos3133.3%1
mass-assignment22100.0%2
improper-type-validation22100.0%2
idor11100.0%1
Proof: all 108 vulnerabilities, with the finding that matched eachshow ▾

Each row is one labelled vulnerability: the application, its type, the labelled source/sink location, and the rule, file and line of the TrustLedger finding that matched it. The same list is machine-readable at /benchmarks/vulnbench.json.

AppTypeLabelled locationResultMatched by
ardanservicecommand-injectionapp/domain/checkapp/diagnostics.go:10foundcommand-injection · app/domain/checkapp/checkapp.go:55
ardanservicesql-injectionapp/domain/userapp/department_search.go:12foundsql-injection · app/domain/userapp/userapp.go:184
ardanservicessrfapp/domain/userapp/webhook.go:9foundssrf · app/domain/userapp/userapp.go:163
ardanserviceopen-redirectapp/domain/userapp/redirect.go:9foundopen-redirect · app/domain/userapp/userapp.go:174
ardanservicepath-traversalapp/domain/auditapp/reports.go:12foundpath-traversal · app/domain/auditapp/auditapp.go:66
blanqxssapp/components/SharedPathTooltip.vue:24foundxss · app/components/SharedPathTooltip.vue:24
blanqxssapp/composables/useCampaignMessage.ts:16foundxss · app/composables/useCampaignMessage.ts:16
blanqxssapp/composables/useReferralSource.ts:16foundxss · app/composables/useReferralSource.ts:16
blanqprototype-pollutionapp/composables/useSharedPreferences.ts:14foundprototype-pollution · app/composables/useSharedPreferences.ts:14
blanqprototype-pollutionapp/composables/useSharedPreferences.ts:22foundprototype-pollution · app/composables/useSharedPreferences.ts:22
blanqredosserver/api/auth/email-policy.get.js:10missed—
blanqssrfserver/api/integrations/logo-preview.get.js:15missed—
boilwatercommand-injectionconfig/dns-diagnostics.js:9foundcommand-injection · controllers/api.js:1751
boilwateropen-redirectconfig/redirect-handler.js:1foundopen-redirect · controllers/api.js:1769
boilwaterpath-traversalconfig/document-downloads.js:6foundpath-traversal · controllers/api.js:1778
boilwateropen-redirectcontrollers/user.js:649foundopen-redirect · controllers/user.js:649
coffeeshopsql-injectionsrc/main/java/nl/scottjohnson/coffeeshop/repository/SearchRepository.java:25foundsql-injection · src/main/java/nl/scottjohnson/coffeeshop/controller/HomeController.java:30
coffeeshoppath-traversalsrc/main/java/nl/scottjohnson/coffeeshop/controller/UploadController.java:34foundpath-traversal · src/main/java/nl/scottjohnson/coffeeshop/controller/UploadController.java:34
coffeeshopxsssrc/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:91foundxss · src/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:91
coffeeshopxsssrc/main/java/nl/scottjohnson/coffeeshop/controller/ProductController.java:101missed—
coffeeshoptemplate-injectionsrc/main/resources/templates/fragments/productsTable.html:45missed—
conduitsql-injectionconduit/articles/search.py:19foundsql-injection · conduit/articles/views.py:183
conduitpath-traversalconduit/articles/export.py:19foundpath-traversal · conduit/articles/views.py:197
conduitcommand-injectionconduit/articles/pdf.py:16foundcommand-injection · conduit/articles/views.py:208
conduitinsecure-deserializationconduit/articles/views.py:229foundinsecure-deserialization · conduit/articles/views.py:229
conduitcode-injectionconduit/articles/views.py:244foundeval-exec · conduit/articles/views.py:244
conduitssrfconduit/user/avatar.py:8foundssrf · conduit/user/views.py:66
flask-bonesopen-redirectapp/auth/views.py:36foundopen-redirect · app/auth/views.py:36
flask-bonesopen-redirectapp/auth/views.py:120foundopen-redirect · app/auth/views.py:120
flask-bonescommand-injectionapp/utils.py:18foundcommand-injection · app/user/views.py:62
flask-bonescommand-injectionapp/utils.py:23foundcommand-injection · app/user/views.py:72
flask-bonesinsecure-deserializationapp/user/views.py:25foundinsecure-deserialization · app/user/views.py:25
flask-bonesinsecure-deserializationapp/auth/views.py:67foundinsecure-deserialization · app/auth/views.py:67
flask-bonespath-traversalapp/user/reports.py:9foundpath-traversal · app/user/views.py:82
flask-bonespath-traversalapp/user/reports.py:16foundpath-traversal · app/user/views.py:92
flask-bonesmass-assignmentapp/user/views.py:102foundmass-assignment · app/user/views.py:102
flask-bonesidorapp/user/views.py:115foundbola-missing-ownership-check · app/user/views.py:114
go-samplesql-injectiondelivery/http/search.go:23foundsql-injection · delivery/http/api.go:33
go-samplesql-injectiondelivery/http/search.go:52foundsql-injection · delivery/http/api.go:51
go-samplesql-injectiondelivery/http/search.go:85foundsql-injection · delivery/http/api.go:79
go-samplexssdelivery/http/report.go:28missed—
go-samplexssdelivery/http/report.go:39missed—
gobasecommand-injectioninternal/gateways/web/controllers/apiv1/workspace/diagnostics.go:10foundcommand-injection · internal/gateways/web/controllers/apiv1/workspace/controller.go:37
gobaseopen-redirectinternal/gateways/web/controllers/apiv1/workspace/navigation.go:8foundopen-redirect · internal/gateways/web/controllers/apiv1/workspace/controller.go:30
gobasepath-traversalinternal/gateways/web/controllers/apiv1/workspace/reports.go:12foundpath-traversal · internal/gateways/web/controllers/apiv1/workspace/controller.go:49
gobasessrfinternal/gateways/web/controllers/apiv1/workspace/branding.go:9foundssrf · internal/gateways/web/controllers/apiv1/workspace/controller.go:62
gobasetemplate-injectioninternal/gateways/web/controllers/apiv1/workspace/templates.go:9foundssti · internal/gateways/web/controllers/apiv1/workspace/controller.go:75
goxygencommand-injectionserver/web/diagnostics.go:10foundcommand-injection · server/web/diagnostics_handler.go:7
goxygenopen-redirectserver/web/navigation_redirect.go:6foundopen-redirect · server/web/navigation.go:10
goxygenpath-traversalserver/web/report_files.go:10foundpath-traversal · server/web/reports.go:9
goxygenssrfserver/web/branding_assets.go:9foundssrf · server/web/branding.go:9
goxygenxsswebapp/src/App.vue:48foundxss · webapp/src/App.vue:48
goxygenprototype-pollutionwebapp/src/workspacePreferences.js:14foundprototype-pollution · webapp/src/workspacePreferences.js:12
halloweenimproper-type-validationsrc/api/user/userController.ts:17foundtype-confusion · src/api/user/userController.ts:16
halloweenimproper-type-validationsrc/api/user/userController.ts:95foundtype-confusion · src/api/user/userController.ts:95
halloweenmass-assignmentsrc/api/user/userService.ts:87foundmass-assignment · src/api/user/userController.ts:51
iguanacommand-injectionsrc/gitintegration/frontend.py:45foundcommand-injection · src/gitintegration/views.py:123
iguanaredossrc/search/views.py:67foundredos · src/search/views.py:67
iguanaredossrc/search/frontend.py:133missed—
iguanasql-injectionsrc/issue/search.py:19foundsql-injection · src/issue/views.py:423
iguanacode-injectionsrc/issue/views.py:440foundeval-exec · src/issue/views.py:440
iguanapath-traversalsrc/project/export.py:21foundpath-traversal · src/project/views.py:129
keystonebankpath-traversalbackend/services/BankStatements.ts:14missed—
keystonebankprototype-pollutionbackend/routes/memos.ts:52foundprototype-pollution · backend/routes/memos.ts:52
keystonebanksql-injectionbackend/routes/finchat.ts:43partly (looser rule only)sql-injection · backend/routes/finchat.ts:43
keystonebankxssbackend/routes/bank_profile_template.ts:27missed—
lastsaasxssfrontend/src/pages/public/CustomPage.tsx:65partly (looser rule only)xss · frontend/src/pages/public/CustomPage.tsx:65
lastsaasopen-redirectfrontend/src/pages/auth/LoginPage.tsx:29foundopen-redirect · frontend/src/pages/auth/LoginPage.tsx:29
lastsaasssrfbackend/internal/api/handlers/branding_assets.go:19foundssrf · backend/internal/api/handlers/branding.go:160
lastsaascommand-injectionbackend/internal/health/diagnostics.go:11foundcommand-injection · backend/internal/api/handlers/health_diagnostics.go:17
lastsaaspath-traversalbackend/internal/api/handlers/branding_assets.go:43foundpath-traversal · backend/internal/api/handlers/branding.go:177
pygmysql-injectionpygmy/model/search.py:20foundsql-injection · pygmy/rest/user.py:114
pygmypath-traversalpygmy/utilities/export.py:20foundpath-traversal · pygmy/rest/user.py:147
pygmycommand-injectionpygmy/core/network.py:8foundcommand-injection · pygmy/rest/diagnostics.py:15
pygmycode-injectionpygmy/rest/user.py:135foundeval-exec · pygmy/rest/user.py:135
pygmyssrfpygmy/utilities/reachability.py:8partly (looser rule only)ssrf · pygmy/rest/shorturl.py:61
robs-awesomesql-injectionbackend/services/reports.py:9foundsql-injection · backend/www.py:56
robs-awesomepath-traversalbackend/www.py:99foundpath-traversal · backend/www.py:99
robs-awesomecommand-injectionbackend/www.py:108foundcommand-injection · backend/www.py:108
robs-awesomessrfbackend/services/integrations.py:6foundssrf · backend/www.py:118
robs-awesomeinsecure-deserializationbackend/services/drafts.py:8foundinsecure-deserialization · backend/www.py:80
robs-awesomeopen-redirectbackend/services/redirects.py:6missed—
robs-awesomecode-injectionbackend/www.py:70foundeval-exec · backend/www.py:70
saas-starter-kitcommand-injectioncmd/web-app/handlers/integration_diagnostics.go:11foundcommand-injection · cmd/web-app/handlers/integrations.go:20
saas-starter-kitopen-redirectcmd/web-app/handlers/user.go:108missed—
saas-starter-kitpath-traversalcmd/web-app/handlers/account_reports.go:10foundpath-traversal · cmd/web-app/handlers/account.go:80
saas-starter-kitsql-injectioncmd/web-api/handlers/checklist.go:379foundsql-injection · cmd/web-api/handlers/checklist.go:379
saas-starter-kitssrfcmd/web-app/handlers/account_assets.go:10foundssrf · cmd/web-app/handlers/account.go:65
saas-starter-kittemplate-injectioncmd/web-app/handlers/integrations.go:40foundssti · cmd/web-app/handlers/integrations.go:40
saas-starter-liteopen-redirectapps/api/src/auth/auth.controller.ts:66foundopen-redirect · apps/api/src/auth/auth.controller.ts:66
saas-starter-litetemplate-injectionapps/api/src/auth/auth.controller.ts:60foundssti · apps/api/src/auth/auth.controller.ts:60
saas-starter-litexssapps/api/src/auth/auth.controller.ts:55foundxss · apps/api/src/auth/auth.controller.ts:55
saas-starter-litexssapps/web/src/app/pages/auth/register/register.ts:148partly (looser rule only)xss · apps/web/src/app/pages/auth/register/register.ts:148
saas-starter-litecommand-injectionapps/api/src/users/users.service.ts:39missed—
saas-starter-litesql-injectionapps/api/src/users/users.service.ts:55foundsql-injection · apps/api/src/users/users.controller.ts:91
sassyregxsssrc/views/examples/PackageProfile.js:168foundxss · src/views/examples/PackageProfile.js:168
sassyregxsssrc/views/examples/PackageProfile.js:325partly (looser rule only)xss · src/views/examples/PackageProfile.js:322
slingshotxsssrc/components/AboutPage.js:14foundxss · src/components/AboutPage.js:14
slingshotxsssrc/components/HomePage.js:13foundxss · src/components/HomePage.js:13
slingshotprototype-pollutionsrc/utils/sharedPreferences.js:10foundprototype-pollution · src/utils/sharedPreferences.js:10
slingshotprototype-pollutionsrc/utils/sharedPreferences.js:26foundprototype-pollution · src/utils/sharedPreferences.js:26
vinyl-marketplacesql-injectionsrc/main/java/io/acme/engineering/vynil_marketplace/service/VinylService.java:41foundsql-injection · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:42
vinyl-marketplaceinsecure-deserializationsrc/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:87foundinsecure-deserialization · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:88
vinyl-marketplacepath-traversalsrc/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:128foundpath-traversal · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:128
vinyl-marketplacepath-traversalsrc/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:167foundpath-traversal · src/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:167
vinyl-marketplacexsssrc/main/resources/static/index.html:297missed—
vinyl-marketplacexsssrc/main/resources/static/index.html:298missed—
vinyl-marketplacexsssrc/main/java/io/acme/engineering/vynil_marketplace/controller/VinylApiController.java:118missed—

How this compares with what Snyk published

Attacker-reachable vulnerability recall as printed on the release page; we did not re-run these tools. Source: https://vulnbench.com/releases/2.0, copied 2026-11-12.

ScannerMatchedRecall (macro)Precision lower bound
TrustLedger (this run)8880.1%40.7%
Snyk Code8982.4%20.9%
Claude Code /security-review, Opus 5 xhigh8579.3%74.6%
Claude Code, Sonnet 5 xhigh7367.0%74.5%
Codex Security, GPT-5.6 Luna4541.2%20.8%

Read this as a comparison of one run against published numbers, not a leaderboard entry. We scored with our own implementation of the published rule, not Snyk's harness; their results carry resampling intervals over the 20 applications that ours does not, and their precision figures are computed over a different set of reported findings. The scorer, the probe and the pinned corpus commit (385a375) are in the repository under benchmarks/, so the run can be repeated. Engine commit 23da128, measured 2026-11-12.

AI-attribution validation

The AI-generated estimate is checked against a small labelled corpus in the repository (10 files, TypeScript/JavaScript only). It is a regression check, not a statistically strong evaluation — treat these numbers as indicative, and the estimate itself as a probability, not proof of authorship.

Precision

100.0%

Recall

60.0%

Accuracy

80.0%

threshold 0.5

ROC-AUC

1.00

Tool attribution

100.0%

10 labelled samples

Read the method and its limits in the AI attribution documentation.

False positives and false negatives we know about

Representative cases from the corpora above, with the reason. Some are engine gaps we are fixing; some are deliberate choices where we think the corpus label is wrong for real-world code — we say which.

False positives — flagged, but safe

sql-injectionOWASP Benchmark (Java) · BenchmarkTest02274

A switch on a constant character

The helper switches on "ABC".charAt(1), which is always 'B', and that arm assigns a constant; the arms that assign the request value never run. The engine does not evaluate the switch subject, so it keeps every arm and the line-level concatenation match stands. Engine gap; most of the remaining BenchmarkJava SQL false positives are constant-folding tricks of this kind (constant switches and if-conditions, fixed list and map positions).

log-injectionSAST Benchmark — Rust · BenchmarkTest00158

A sanitizer written as a character filter

The username is cleaned with chars().filter(|c| !c.is_ascii_control()), which removes CR and LF, but the engine does not recognise a closure-based filter as a log sanitizer, so the log line is still reported. Engine gap.

idorSAST Benchmark — Go · BenchmarkTest01167

Ownership proven by an HMAC-signed token

The handler verifies an HMAC-signed resource token that binds the caller to the resource id before the lookup. The engine recognises comparisons with the authenticated principal but not a signature check, so it reports the lookup. Engine gap.

ssrfSAST Benchmark — Ruby · BenchmarkTest00997

An https-only check is labelled safe

The corpus treats a request URL as safe once its scheme is https. Restricting the scheme does not stop a request to an internal host (https://169.254.169.254/, an internal service), so we keep reporting it. A deliberate disagreement with the label.

weak-signing-secretSAST Benchmark — Go · BenchmarkTest01128

A signing-method check does not make a literal key safe

The test checks that the token uses an HMAC method and then verifies it with the key []byte("supersecret"). The corpus labels it safe because the algorithm is pinned; we still report it, because a key written in the source lets anyone mint a token the parser accepts.

insecure-deserializationSAST Benchmark — PHP · BenchmarkTest00985

An instanceof check after unserialize is too late

A request value is passed to igbinary_unserialize and the result is then checked with instanceof. The corpus labels it safe; we report it, because the object is already built (and its destructors and wakeup methods run) before the check, so a gadget chain does not need the type to match.

False negatives — missed

weak-cryptoOWASP Benchmark (Java) · BenchmarkTest00003

A hash algorithm chosen in a properties file

The algorithm name is read with getProperty("hashAlg1", ...) from benchmark.properties, where it is set to a weak hash. The engine does not resolve values from .properties files, so every hash case configured this way is missed. Engine gap; on the roadmap.

ssrfSAST Benchmark — Rust · BenchmarkTest00125

A stub HTTP client

The request URL reaches simulated_reqwest_get(), a function the corpus defines to stand in for an HTTP client and which only formats a string. We report calls to real HTTP libraries, not functions that merely have a client-like name, so these cases count as misses. A deliberate choice.

xxeSAST Benchmark — Ruby · BenchmarkTest01120

Nokogiri's default parse is labelled vulnerable

Nokogiri::XML(input) does not substitute external entities by default; the corpus labels it vulnerable and labels the noent option — which turns substitution on — as safe. We follow the library's real behaviour, so Ruby XXE scores 0 on this corpus. A deliberate disagreement with the label.

csrfSAST Benchmark — PHP · BenchmarkTest00053

CSRF without a session or cookie

The corpus labels a state-changing handler with no CSRF token as vulnerable even when it reads no session or cookie. Cross-site request forgery needs a credential the browser attaches by itself, so we report CSRF only for session- or cookie-authenticated handlers. A deliberate choice.

weak-cryptoOWASP Juice Shop · weakPasswordChallenge

A weak password hash behind a helper in another file

models/user.ts stores security.hash(password), and the weak MD5 hash lives in lib/insecurity.ts. The weak hash itself is reported there, but not the use of it for passwords in the model. Engine gap.

insecure-deserializationSAST Benchmark — Go · BenchmarkTest00250

yaml.Unmarshal into interface{} is labelled vulnerable

The corpus files decoding a request body with gopkg.in/yaml.v3 into an interface{} under deserialization. Go's YAML decoder builds plain maps, slices and scalars and runs no code, so there is no gadget to reach; we do not report it, and score it as a miss rather than change the label.

insecure-deserializationSAST Benchmark — Rust · BenchmarkTest00060

A same-file function named like a deserializer

The test calls deserialize_object(body.as_bytes()), a function defined a few lines below that returns a constant. Nothing in the file reads or builds an object, and we do not treat a locally defined function as a library sink because of its name, so the case counts as a miss.

Known limitations

  • Benchmarks are small, synthetic programs. They measure whether known patterns are recognised, not how the engines behave on your codebase, and scores on different corpora are not comparable with each other.
  • Trace accuracy is not measured: no public corpus labels the expected source-to-sink path.
  • Kotlin has no public labelled corpus and is not in this report; it is covered by unit tests and checked against the AndroGoat and Goatlin apps by hand.
  • Juliet cases are scored by region (a finding in the bad or good function); Juice Shop challenges count as detected when a finding lands within two lines of a marked line.
  • The internal cross-file suite was written by us. It is included because no public corpus targets cross-file flows, but it is less independent than the public corpora.
  • The AI-attribution check uses 10 labelled TypeScript/JavaScript files: a regression check, not a statistically strong evaluation.
  • Throughput is measured in batch on a developer workstation, on the corpora scanned in a single pass. It is not end-to-end pull-request latency.
  • Where we disagree with a corpus label we keep the label in the score and explain the disagreement in the examples above, rather than changing the label.
  • Many Rust cases, and some PHP and Ruby ones, call a function defined in the same file that only imitates a library call (simulated_reqwest_get, aes_ecb_encrypt, native_tls_danger). We scan the code as written and do not treat a locally defined function as a library sink because of its name, so those cases count as misses; this is the main reason Rust's recall is low, and it says little about code that calls the real libraries.
  • The sast-benchmark and Juliet C/C++ corpora were re-run on 2026-11-16. The OWASP Benchmark (Java, Python), Juliet C# and Juice Shop corpora were not available for that run, so each of them keeps the engine commit named against it rather than being presented as measured on today's engine.
  • Juliet C/C++: cases that call virtual methods through a base class defined in another file, and the paired cases whose bad and good paths depend on two independent calls of the same random-looking condition, are not detected; we do not tune for the second kind because a path-insensitive analysis cannot tell it from its safe twin.

Comparison with other scanners

We have not yet run CodeQL or Semgrep on these corpora under identical conditions, so we publish no comparison numbers. We will add one when we have run both tools on the same corpus versions, with their default security rules, scored by the same scripts.

Planned method: CodeQL and Semgrep run on the same corpus versions with their default security query packs, scored by the same scripts and matching rules; we will publish categories where they beat us as prominently as those where we lead.

Methodology

What runs

The production scanner (runScan) — every language engine, cross-file analysis and the line-level detectors — on each corpus in batches, exactly as a pull request is scanned. No benchmark-specific configuration. Helper files a corpus ships next to its test cases are scanned with them, as a whole-repository scan would.

How it is scored

A labelled case is a true positive when a finding of that category's rules lands in the case (per file, per method for Juliet's bad/good regions); a safe case with such a finding is a false positive. Counts are of labelled samples, each either vulnerable or safe: a Juliet case has one of each, so it counts twice. The mapping from our rule ids to each corpus's categories is part of the published harness.

Reproducing it

The scorers and the probe harness are in the repository under benchmarks/, with the corpus versions and download links listed in its README. Engine commit 756c80d, 7b67b01+, 0e73164, b4505ab+, 528472b+ (per corpus above); report generated 2026-11-16.

What these numbers do not say

Benchmarks are small, synthetic programs. They measure whether known patterns are recognised, not how a scanner behaves on your codebase. Scores across different corpora are not comparable with each other.