{"generatedAt":"2026-11-16T15:50:08.713Z","engineCommit":"756c80d","environment":"Measured in batch on a Windows developer workstation (single Node.js process, Jest harness) — a throughput figure, not end-to-end pull-request latency.","corpora":[{"id":"sast-benchmark-go","name":"SAST Benchmark — Go","url":"https://github.com/0llino/sast-benchmark","version":"0.5.1","languages":["Go"],"scoring":"Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file.","runAt":"2026-11-16T12:00:00.000Z","files":1352,"wallMs":77225,"categories":[{"name":"authnfailure","tp":11,"fn":14,"fp":1,"tn":24},{"name":"authzfailure","tp":5,"fn":20,"fp":2,"tn":23},{"name":"cmdi","tp":22,"fn":8,"fp":8,"tn":22},{"name":"codeinj","tp":20,"fn":5,"fp":0,"tn":25},{"name":"csrf","tp":10,"fn":15,"fp":0,"tn":25},{"name":"deserial","tp":8,"fn":17,"fp":2,"tn":23},{"name":"fileupload","tp":15,"fn":10,"fp":2,"tn":23},{"name":"hardcodedcreds","tp":17,"fn":8,"fp":0,"tn":25},{"name":"infodisclosure","tp":12,"fn":13,"fp":0,"tn":25},{"name":"inputval","tp":9,"fn":16,"fp":0,"tn":25},{"name":"ldapi","tp":17,"fn":8,"fp":0,"tn":25},{"name":"loginjection","tp":20,"fn":5,"fp":1,"tn":24},{"name":"nosql","tp":16,"fn":9,"fp":0,"tn":25},{"name":"pathtraver","tp":27,"fn":3,"fp":5,"tn":25},{"name":"race_condition","tp":8,"fn":17,"fp":0,"tn":25},{"name":"redirect","tp":17,"fn":8,"fp":2,"tn":23},{"name":"securecookie","tp":21,"fn":4,"fp":0,"tn":25},{"name":"sqli","tp":56,"fn":9,"fp":3,"tn":62},{"name":"ssrf","tp":17,"fn":8,"fp":3,"tn":22},{"name":"tlsverify","tp":16,"fn":9,"fp":2,"tn":23},{"name":"trustbound","tp":20,"fn":5,"fp":2,"tn":23},{"name":"weakcipher","tp":22,"fn":3,"fp":0,"tn":25},{"name":"weakhash","tp":22,"fn":3,"fp":2,"tn":23},{"name":"weakrand","tp":15,"fn":10,"fp":0,"tn":25},{"name":"xss","tp":18,"fn":7,"fp":4,"tn":21}],"engineCommit":"7b67b01+"},{"id":"sast-benchmark-php","name":"SAST Benchmark — PHP","url":"https://github.com/0llino/sast-benchmark","version":"0.3.2","languages":["PHP"],"scoring":"Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file.","runAt":"2026-11-16T12:00:00.000Z","files":1251,"wallMs":17530,"categories":[{"name":"cmdi","tp":18,"fn":7,"fp":1,"tn":24},{"name":"codeinj","tp":21,"fn":4,"fp":1,"tn":24},{"name":"csrf","tp":1,"fn":24,"fp":0,"tn":25},{"name":"deserial","tp":18,"fn":7,"fp":2,"tn":23},{"name":"extract","tp":7,"fn":18,"fp":1,"tn":24},{"name":"fileinclusion","tp":22,"fn":3,"fp":2,"tn":23},{"name":"fileupload","tp":22,"fn":3,"fp":0,"tn":25},{"name":"hardcodedcreds","tp":16,"fn":9,"fp":0,"tn":25},{"name":"headerinj","tp":17,"fn":8,"fp":1,"tn":24},{"name":"ldapi","tp":23,"fn":2,"fp":1,"tn":24},{"name":"massassign","tp":14,"fn":11,"fp":2,"tn":23},{"name":"pathtraver","tp":21,"fn":4,"fp":5,"tn":20},{"name":"redirect","tp":19,"fn":6,"fp":4,"tn":21},{"name":"securecookie","tp":11,"fn":14,"fp":0,"tn":25},{"name":"sqli","tp":24,"fn":1,"fp":2,"tn":23},{"name":"ssrf","tp":20,"fn":5,"fp":10,"tn":15},{"name":"ssti","tp":12,"fn":13,"fp":1,"tn":24},{"name":"typejuggling","tp":6,"fn":19,"fp":0,"tn":25},{"name":"unsafereflect","tp":21,"fn":4,"fp":2,"tn":23},{"name":"variablevars","tp":17,"fn":8,"fp":0,"tn":25},{"name":"weakcipher","tp":22,"fn":3,"fp":1,"tn":24},{"name":"weakhash","tp":17,"fn":8,"fp":0,"tn":25},{"name":"weakrand","tp":21,"fn":4,"fp":0,"tn":25},{"name":"xss","tp":24,"fn":1,"fp":2,"tn":23},{"name":"xxe","tp":5,"fn":20,"fp":3,"tn":22}],"engineCommit":"7b67b01+"},{"id":"sast-benchmark-ruby","name":"SAST Benchmark — Ruby","url":"https://github.com/0llino/sast-benchmark","version":"0.3.2","languages":["Ruby"],"scoring":"Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file.","runAt":"2026-11-16T12:00:00.000Z","files":1351,"wallMs":22221,"categories":[{"name":"authnfailure","tp":4,"fn":21,"fp":1,"tn":24},{"name":"authzfailure","tp":0,"fn":25,"fp":0,"tn":25},{"name":"cmdi","tp":17,"fn":8,"fp":2,"tn":23},{"name":"codeinj","tp":21,"fn":4,"fp":2,"tn":23},{"name":"csrf","tp":0,"fn":25,"fp":0,"tn":25},{"name":"deserial","tp":17,"fn":8,"fp":0,"tn":25},{"name":"dynmethod","tp":18,"fn":7,"fp":1,"tn":24},{"name":"fileinclusion","tp":23,"fn":2,"fp":2,"tn":23},{"name":"fileupload","tp":14,"fn":11,"fp":2,"tn":23},{"name":"hardcodedcreds","tp":15,"fn":10,"fp":0,"tn":25},{"name":"headerinj","tp":17,"fn":8,"fp":2,"tn":23},{"name":"ldapi","tp":21,"fn":4,"fp":0,"tn":25},{"name":"loginjection","tp":18,"fn":7,"fp":3,"tn":22},{"name":"massassign","tp":18,"fn":7,"fp":0,"tn":25},{"name":"pathtraver","tp":22,"fn":3,"fp":5,"tn":20},{"name":"redirect","tp":22,"fn":3,"fp":8,"tn":17},{"name":"regex","tp":15,"fn":10,"fp":3,"tn":22},{"name":"securecookie","tp":9,"fn":16,"fp":1,"tn":24},{"name":"sqli","tp":25,"fn":0,"fp":0,"tn":25},{"name":"ssrf","tp":19,"fn":6,"fp":5,"tn":20},{"name":"ssti","tp":14,"fn":11,"fp":0,"tn":25},{"name":"unsafereflect","tp":21,"fn":4,"fp":3,"tn":22},{"name":"weakcipher","tp":24,"fn":1,"fp":3,"tn":22},{"name":"weakhash","tp":18,"fn":7,"fp":0,"tn":25},{"name":"weakrand","tp":20,"fn":5,"fp":0,"tn":25},{"name":"xss","tp":16,"fn":9,"fp":3,"tn":22},{"name":"xxe","tp":0,"fn":25,"fp":0,"tn":25}],"engineCommit":"7b67b01+"},{"id":"sast-benchmark-rust","name":"SAST Benchmark — Rust","url":"https://github.com/0llino/sast-benchmark","version":"0.5.2","languages":["Rust"],"scoring":"Each test case is one file labelled vulnerable or safe for a category; a case is flagged when a finding of that category's rules is in its file.","runAt":"2026-11-16T12:00:00.000Z","files":1251,"wallMs":35073,"categories":[{"name":"authnfailure","tp":3,"fn":22,"fp":0,"tn":25},{"name":"authzfailure","tp":0,"fn":25,"fp":0,"tn":25},{"name":"cmdi","tp":23,"fn":2,"fp":6,"tn":19},{"name":"crypto","tp":12,"fn":13,"fp":5,"tn":20},{"name":"csrf","tp":0,"fn":25,"fp":0,"tn":25},{"name":"deserial","tp":0,"fn":25,"fp":0,"tn":25},{"name":"fileupload","tp":19,"fn":6,"fp":4,"tn":21},{"name":"hardcodedcreds","tp":15,"fn":10,"fp":0,"tn":25},{"name":"infodisclosure","tp":1,"fn":24,"fp":0,"tn":25},{"name":"inputval","tp":1,"fn":24,"fp":0,"tn":25},{"name":"intoverflow","tp":21,"fn":4,"fp":1,"tn":24},{"name":"ldapi","tp":19,"fn":6,"fp":6,"tn":19},{"name":"loginjection","tp":21,"fn":4,"fp":5,"tn":20},{"name":"memsafety","tp":17,"fn":8,"fp":0,"tn":25},{"name":"nosql","tp":0,"fn":25,"fp":0,"tn":25},{"name":"pathtraver","tp":24,"fn":1,"fp":0,"tn":25},{"name":"race_condition","tp":3,"fn":22,"fp":0,"tn":25},{"name":"redirect","tp":23,"fn":2,"fp":8,"tn":17},{"name":"redos","tp":17,"fn":8,"fp":1,"tn":24},{"name":"securecookie","tp":18,"fn":7,"fp":1,"tn":24},{"name":"sqli","tp":22,"fn":3,"fp":1,"tn":24},{"name":"ssrf","tp":2,"fn":23,"fp":0,"tn":25},{"name":"tlsverify","tp":6,"fn":19,"fp":0,"tn":25},{"name":"weakrand","tp":0,"fn":25,"fp":0,"tn":25},{"name":"xss","tp":24,"fn":1,"fp":5,"tn":20}],"engineCommit":"7b67b01+"},{"id":"owasp-benchmark-java","name":"OWASP Benchmark (Java)","url":"https://github.com/OWASP-Benchmark/BenchmarkJava","version":"1.2","languages":["Java"],"scoring":"Each test case is one servlet labelled vulnerable or safe for a category; helper classes are scanned alongside, as in a whole-repository scan.","runAt":"2026-11-03T05:42:26.5238820Z","files":null,"wallMs":null,"categories":[{"name":"cmdi","tp":126,"fn":0,"fp":0,"tn":125},{"name":"crypto","tp":130,"fn":0,"fp":0,"tn":116},{"name":"hash","tp":89,"fn":40,"fp":0,"tn":107},{"name":"ldapi","tp":27,"fn":0,"fp":0,"tn":32},{"name":"pathtraver","tp":115,"fn":18,"fp":0,"tn":135},{"name":"securecookie","tp":36,"fn":0,"fp":0,"tn":31},{"name":"sqli","tp":272,"fn":0,"fp":18,"tn":214},{"name":"trustbound","tp":74,"fn":9,"fp":0,"tn":43},{"name":"weakrand","tp":200,"fn":18,"fp":0,"tn":275},{"name":"xpathi","tp":15,"fn":0,"fp":0,"tn":20},{"name":"xss","tp":246,"fn":0,"fp":0,"tn":209}],"engineCommit":"0e73164"},{"id":"owasp-benchmark-python","name":"OWASP Benchmark (Python)","url":"https://github.com/OWASP-Benchmark/BenchmarkPython","version":"0.1","languages":["Python"],"scoring":"Each test case is one Flask view labelled vulnerable or safe for a category; helper modules are scanned alongside.","runAt":"2026-11-02T16:06:06.2994421Z","files":null,"wallMs":null,"categories":[{"name":"cmdi","tp":13,"fn":0,"fp":0,"tn":7},{"name":"codeinj","tp":20,"fn":0,"fp":2,"tn":31},{"name":"deserialization","tp":18,"fn":0,"fp":3,"tn":33},{"name":"hash","tp":71,"fn":0,"fp":0,"tn":80},{"name":"ldapi","tp":16,"fn":0,"fp":0,"tn":13},{"name":"pathtraver","tp":65,"fn":0,"fp":0,"tn":103},{"name":"redirect","tp":13,"fn":0,"fp":0,"tn":21},{"name":"securecookie","tp":24,"fn":0,"fp":0,"tn":15},{"name":"sqli","tp":5,"fn":0,"fp":0,"tn":11},{"name":"trustbound","tp":18,"fn":0,"fp":0,"tn":19},{"name":"weakrand","tp":99,"fn":0,"fp":0,"tn":227},{"name":"xpathi","tp":51,"fn":0,"fp":0,"tn":135},{"name":"xss","tp":31,"fn":0,"fp":1,"tn":57},{"name":"xxe","tp":8,"fn":0,"fp":0,"tn":20}],"engineCommit":"b4505ab+"},{"id":"juliet-cs","name":"NIST Juliet (C#)","url":"https://samate.nist.gov/SARD/test-suites","version":"SARD","languages":["C#"],"scoring":"Every case has a vulnerable (bad) and a safe (good) region; a case counts as detected when a finding of its CWE's rules lands in the bad region, and as a false positive when one lands in a good region. Cases with a web or network source.","runAt":"2026-11-01T08:47:45.0730520Z","files":null,"wallMs":null,"categories":[{"name":"CWE89_SQL_Injection/s01","tp":370,"fn":0,"fp":101,"tn":269},{"name":"CWE89_SQL_Injection/s02","tp":296,"fn":0,"fp":81,"tn":215},{"name":"CWE78_OS_Command_Injection","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE80_XSS","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE81_XSS_Error_Message","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE83_XSS_Attribute","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE23_Relative_Path_Traversal","tp":197,"fn":25,"fp":0,"tn":222},{"name":"CWE36_Absolute_Path_Traversal","tp":197,"fn":25,"fp":0,"tn":222},{"name":"CWE601_Open_Redirect","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE113_HTTP_Response_Splitting/s01","tp":328,"fn":42,"fp":10,"tn":360},{"name":"CWE113_HTTP_Response_Splitting/s02","tp":263,"fn":33,"fp":8,"tn":288},{"name":"CWE90_LDAP_Injection","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE643_Xpath_Injection","tp":198,"fn":24,"fp":6,"tn":216},{"name":"CWE94_Improper_Control_of_Generation_of_Code","tp":198,"fn":24,"fp":6,"tn":216},{"name":"CWE470_Unsafe_Reflection","tp":192,"fn":30,"fp":0,"tn":222},{"name":"CWE117_Improper_Output_Neutralization_for_Logs","tp":198,"fn":24,"fp":6,"tn":216}]},{"id":"juliet-c","name":"NIST Juliet (C/C++)","url":"https://samate.nist.gov/SARD/test-suites","version":"1.3","languages":["C","C++"],"scoring":"Every case has a vulnerable (bad) and a safe (good) function; a case counts as detected when a finding of its CWE's rules lands in a bad function, and as a false positive when one lands in a good function.","runAt":"2026-11-16T15:50:08.713Z","files":52291,"wallMs":1737150,"categories":[{"name":"CWE78_OS_Command_Injection","tp":4400,"fn":400,"fp":0,"tn":4800},{"name":"CWE134_Uncontrolled_Format_String","tp":2640,"fn":240,"fp":0,"tn":2880},{"name":"CWE23_Relative_Path_Traversal","tp":2100,"fn":300,"fp":0,"tn":2400},{"name":"CWE36_Absolute_Path_Traversal","tp":2100,"fn":300,"fp":0,"tn":2400},{"name":"CWE90_LDAP_Injection","tp":440,"fn":40,"fp":0,"tn":480},{"name":"CWE114_Process_Control","tp":440,"fn":136,"fp":0,"tn":576},{"name":"CWE242_Use_of_Inherently_Dangerous_Function","tp":18,"fn":0,"fp":0,"tn":18},{"name":"CWE377_Insecure_Temporary_File","tp":144,"fn":0,"fp":0,"tn":144},{"name":"CWE415_Double_Free","tp":880,"fn":82,"fp":0,"tn":962},{"name":"CWE416_Use_After_Free","tp":438,"fn":21,"fp":0,"tn":459},{"name":"CWE690_NULL_Deref_From_Return","tp":820,"fn":140,"fp":0,"tn":960},{"name":"CWE762_Mismatched_Memory_Management_Routines","tp":3182,"fn":382,"fp":0,"tn":3564},{"name":"CWE789_Uncontrolled_Mem_Alloc","tp":672,"fn":288,"fp":0,"tn":960},{"name":"CWE680_Integer_Overflow_to_Buffer_Overflow","tp":336,"fn":240,"fp":0,"tn":576},{"name":"CWE121_Stack_Based_Buffer_Overflow","tp":3210,"fn":1734,"fp":0,"tn":4944},{"name":"CWE122_Heap_Based_Buffer_Overflow","tp":3756,"fn":2166,"fp":0,"tn":5922}],"engineCommit":"528472b+"},{"id":"trustledger-cross-file","name":"TrustLedger cross-file suite (internal)","url":"src/__tests__/benchmarks/crossFile in the TrustLedger repository","version":"756c80d","languages":["TypeScript","Python","Java","C#","Go","PHP"],"scoring":"Our own suite: 72 small multi-file programs, each with a flow that crosses files (vulnerable) or a safe twin. Less independent than the public corpora, since we wrote it; included because no public corpus targets cross-file flows.","runAt":"2026-11-01T10:33:47.6357946Z","files":null,"wallMs":null,"categories":[{"name":"csharp","tp":5,"fn":0,"fp":0,"tn":4},{"name":"go","tp":7,"fn":0,"fp":0,"tn":4},{"name":"java","tp":7,"fn":0,"fp":0,"tn":4},{"name":"php","tp":8,"fn":0,"fp":0,"tn":5},{"name":"python","tp":8,"fn":0,"fp":0,"tn":5},{"name":"typescript","tp":10,"fn":0,"fp":0,"tn":5}]}],"juice":{"name":"OWASP Juice Shop","url":"https://github.com/juice-shop/juice-shop","version":"20.3.0","runAt":"2026-11-01T10:30:10.2904710Z","files":966,"wallMs":123834,"challenges":[{"id":"accessLogDisclosureChallenge","detected":true,"via":"directory-listing","location":"server.ts:302"},{"id":"adminSectionChallenge","detected":false,"via":null,"location":"frontend/src/app/app.routing.ts:79"},{"id":"changeProductChallenge","detected":false,"via":null,"location":"server.ts:398"},{"id":"chatbotGreedyInjectionChallenge","detected":false,"via":null,"location":"routes/chat.ts:179"},{"id":"chatbotPromptInjectionChallenge","detected":false,"via":null,"location":"routes/chat.ts:179"},{"id":"dbSchemaChallenge","detected":true,"via":"sql-injection","location":"routes/search.ts:23"},{"id":"directoryListingChallenge","detected":true,"via":"directory-listing","location":"server.ts:290"},{"id":"exposedMetricsChallenge","detected":true,"via":"unauthenticated-diagnostics","location":"server.ts:759"},{"id":"forgedReviewChallenge","detected":true,"via":"nosql-injection","location":"routes/updateProductReviews.ts:16"},{"id":"iacLeakedKeyChallenge","detected":true,"via":"hardcoded-secret","location":"infrastructure/terraform/networking.tf:171"},{"id":"localXssChallenge","detected":true,"via":"xss","location":"frontend/src/app/search-result/search-result.component.ts:146"},{"id":"loginAdminChallenge","detected":true,"via":"sql-injection","location":"routes/login.ts:34"},{"id":"loginBenderChallenge","detected":true,"via":"sql-injection","location":"routes/login.ts:34"},{"id":"loginJimChallenge","detected":true,"via":"sql-injection","location":"routes/login.ts:34"},{"id":"nftMintChallenge","detected":false,"via":null,"location":"data/static/web3-snippets/HoneyPotNFT.sol:26"},{"id":"nftUnlockChallenge","detected":false,"via":null,"location":"data/static/web3-snippets/JuiceShopSBT.sol:18"},{"id":"noSqlReviewsChallenge","detected":true,"via":"nosql-injection","location":"routes/updateProductReviews.ts:18"},{"id":"redirectChallenge","detected":true,"via":"open-redirect","location":"lib/insecurity.ts:136"},{"id":"redirectCryptoCurrencyChallenge","detected":false,"via":null,"location":"lib/insecurity.ts:124"},{"id":"registerAdminChallenge","detected":false,"via":null,"location":"server.ts:539"},{"id":"resetPasswordBenderChallenge","detected":false,"via":null,"location":"data/static/securityQuestions.yml:21"},{"id":"resetPasswordBjoernChallenge","detected":false,"via":null,"location":"data/static/securityQuestions.yml:19"},{"id":"resetPasswordBjoernOwaspChallenge","detected":false,"via":null,"location":"data/static/securityQuestions.yml:15"},{"id":"resetPasswordJimChallenge","detected":false,"via":null,"location":"data/static/securityQuestions.yml:3"},{"id":"resetPasswordMortyChallenge","detected":true,"via":"rate-limit-bypass","location":"server.ts:374"},{"id":"resetPasswordUvoginChallenge","detected":false,"via":null,"location":"data/static/securityQuestions.yml:25"},{"id":"restfulXssChallenge","detected":true,"via":"xss","location":"frontend/src/app/search-result/search-result.component.ts:113"},{"id":"scoreBoardChallenge","detected":false,"via":null,"location":"frontend/src/app/app.routing.ts:186"},{"id":"tokenSaleChallenge","detected":false,"via":null,"location":"frontend/src/app/app.routing.ts:261"},{"id":"unionSqlInjectionChallenge","detected":true,"via":"sql-injection","location":"routes/search.ts:23"},{"id":"vulnerableDockerImageChallenge","detected":false,"via":null,"location":"infrastructure/docker-compose.yml:59"},{"id":"weakPasswordChallenge","detected":false,"via":null,"location":"models/user.ts:73"},{"id":"web3SandboxChallenge","detected":false,"via":null,"location":"frontend/src/app/app.routing.ts:239"},{"id":"web3WalletChallenge","detected":false,"via":null,"location":"data/static/web3-snippets/ETHWalletBank.sol:34"},{"id":"xssBonusChallenge","detected":true,"via":"xss","location":"frontend/src/app/search-result/search-result.component.ts:146"}]},"aiAttribution":{"samples":10,"threshold":0.5,"precision":1,"recall":0.6,"f1":0.7499999999999999,"accuracy":0.8,"rocAuc":1,"toolSamples":10,"toolAccuracy":1},"falsePositives":[{"corpus":"OWASP Benchmark (Java)","caseId":"BenchmarkTest02274","rule":"sql-injection","title":"A switch on a constant character","explanation":"The helper switches on \"ABC\".charAt(1), which is always 'B', and that arm assigns a constant; the arms that assign the request value never run. The engine does not evaluate the switch subject, so it keeps every arm and the line-level concatenation match stands. Engine gap; most of the remaining BenchmarkJava SQL false positives are constant-folding tricks of this kind (constant switches and if-conditions, fixed list and map positions)."},{"corpus":"SAST Benchmark — Rust","caseId":"BenchmarkTest00158","rule":"log-injection","title":"A sanitizer written as a character filter","explanation":"The username is cleaned with chars().filter(|c| !c.is_ascii_control()), which removes CR and LF, but the engine does not recognise a closure-based filter as a log sanitizer, so the log line is still reported. Engine gap."},{"corpus":"SAST Benchmark — Go","caseId":"BenchmarkTest01167","rule":"idor","title":"Ownership proven by an HMAC-signed token","explanation":"The handler verifies an HMAC-signed resource token that binds the caller to the resource id before the lookup. The engine recognises comparisons with the authenticated principal but not a signature check, so it reports the lookup. Engine gap."},{"corpus":"SAST Benchmark — Ruby","caseId":"BenchmarkTest00997","rule":"ssrf","title":"An https-only check is labelled safe","explanation":"The corpus treats a request URL as safe once its scheme is https. Restricting the scheme does not stop a request to an internal host (https://169.254.169.254/, an internal service), so we keep reporting it. A deliberate disagreement with the label."},{"corpus":"SAST Benchmark — Go","caseId":"BenchmarkTest01128","rule":"weak-signing-secret","title":"A signing-method check does not make a literal key safe","explanation":"The test checks that the token uses an HMAC method and then verifies it with the key []byte(\"supersecret\"). The corpus labels it safe because the algorithm is pinned; we still report it, because a key written in the source lets anyone mint a token the parser accepts."},{"corpus":"SAST Benchmark — PHP","caseId":"BenchmarkTest00985","rule":"insecure-deserialization","title":"An instanceof check after unserialize is too late","explanation":"A request value is passed to igbinary_unserialize and the result is then checked with instanceof. The corpus labels it safe; we report it, because the object is already built (and its destructors and wakeup methods run) before the check, so a gadget chain does not need the type to match."}],"falseNegatives":[{"corpus":"OWASP Benchmark (Java)","caseId":"BenchmarkTest00003","rule":"weak-crypto","title":"A hash algorithm chosen in a properties file","explanation":"The algorithm name is read with getProperty(\"hashAlg1\", ...) from benchmark.properties, where it is set to a weak hash. The engine does not resolve values from .properties files, so every hash case configured this way is missed. Engine gap; on the roadmap."},{"corpus":"SAST Benchmark — Rust","caseId":"BenchmarkTest00125","rule":"ssrf","title":"A stub HTTP client","explanation":"The request URL reaches simulated_reqwest_get(), a function the corpus defines to stand in for an HTTP client and which only formats a string. We report calls to real HTTP libraries, not functions that merely have a client-like name, so these cases count as misses. A deliberate choice."},{"corpus":"SAST Benchmark — Ruby","caseId":"BenchmarkTest01120","rule":"xxe","title":"Nokogiri's default parse is labelled vulnerable","explanation":"Nokogiri::XML(input) does not substitute external entities by default; the corpus labels it vulnerable and labels the noent option — which turns substitution on — as safe. We follow the library's real behaviour, so Ruby XXE scores 0 on this corpus. A deliberate disagreement with the label."},{"corpus":"SAST Benchmark — PHP","caseId":"BenchmarkTest00053","rule":"csrf","title":"CSRF without a session or cookie","explanation":"The corpus labels a state-changing handler with no CSRF token as vulnerable even when it reads no session or cookie. Cross-site request forgery needs a credential the browser attaches by itself, so we report CSRF only for session- or cookie-authenticated handlers. A deliberate choice."},{"corpus":"OWASP Juice Shop","caseId":"weakPasswordChallenge","rule":"weak-crypto","title":"A weak password hash behind a helper in another file","explanation":"models/user.ts stores security.hash(password), and the weak MD5 hash lives in lib/insecurity.ts. The weak hash itself is reported there, but not the use of it for passwords in the model. Engine gap."},{"corpus":"SAST Benchmark — Go","caseId":"BenchmarkTest00250","rule":"insecure-deserialization","title":"yaml.Unmarshal into interface{} is labelled vulnerable","explanation":"The corpus files decoding a request body with gopkg.in/yaml.v3 into an interface{} under deserialization. Go's YAML decoder builds plain maps, slices and scalars and runs no code, so there is no gadget to reach; we do not report it, and score it as a miss rather than change the label."},{"corpus":"SAST Benchmark — Rust","caseId":"BenchmarkTest00060","rule":"insecure-deserialization","title":"A same-file function named like a deserializer","explanation":"The test calls deserialize_object(body.as_bytes()), a function defined a few lines below that returns a constant. Nothing in the file reads or builds an object, and we do not treat a locally defined function as a library sink because of its name, so the case counts as a miss."}],"limitations":["Benchmarks are small, synthetic programs. They measure whether known patterns are recognised, not how the engines behave on your codebase, and scores on different corpora are not comparable with each other.","Trace accuracy is not measured: no public corpus labels the expected source-to-sink path.","Kotlin has no public labelled corpus and is not in this report; it is covered by unit tests and checked against the AndroGoat and Goatlin apps by hand.","Juliet cases are scored by region (a finding in the bad or good function); Juice Shop challenges count as detected when a finding lands within two lines of a marked line.","The internal cross-file suite was written by us. It is included because no public corpus targets cross-file flows, but it is less independent than the public corpora.","The AI-attribution check uses 10 labelled TypeScript/JavaScript files: a regression check, not a statistically strong evaluation.","Throughput is measured in batch on a developer workstation, on the corpora scanned in a single pass. It is not end-to-end pull-request latency.","Where we disagree with a corpus label we keep the label in the score and explain the disagreement in the examples above, rather than changing the label.","Many Rust cases, and some PHP and Ruby ones, call a function defined in the same file that only imitates a library call (simulated_reqwest_get, aes_ecb_encrypt, native_tls_danger). We scan the code as written and do not treat a locally defined function as a library sink because of its name, so those cases count as misses; this is the main reason Rust's recall is low, and it says little about code that calls the real libraries.","The sast-benchmark and Juliet C/C++ corpora were re-run on 2026-11-16. The OWASP Benchmark (Java, Python), Juliet C# and Juice Shop corpora were not available for that run, so each of them keeps the engine commit named against it rather than being presented as measured on today's engine.","Juliet C/C++: cases that call virtual methods through a base class defined in another file, and the paired cases whose bad and good paths depend on two independent calls of the same random-looking condition, are not detected; we do not tune for the second kind because a path-insensitive analysis cannot tell it from its safe twin."],"comparison":{"status":"not-run","note":"We have not yet run CodeQL or Semgrep on these corpora under identical conditions, so we publish no comparison numbers. We will add one when we have run both tools on the same corpus versions, with their default security rules, scored by the same scripts."},"history":[{"commit":"b73e12f","date":"2026-10-24","corpus":"OWASP Benchmark (Python)","score":95,"note":"From 12 before this commit (recorded in the commit message)"},{"commit":"bbe15dd","date":"2026-10-29","corpus":"OWASP Benchmark (Java)","score":65.4,"note":"Test cases only; helper classes not scanned"},{"commit":"bbe15dd","date":"2026-10-29","corpus":"SAST Benchmark — Go","score":35,"note":"Category average"},{"commit":"bbe15dd","date":"2026-10-29","corpus":"SAST Benchmark — PHP","score":42,"note":"Category average"},{"commit":"bbe15dd","date":"2026-10-29","corpus":"SAST Benchmark — Ruby","score":36,"note":"Category average"},{"commit":"bbe15dd","date":"2026-10-29","corpus":"SAST Benchmark — Rust","score":25,"note":"Category average"},{"commit":"756c80d","date":"2026-11-01","corpus":"OWASP Benchmark (Java)","score":87.5,"note":"Category average; helper classes scanned alongside"},{"commit":"756c80d","date":"2026-11-01","corpus":"SAST Benchmark — Go","score":55,"note":"Category average"},{"commit":"756c80d","date":"2026-11-01","corpus":"SAST Benchmark — PHP","score":56,"note":"Category average"},{"commit":"756c80d","date":"2026-11-01","corpus":"SAST Benchmark — Ruby","score":52,"note":"Category average"},{"commit":"756c80d","date":"2026-11-01","corpus":"SAST Benchmark — Rust","score":39,"note":"Category average"},{"commit":"0e73164","date":"2026-11-03","corpus":"OWASP Benchmark (Java)","score":93.5,"note":"Category average; constant-returning methods of other classes and cross-file pass-through methods in local helpers"},{"commit":"b4505ab+","date":"2026-11-02","corpus":"OWASP Benchmark (Python)","score":98.8,"note":"Category average; measured on b4505ab plus the uncommitted Python changes of that day"},{"commit":"7b67b01+","date":"2026-11-16","corpus":"SAST Benchmark — Go","score":58.1,"note":"Category average; Go template aliases and JWT key callbacks, PHP/Ruby LDAP DNs and PHP database passwords, Ruby file inclusion and overlapping-alternation ReDoS"},{"commit":"7b67b01+","date":"2026-11-16","corpus":"SAST Benchmark — PHP","score":60.5,"note":"Category average; Go template aliases and JWT key callbacks, PHP/Ruby LDAP DNs and PHP database passwords, Ruby file inclusion and overlapping-alternation ReDoS"},{"commit":"7b67b01+","date":"2026-11-16","corpus":"SAST Benchmark — Ruby","score":56.6,"note":"Category average; Go template aliases and JWT key callbacks, PHP/Ruby LDAP DNs and PHP database passwords, Ruby file inclusion and overlapping-alternation ReDoS"},{"commit":"7b67b01+","date":"2026-11-16","corpus":"SAST Benchmark — Rust","score":39.7,"note":"Category average; unchanged -- the missed cases call same-file stand-ins for library functions (see limitations)"},{"commit":"528472b+","date":"2026-11-16","corpus":"NIST Juliet (C/C++)","score":84,"note":"Category average (TPR minus FPR); pointer state through helpers, references, members, globals and other files (double free, mismatched deallocation, NULL dereference, use after free, buffer capacity), cross-file taint through returns, out-parameters and globals, and constant-expression bounds"}]}