📦

Audit History

risk-control-engineer - 4 audits

Version comparison

Capability and finding changes across audited versions, newest first.

VersionDateResultReview itemsChange vs previous
v4 LatestJul 23, 2026, 11:57 PM 2 confirmed1No capability change
v3 Jul 15, 2026, 03:35 PM 2 confirmed1No capability change
v2 Jul 15, 2026, 03:35 PM 2 confirmed1No capability change
v1 Jul 15, 2026, 03:35 PM 2 confirmed1Baseline

Jul 23, 2026, 11:57 PM

The Python rule-engine example performs dynamic evaluation of rule text and is a confirmed code-execution risk. The 21 command alerts, two reconnaissance alerts, and entropy alert are false positives caused by Markdown fences, graph variables, and multilingual text. Two additional control-quality defects could hide rule failures or misstate backtest performance.

1
Files scanned
455
Lines analyzed
5
Review items
0
False positives ignored

Confirmed security concerns (2)

Medium
Rule Evaluation Fails Open
The rule engine catches every evaluation error and returns false. Invalid or malicious conditions can silently bypass controls instead of producing a review or rejection.
The broad exception handler directly converts every rule evaluation failure into a non-match. This behavior is explicit in the example.
Medium
Backtest Metrics Misstate Control Performance
The capture rate measures fraud prevalence instead of captured fraud, while the false-positive rate uses all labeled rows as its denominator. Decisions based on these metrics may weaken fraud controls.
The formulas visibly omit rule outcomes from capture rate and do not normalize false positives by legitimate cases. Standard confusion-matrix definitions confirm both errors.
Capability review items (1)

These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.

High
Dynamic code evaluation with eval()
return bool(eval(condition, {\"__builtins__\": {}}, allowed_vars))
The rule engine evaluates condition text through eval with feature values as locals. Removing builtins does not prevent malicious expressions from traversing Python objects, so untrusted rules can execute unintended code.
Audited by: codex

Jul 15, 2026, 03:35 PM

Most static alerts are false positives caused by Markdown fences, Chinese text entropy, and graph-community iteration. The Python rule engine still uses eval on condition strings and fails open when evaluation raises an exception. The sample backtest also miscalculates capture and false-positive rates, so it should not guide production decisions unchanged.

1
Files scanned
455
Lines analyzed
5
Review items
0
False positives ignored

Confirmed security concerns (2)

High
Rule Evaluation Fails Open
Any exception while checking a fraud rule returns false, so malformed conditions silently bypass that control instead of stopping or escalating the decision.
The exception handler directly returns false for every evaluation error. In this engine, false means the rule does not trigger.
Medium
Backtest Metrics Misstate Control Effectiveness
Capture rate is calculated as fraud prevalence, while false-positive rate uses all labeled rows as the denominator. Both can mislead deployment decisions.
The formulas do not condition capture rate on detected fraud or false-positive rate on legitimate cases. The code is explicit.
Capability review items (1)

These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.

High
Dynamic code evaluation with eval()
return bool(eval(condition, {"__builtins__": {}}, allowed_vars))
The rule engine evaluates condition strings at runtime. Removing built-ins reduces exposure, but externally supplied expressions can still access unsafe object attributes.
Audited by: codex

Jul 15, 2026, 03:35 PM

Most static alerts are false positives caused by Markdown fences, Chinese text entropy, and graph-community iteration. The Python rule engine still uses eval on condition strings and fails open when evaluation raises an exception. The sample backtest also miscalculates capture and false-positive rates, so it should not guide production decisions unchanged.

1
Files scanned
455
Lines analyzed
5
Review items
0
False positives ignored

Confirmed security concerns (2)

High
Rule Evaluation Fails Open
Any exception while checking a fraud rule returns false, so malformed conditions silently bypass that control instead of stopping or escalating the decision.
The exception handler directly returns false for every evaluation error. In this engine, false means the rule does not trigger.
Medium
Backtest Metrics Misstate Control Effectiveness
Capture rate is calculated as fraud prevalence, while false-positive rate uses all labeled rows as the denominator. Both can mislead deployment decisions.
The formulas do not condition capture rate on detected fraud or false-positive rate on legitimate cases. The code is explicit.
Capability review items (1)

These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.

High
Dynamic code evaluation with eval()
return bool(eval(condition, {"__builtins__": {}}, allowed_vars))
The rule engine evaluates condition strings at runtime. Removing built-ins reduces exposure, but externally supplied expressions can still access unsafe object attributes.
Audited by: codex

Jul 15, 2026, 03:35 PM

Most static alerts are false positives caused by Markdown fences, Chinese text entropy, and graph-community iteration. The Python rule engine still uses eval on condition strings and fails open when evaluation raises an exception. The sample backtest also miscalculates capture and false-positive rates, so it should not guide production decisions unchanged.

1
Files scanned
455
Lines analyzed
5
Review items
0
False positives ignored

Confirmed security concerns (2)

High
Rule Evaluation Fails Open
Any exception while checking a fraud rule returns false, so malformed conditions silently bypass that control instead of stopping or escalating the decision.
The exception handler directly returns false for every evaluation error. In this engine, false means the rule does not trigger.
Medium
Backtest Metrics Misstate Control Effectiveness
Capture rate is calculated as fraud prevalence, while false-positive rate uses all labeled rows as the denominator. Both can mislead deployment decisions.
The formulas do not condition capture rate on detected fraud or false-positive rate on legitimate cases. The code is explicit.
Capability review items (1)

These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.

High
Dynamic code evaluation with eval()
return bool(eval(condition, {"__builtins__": {}}, allowed_vars))
The rule engine evaluates condition strings at runtime. Removing built-ins reduces exposure, but externally supplied expressions can still access unsafe object attributes.
Audited by: codex