Skip to content
Research reports

Notealy / Aegis research · Report 001

Research preview

Baseline evaluation of Aegis-1

We measured the Aegis-1 development build and its pinned reference baseline on 24 held-out security reasoning cases, across detection, classification, and end-to-end analysis. Both models find most vulnerable code but also flag most secure code, and neither assigns a correct CWE category. These results set the training targets for the first fine-tuned version.

Published
Models
Aegis-1 · Development build, Baseline · Reference model

01Key findings

  1. 01

    Detection is sensitive but not selective. Aegis-1 found 11 of 12 vulnerable samples, but also flagged 8 of 12 secure counterparts as vulnerable (baseline: 10 of 12 and 10 of 12).

  2. 02

    CWE classification is absent. Even when the vulnerability was already described, both models assigned a correct CWE category in 0 of 12 cases, often naming the weakness correctly in prose and then emitting an unrelated identifier.

  3. 03

    Asking for the full analysis at once makes Aegis-1 more conservative: false positives fall to 1 of 12, but it now misses 8 of 12 vulnerable samples.

  4. 04

    Differences between Aegis-1 and the baseline fall within the confidence intervals. The two share the same weights, so any gap reflects instructions, not training.

02Setup

Models
Aegis-1 development build and the pinned reference baseline: the same open-weight 7B base model, differing only in system instructions.
Test set
security-reasoning-test-v1: 24 held-out synthetic code samples, 12 vulnerable and 12 secure counterparts.
Classification reference
MITRE CWE catalog, version 5.0.
Decoding
Identical for both models: temperature 0.2, fixed seed, schema-constrained output, reasoning enabled.
Scope
The model alone: single turn, no tools, no agent loop.
Scoring
Deterministic and automatic. Proportions are shown as k/n with 95% Wilson confidence intervals.

03Results

Detection

Is the sample vulnerable? Every vulnerable sample has a secure counterpart, so flagging everything scores poorly on precision and false positives.

Detection results (Mode A, detection only)

  • Accuracy

    Aegis-1
    0.62 (15/24; 0.43–0.79)
    Baseline
    0.50 (12/24; 0.31–0.69)

  • Precision

    Aegis-1
    0.58 (11/19; 0.36–0.77)
    Baseline
    0.50 (10/20; 0.30–0.70)

  • Recall

    Aegis-1
    0.92 (11/12; 0.65–0.99)
    Baseline
    0.83 (10/12; 0.55–0.95)

  • False positive rate

    Aegis-1
    0.67 (8/12; 0.39–0.86)
    Baseline
    0.83 (10/12; 0.55–0.95)

  • F1 score

    Aegis-1
    0.710
    Baseline
    0.625

  • Correct location, when detected

    Aegis-1
    0.82 (9/11; 0.52–0.95)
    Baseline
    0.80 (8/10; 0.49–0.94)

Classification

The vulnerability is described to the model, so detection is not required. It only has to name the correct MITRE CWE category.

Classification results (Mode B, verified description given)

  • Correct CWE (exact, accepted alternative, or more specific child)

    Aegis-1
    0.00 (0/12; 0.00–0.24)
    Baseline
    0.00 (0/12; 0.00–0.24)

  • Correct family, including a generic parent

    Aegis-1
    0.00 (0/12; 0.00–0.24)
    Baseline
    0.00 (0/12; 0.00–0.24)

  • Identifiers that do not exist in the catalog

    Aegis-1
    0.00 (0/12; 0.00–0.24)
    Baseline
    0.00 (0/12; 0.00–0.24)

  • Abstentions

    Aegis-1
    0.00 (0/12; 0.00–0.24)
    Baseline
    0.08 (1/12; 0.01–0.35)
Every identifier returned was a real CWE entry; all of them were unrelated to the weakness present.

End-to-end analysis

The model produces the full analysis at once: verdict, CWE category, location, and remediation. A case counts as correct only if the whole assessment is right.

End-to-end results (Mode C)

  • Fully correct, all cases

    Aegis-1
    0.33 (8/24; 0.18–0.53)
    Baseline
    0.25 (6/24; 0.12–0.45)

  • Recall

    Aegis-1
    0.33 (4/12; 0.14–0.61)
    Baseline
    0.58 (7/12; 0.32–0.81)

  • False positive rate

    Aegis-1
    0.08 (1/12; 0.01–0.35)
    Baseline
    0.50 (6/12; 0.25–0.75)

  • Precision

    Aegis-1
    0.80 (4/5; 0.38–0.96)
    Baseline
    0.54 (7/13; 0.29–0.77)

  • Correct CWE, when detected

    Aegis-1
    0.00 (0/4)
    Baseline
    0.00 (0/7)

  • Remediation rubric, mean (detected cases)

    Aegis-1
    0.125
    Baseline
    0.357
All fully correct cases are secure samples correctly left unflagged; no vulnerable sample was both detected and correctly classified.

Output reliability

Whether the model returned a usable, well-formed answer at all, measured on the detection run.

Output reliability (detection run)

  • Strictly schema-compliant output

    Aegis-1
    0.96 (23/24)
    Baseline
    1.00 (24/24)

  • Answer cut off at the token limit

    Aegis-1
    0.04 (1/24)
    Baseline
    0.00 (0/24)

  • Reasoning stuck in a repetition loop

    Aegis-1
    0.04 (1/24)
    Baseline
    0.00 (0/24)

  • Median latency

    Aegis-1
    8.1 s
    Baseline
    7.7 s

04Error analysis

Each failed case is labeled with what went wrong. The labels show where training has to focus.

Error labels by run (a case can carry several)

  • False positive

    Meaning
    Secure code judged vulnerable
    Aegis-1, detection
    8
    Baseline, detection
    10
    Aegis-1, end-to-end
    1
    Baseline, end-to-end
    6

  • Not detected

    Meaning
    Vulnerable code judged secure
    Aegis-1, detection
    0
    Baseline, detection
    2
    Aegis-1, end-to-end
    8
    Baseline, end-to-end
    5

  • Evidence mislocated

    Meaning
    Detected, but the cited lines miss the root cause
    Aegis-1, detection
    2
    Baseline, detection
    2
    Aegis-1, end-to-end
    2
    Baseline, end-to-end
    2

  • Wrong CWE

    Meaning
    Detected, but the CWE is unrelated to the weakness
    Aegis-1, detection
    —
    Baseline, detection
    —
    Aegis-1, end-to-end
    3
    Baseline, end-to-end
    3

  • Prohibited mapping

    Meaning
    A grouping, view, or deprecated CWE entry
    Aegis-1, detection
    0
    Baseline, detection
    0
    Aegis-1, end-to-end
    0
    Baseline, end-to-end
    4

  • Unsupported claim

    Meaning
    Contradictory fields or evidence out of range
    Aegis-1, detection
    0
    Baseline, detection
    1
    Aegis-1, end-to-end
    3
    Baseline, end-to-end
    1

  • Output truncated

    Meaning
    Generation hit the token limit
    Aegis-1, detection
    1
    Baseline, detection
    0
    Aegis-1, end-to-end
    1
    Baseline, end-to-end
    0
In the classification run, the wrong-CWE label applied to 12 of 12 cases for Aegis-1 and 11 of 12 for the baseline.

05What this means

  • Training has to teach the CWE taxonomy directly, using code-level examples rather than descriptions alone.
  • Training data needs reviewed pairs of vulnerable and secure code, so the model learns to follow whether untrusted input actually reaches a sensitive operation instead of flagging dangerous-looking APIs.
  • Future reports will compare fine-tuned Aegis versions against this same baseline and test set, so progress can be measured rather than asserted.

06Limitations

  • 24 cases and a single seed: only large effects are detectable, and the confidence intervals are wide.
  • The cases are synthetic and were reviewed by the model that authored them; human review is pending.
  • Remediation is scored with a deterministic concept rubric. Root-cause explanation quality needs human review and is not reported as a number.
  • The test set is kept private to prevent it from leaking into future training data. Only aggregate results are published.
  • These measure a model in isolation. They are not measurements of Notealy Platform, and they are not production security guarantees.

This report describes research measurements of a model in development. It is not a claim of production readiness, and benchmark results are not security guarantees. See the Aegis progress page for methodology and milestones.