AI Cyber Capability Evaluations: Reading New Evidence Safely

Cyber evaluations are evidence about a configured system

Evidence cutoff: 2026-09-17. A frontier-model cyber result belongs to model + scaffold + access + budget, not to a model name alone. It can provide material evidence of what a system achieved on a declared task population. It does not directly establish real-world compromise probability, attacker intent, target access, victim exposure, persistence, operational stealth, deployment availability, or safeguard bypass.

This article complements Frontier Model Evaluations, which supplies the broad reading framework, and AI Security Benchmarks, which explains metric interpretation. The purpose here is a dated cyber-evidence bridge: what each current source supports and what remains between a benchmark and practical risk.

A small representative evidence set

OpenAI’s GPT-6 Astra System Card was published September 3, 2026 and updated September 9. It reports Astra as Critical under OpenAI’s Preparedness Framework, describes internal cyber evaluations under particular tool access, and reports an external Irregular evaluation: 86 of 226 FrontierCyber challenges solved in sandboxed settings without public internet. It also describes longer-horizon internal evaluations with a standard Codex harness at Ultra reasoning effort, web access, and up to 64 subagents. These are consequential capability claims under high access and budget; they are not a public estimate of what every deployed user can do.

METR’s Frontier Risk Report (February to March 2026), published May 19, aggregates evaluation evidence and discusses method, model access, and limits. Its framing is valuable because it separates public and internal frontiers and notes that instructions, scaffolding, and inference budget can shift outcomes. It is a report, not a probability model for specific victim organizations.

The UK AI Security Institute’s Frontier AI Trends Report (official report, 2026) reports cyber evaluations across difficulty levels and explicitly investigates enhanced scaffolding, including improved system prompts and interactive tool access. Its trend measurements help compare declared evaluation regimes, but a trend line does not identify an attacker, target, or deployed control environment. The Frontier Model Forum’s Frontier Capability Assessments technical report gives a cross-provider framework for capability assessment methods; it does not certify a model as safe or unsafe.

CYBER EVALUATION EVIDENCE STACK

Layer	Question to ask	Why it changes interpretation
Model/version	Which exact release and date?	Provider updates can change behavior and safeguards.
Task	What task population and success criterion?	A CTF, software issue, and hardened-target simulation are not interchangeable.
Scaffold/access	Model-only, browser, shell, code execution, internet, subagents, human assistance?	Tools can be decisive capability multipliers.
Budget	Attempts, retries, tokens, wall-clock, test-time compute?	A best-of-many result differs from a single attempt.
Scorer	Automated grader, expert review, or operational consequence?	Score validity and error modes determine what success means.
Reliability	How many runs, confidence intervals, variance, partial completions?	One trajectory cannot establish repeatability.
Safeguards	Pre- or post-mitigation; research or production access?	Safeguards alter availability, not the underlying measured skill.
Deployment relevance	Which access and controls match the intended deployment?	Transfer requires a documented bridge, not a label.

An absence of benchmark success does not prove zero capability. The task may be too narrow, the budget too small, tools unavailable, the scorer imperfect, or the model aware of evaluation. Conversely, a successful benchmark trajectory does not prove a model will autonomously find a suitable real target, hold credentials, evade detection, or maintain a campaign. Those are additional hypotheses.

CAPABILITY-TO-RISK GAP MATRIX

Evidence	What it supports	Missing bridge	Operational implication	Confidence
Astra system-card cyber results	Named model completed declared cyber tasks with stated harness/access	External target access, public availability, reliability in other scaffolds	Treat high-access agent deployments as requiring strong safeguards	Medium: provider report, detailed but not independent unrestricted audit.
Irregular FrontierCyber result	Performance on 226 sandboxed challenges, reported 86 solves	Internet, production hardening, operational campaign behavior	Use as task-family evidence, not an intrusion probability	Medium: external evaluator, but selected suite and sandbox bound scope.
METR Frontier Risk Report	Comparative/aggregated evidence and methodological cautions	A given provider’s deployment and attacker pathway	Require access and budget disclosure in risk review	Medium: independent report, heterogeneous source evidence.
AISI trends/enhanced scaffold work	Results change with task difficulty and tool scaffolding	Specific production policy and target environment	Test local tool access, not just model identity	Medium: official evaluator, trend and environment limits.

The phrase “capable of cyber tasks” should be followed by the task, harness, budget, scorer, and uncertainty. This is not pedantry. A browser-enabled agent with code execution, web access, and many subagents is a materially different evaluated system from a chat endpoint with no tools. Agentic Benchmarks and Test-Time Compute and Reasoning Benchmarks explain why scaffold and budget must be visible measurement variables.

CYBER EVALUATION RESULT CARD

Field	Record for every claim
Model/version	Exact identifier, release and evaluation dates.
Source/date	Organization, first-party or external, report status.
Task set	Population, realism, denominator, success predicate.
Tools/access	Browser, shell, code execution, internet, data, human help, subagents.
Budget	Attempts, retries, tokens or time, stopping rules.
Result/baseline	Rate or qualitative result with comparator and uncertainty.
Repeatability	Runs, variance, failed/partial trajectories, evaluator caveats.
Safeguards	Model behavior, monitoring, account, infrastructure, pre/post state.
Limitations	Missing access, simulation fidelity, scorer limits, deployment mismatch.
Claim scope	The narrow conclusion actually supported.

Build the card before choosing a risk decision. If a field is absent, write NOT REPORTED rather than silently filling it with a favorable assumption. Ask whether the evaluation used research access unavailable in production, whether mitigations were enabled during measurement, and whether external evaluators could select or inspect the task set. A provider statement may be accurate and still not answer every deployment question.

The real-world risk bridge

Benchmark capability → repeatability/reliability → required access → tool availability → attacker cost → safeguards → target exposure → detection/response → plausible operational risk.

This is a reasoning sequence, not a numerical model. Each arrow can weaken or strengthen the next. Repeatable task completion may still require source code, network reachability, a permissive tool policy, long budget, and a poorly monitored target. Conversely, modest benchmark performance can be operationally important where a tool is cheap, targets are exposed, and detection is slow. The appropriate response is to document the arrows for a specific deployment rather than extrapolate a global probability.

Separate underlying capability from deployed safeguards. OpenAI’s Astra card describes strengthened isolation, monitoring, and model-behavior safeguards in addition to capability evidence. That supports the existence of reported layers, not the conclusion that capability disappeared or that every deployment has equivalent enforcement. Pre-mitigation and post-mitigation measurements should never be averaged into one label. Anthropic Cyber Evaluation Incidents illustrates why evaluation environment boundaries and response design are themselves risk variables.

Practical reading workflow

First, identify the evaluated system and release. Second, normalize task success: what was completed, by whom, under which scorer? Third, record access and budget. Fourth, find the baseline and variance. Fifth, identify safeguards and whether they were measured or merely described. Sixth, map the local deployment: permissions, tools, user eligibility, egress, approval, logging, and response. Finally, choose a bounded action—adopt with constraints, pilot, defer, or request more evidence.

Run local acceptance tests that mirror legitimate tasks and denied variants. Use synthetic or authorized targets only. Test cancellation, retries, tool policy changes, and telemetry gaps. Preserve traces and escalation ownership. This is not a demand to reproduce advanced cyber evaluations; it is an attempt to ensure that a deployment’s authority model matches its risk claim.

As of 2026-09-17, the selected sources support a cautious conclusion: frontier cyber evaluations can show meaningful, sometimes tool-sensitive capability under named protocols. They do not directly establish real-world compromise probability or a universal safety conclusion. The most defensible deployment decision carries the full result card and names the missing bridges.

Evaluation quality and independent challenge

Evaluation quality is itself evidence. Prefer a stated task selection process, usable denominator, named grader, and account of failed or incomplete attempts. Ask whether the tester had privileged model access, whether the model received a specialized system prompt, and whether a benchmark was withheld from training. An external evaluator improves provenance, but external authorship alone does not guarantee unrestricted access to every model configuration, dataset, or internal safeguard.

For a local governance record, attach the result card to an owner, a review date, and a retest trigger. Material changes include a new model revision, expanded web access, added credentials, a larger retry budget, new subagents, or a different target class. If those changes occur, prior capability evidence should be reread as historical rather than silently treated as current deployment evidence.

From cyber benchmark to operational risk

A task success is evidence at the first link of a longer chain, not direct evidence for the final outcome. Ask whether the result repeats across runs, then whether it generalizes across targets rather than one curated environment. Identify the required scaffold: browser, shell, code execution, network access, subagents, specialized tools, and the time or compute budget. Next identify access prerequisites—source code, credentials, reachable services, or a permissive account—and estimate whether they are available to the actor being considered.

The remaining links can weaken the inference further. Safeguards may block the same capability in deployment; a target may be hardened or not exposed; activity may be detected before a consequential step; and a useful one-off task completion may not support stealth, persistence, or reliable adaptation. Time and cost can also make a technically possible result operationally irrelevant for some actors. Conversely, lower benchmark performance can matter where access is easy and defender response is slow. Record each link as supported, partially supported, unknown, or contradicted instead of converting a score into a probability.

Evaluation configuration comparison

Dimension	Evaluation A: sandbox benchmark	Evaluation B: high-access agent evaluation	Why difference matters
Model revision	Named released or shared version	Named revision with provider configuration	Revision changes behavior and safeguards.
Scaffold	Model or narrow agent loop	Tool-using agent with orchestration	System result is not model-only result.
Browser/network	Usually isolated or absent	May include web/network access	Reachability changes task opportunity.
Shell/code execution	Restricted simulated tools	Declared execution/research tools	Execution enables different task classes.
Subagents	None or fixed	Possible delegated workers	Parallelism changes coverage and budget.
Attempt budget	Fixed trials	Retries, tokens, or long horizon	Best-of-many differs from one attempt.
Human assistance	Limited grading/support	Experts may set environment	Assistance affects attribution.
Target hardening	Benchmark-specific	Simulated or named hardened setup	Difficulty labels are not interchangeable.
Success criterion	Automated task predicate	Expert and technical completion review	Percentages may count different outcomes.
Safeguards	Test configuration	Pre/post mitigation and deployment layers	Safeguards affect availability, not necessarily skill.

PermsAI evidence taxonomy

This is a PermsAI evidence taxonomy, not an AISI, METR, or lab standard. EVIDENCE OF NARROW CAPABILITY means a declared task succeeded under one protocol. EVIDENCE OF REPEATABLE CAPABILITY requires repeated success and variance reporting. EVIDENCE OF TRANSFER requires performance across materially different targets, tasks, or configurations. EVIDENCE OF OPERATIONALLY RELEVANT CAPABILITY additionally needs a supported bridge through access, tools, safeguards, exposure, and detection conditions. INSUFFICIENT EVIDENCE is the correct state when those fields are absent or incompatible. These states classify claims; they do not rank models.

Sources

• OpenAI, GPT-6 Astra System Card, published September 3, 2026; updated September 9, official system card. Source. Model, protocol, safeguards and limitation claims; provider report scope applies.
• METR, Frontier Risk Report (February to March 2026), May 19, 2026, independent technical report. Source. Comparative methodology and limits; not a victim-risk probability model.
• UK AI Security Institute, Frontier AI Trends Report, 2026, official report. Source. Cyber task trends and scaffold findings; task/access limits apply.
• Frontier Model Forum, Frontier Capability Assessments, technical report, 2025. Source. Assessment-method context; not a certification.
• Irregular, Assessing GPT-6 Astra, September 3, 2026, external technical report. Source. Sandbox evaluation evidence; selected suite and collaboration scope limit transfer.