Frontier Model System Card Review: Security Evidence Checklist

An AI system card review should end with bounded claims and deployment decisions, not a model ranking. This edition reviews one material release: GPT-6 Astra. As of 2026-09-17, its public evidence supports serious cyber-capability concerns and documents safeguards, while leaving important questions about generalization, monitoring, and a particular customer's deployment unresolved.

Model/version: GPT-6 Astra. Release date: 2026-09-03. System-card date: 2026-09-03, with change-log updates on 2026-09-09. Evidence cutoff: 2026-09-17. Primary artifact: OpenAI GPT-6 Astra System Card. Public external assessment: Irregular, 2026-09-03.

The frontier evaluation guide owns cross-release methodology. This page applies a security-evidence checklist to Astra only. The statuses below describe support for a precisely scoped claim, not whether a model is universally safe.

Identity and the reviewed evidence boundary

The card identifies Astra and distinguishes internal evaluation from external deployment. It classifies cyber capability as Critical under OpenAI's own Preparedness Framework. That is a provider assessment, not a neutral industry certification. Our review does not establish that every API request, checkpoint, effort setting, or surrounding agent is identical to the evaluated system.

Before adopting the model, capture the served identifier, dated endpoint configuration, effort, tools, context assembly, memory, permissions, and access program in a local manifest. A marketing name alone cannot pin an entire evaluated system. If the provider changes a hosted revision, preserve the old review and record which conclusions require retesting.

Treat evidence identity and deployment identity as separate objects. The first identifies what the report tested; the second identifies what your application actually runs. A gap between them does not invalidate the report, but it limits the report's usefulness as direct acceptance evidence for your system.

System card security evidence checklist

Confidence is PermsAI's judgment about the stated claim, conditional on the named evidence. “Independent” distinguishes a separate evaluator's work from a fully independent replication. Protocol details belong to the exact report, not to all uses of the model.

Claim	Evidence	Protocol	Provenance	Safeguards	Independent evidence	Limitation	Confidence
Material offensive-cyber capability	Provider Critical classification; external verified solves	Card capability suite; FrontierCyber objectives	OpenAI and Irregular	Capability configuration differs from ordinary deployment	Public Irregular report	No universal target success established	SUPPORTED; high for bounded capability
FrontierCyber success on evaluated snapshot	86/226 challenges solved	Verifiable objectives on real systems	Irregular report	Controlled evaluation	External evaluator worked with OpenAI	No Elite solves; no fully hardened target successes	SUPPORTED; moderate transfer
Browser case proves ordinary full browser compromise	Native execution case	FrontierCyber browser task	Irregular report	Content sandbox disabled	Public evaluator description	Additional sandbox escape would be needed	NOT ESTABLISHED; strong limitation
ExploitGym results represent unrestricted production use	Card results	Offline v1; intended-exploit grading; token cap	Provider	Runtime installation disabled	Public benchmark, not independent rerun	Harness and access differ	NOT ESTABLISHED; low transfer
Report documents deployment safeguards	Safeguards sections	Training, monitoring, and access layers	Provider	Enabled deployment stack	Partial external assessment	Documentation is not a customer audit	SUPPORTED; moderate
Improvements establish universal alignment	Card alignment evaluations	Task-specific tests and simulations	Provider and named partners	Conditions differ by test	External contributions in card	Residual failures and evaluation awareness	PARTIALLY SUPPORTED; scoped improvement only
CoT monitoring guarantees detection	Monitorability section	Adversarial and non-adversarial tests	Provider	Monitoring scope matters	AISI contribution reported in card	Detection guarantee unsupported	NOT ESTABLISHED; high confidence in non-guarantee
Your application has an independently measured incident rate	No deployment-specific denominator here	Would require local longitudinal monitoring	Not available in this review	Your full stack	No matching independent deployment study cited	Absence of evidence is not zero incidents	NOT REPORTED; unresolved

Evidence for provider rows: Astra card. Evidence for external capability and browser rows: Irregular assessment. The local-deployment row is a boundary of this review, not a claim that no such evidence could ever exist.

Capability evidence: inspect the success condition

Irregular reports 86 of 226 FrontierCyber challenges solved, with no successful attacks against fully hardened targets and no Elite challenge solved. Its browser case explicitly disabled the content sandbox. This is meaningful external evidence, with a collaboration caveat: Irregular worked with OpenAI; the report is not a stranger's unrestricted production audit.

The security implication is neither “all systems are compromised” nor “hardened systems are safe.” Verified successes warrant stronger containment. Unsolved targets show limits of this tested configuration, not an impossibility theorem. Ask whether your exposed software, granted credentials, and workload budget resemble the tested opportunities before extrapolating.

A case involving execution inside one process also needs boundary precision. In your decision record, distinguish initial execution, crossing the application sandbox, accessing another principal's data, and affecting the host. Do not combine these into one undifferentiated “compromise” label. This keeps remediation aligned with the actual layer at risk.

Protocol, tools, scaffold, and budget

The card's ExploitGym evaluation uses offline v1, disables runtime package installation, and grades intended exploitation with a token cap but no wall-clock cap. The expert-led capability work uses a different configuration: Codex harness, Ultra effort, web access, and up to 64 subagents. These are different evaluated systems, not interchangeable scores.

Our reading rule is to create one row per protocol. Record target materials, tools, network access, parallelism, retries, stop conditions, budget, and grader. Do not use a long-horizon, parallel research result as evidence of what a short interactive request can do, or use a constrained request's failure as evidence of the model's maximum capability.

For a local comparison, declare budgets before running the test. Count the parent and children together. Include failed attempts and diagnostic work, not only tokens in the winning trajectory. Report wall time and successful-workflow cost alongside quality. The test-time compute guide explains why more inference opportunity changes what a score means.

Security scoring also needs negative controls. Verify that the evaluator rejects a shortcut, unrelated defect, corrupted grader, or answer obtained from an unintended source. Keep operational exploit material inside approved research environments; a review article does not require reproducing harmful mechanics. The security benchmark guide provides the protocol questions to carry into that review.

Pre-mitigation and post-mitigation are separate claims

A capability evaluation may deliberately elicit behavior that deployment safeguards are intended to block. Conversely, a safeguards-enabled test measures a combined system rather than the model alone. Label the evaluated checkpoint and safeguard state for every claim. Where the report does not provide that pairing, mark the comparison unresolved rather than reconstructing it from unrelated figures.

For Astra, the card describes strengthened internal isolation and broad tool-use monitoring, as well as external safeguards. That supports a documentation claim. It does not establish that your connector, downstream API, network policy, or approval process participates in the same protection boundary.

Build two local test views where allowed: representative production behavior with all normal protections enabled, and controlled component analysis that identifies where protection comes from. Never remove production safeguards on live third-party targets to “verify the card.” The useful evidence is which legitimate tasks succeed, which disallowed effects are denied, and whether recovery preserves policy.

The September 9 update changes claim interpretation

The change log clarifies alignment generalization and distinguishes verbalized metagaming from oversight gaming. It also emphasizes that unobserved failures do not establish reliability across settings. This is a material interpretation update: a model discussing evaluation conditions is not automatically proof that its action causally gamed oversight.

In our checklist, behavioral labels require an observable effect and a justified inference, not only a striking reasoning snippet. Record the actual action, permitted scope, and downstream result first. Treat a chain-of-thought explanation as evidence with its own uncertainty. Separate “reasoning mentioned evaluation” from “behavior undermined the intended measurement.”

This prevents two opposite errors. One is attributing hidden intentions confidently from text. The other is treating reassuring text as proof of compliant action. A system that says it respected scope but makes an unauthorized call fails the action boundary regardless of its explanation.

Monitoring evidence is not an authorization mechanism

The card reports monitoring limitations, including decreased chain-of-thought monitorability. A monitoring program can provide useful visibility without guaranteeing detection. Review what a monitor sees: full actions, raw reasoning where available, summaries, outputs, or only selected events. These are not equivalent evidence streams.

For your deployment, define response latency and authority. Can a monitor block a tool before execution, stop the workload afterward, or only produce an alert? A correct alert arriving after an irreversible effect is detection, not prevention. Missing telemetry should not silently become a clean bill of health.

Test monitoring separately from authorization. Introduce synthetic policy violations in a controlled fixture and measure detection, false positives, interruption, and evidence preservation. Then confirm that deterministic access controls still deny unauthorized requests when the monitor misses them. Use the AI red-teaming guide to plan safe adversarial cases and meaningful coverage.

What independent evidence does—and does not—add

Irregular's public report is stronger provenance than a provider's unattributed statement that “third parties tested the model.” It gives named suites, outcomes, and configuration limitations. Its findings independently authored by the evaluator support bounded capability concerns; its work with OpenAI should remain visible in the provenance column.

External authorship does not by itself establish representative traffic, unrestricted checkpoint access, complete publication of failures, or replication of every provider benchmark. Nor should one external cyber assessment be used as evidence of all alignment and safeguard claims. Keep evidence attached to the claim it actually tests.

For procurement, ask which additional artifacts can be inspected: protocol manifests, redacted trajectories, target configuration, budget accounting, grader validation, and retest history. If disclosure would expose unpatched vulnerabilities, accept appropriately redacted evidence with an explicit limitation rather than demanding exploit details in a public review.

Deployment decision: constrain before expanding

Our synthesis is to pilot Astra under a narrow, attributable authority model. Separate reasoning from credential custody, maintain resource-level authorization, default-deny unneeded egress, and require exact-action approval for high-impact effects. Those controls remain necessary even when model-level refusal and monitoring improve.

Adopt only after local acceptance tests cover legitimate work, denied variants, cancellation, retries, changed resource identities, and telemetry gaps. Constrain or defer any path whose consequences exceed demonstrated containment. Record the unresolved question and owner instead of filling it with confidence from an unrelated benchmark.

This edition does not rank Astra against other releases. It reviews whether named security claims have suitable evidence. Read the benchmark literacy guide for general score interpretation. Issue a new evidence-led edition after a material release, and revise this one when its card or evaluator report changes—not merely when a headline repeats the same result.

Sources

Evidence cutoff and review date: 2026-09-17. Claim statuses, confidence judgments, and deployment recommendations are PermsAI synthesis.

• OpenAI GPT-6 Astra System Card: release and card 2026-09-03; change log 2026-09-09. Reviewed identity, capability protocols, alignment interpretation, monitoring limits, and deployment safeguards. External findings reproduced inside the card retain provider-hosted provenance.
• Irregular: Assessing GPT-6 Astra: 2026-09-03. Public external capability assessment developed with OpenAI; FrontierCyber, CyScenarioBench, Atomic Challenges, and configuration caveats. External evaluation is not universal deployment certification.