Vulnerability Intelligence
Malicious Models and Repository Trust: A Triage Guide
How to triage an untrusted model repository using provenance, scanning, isolation, and evidence.
A model repository is a software-and-data supply-chain object, not a bag of weights. A typical repository can contain checkpoints, configuration, tokenizer files, custom Python, scripts, notebooks, adapters, native components, dependency manifests, conversion utilities, and instructions in a model card. Any of those files can change what happens when an engineer, CI job, converter, or serving runtime consumes the repository. If the source or revision is untrusted, do not load it on a developer workstation, production inference node, privileged notebook, or shared GPU host.
This guide owns repository and artifact trust triage. Model File Formats and Unsafe Deserialization Risk explains why a loader can become a code-execution boundary; this page decides what evidence to collect before an artifact is approved. AI Supply Chain Security and Model Security provide broader provenance and artifact controls.
First response: preserve evidence and quarantine
Start with a quarantine state: the artifact may be inspected as evidence, but it is not approved for normal loading, installation, indexing, or serving. Freeze the original bytes where possible. Capture the repository URL or registry, exact revision, branch or tag, download time, claimed publisher, file manifest, model card, license, release notes, and any signatures or attestations. Calculate hashes for each relevant file and for the downloaded bundle. Record who obtained it and why.
Do not run repository setup commands during triage. A README that says “pip install,” “curl,” or “run this script” is an instruction from an untrusted source, not a security decision. Do not install dependencies, enable custom code, or point a notebook at the artifact merely to see whether it works. Preserve the original evidence and work on a copy in an isolated analysis environment.
MALICIOUS MODEL TRIAGE PIPELINE
Discovery → Quarantine → Hash / provenance → File inventory → Format/loader analysis → Custom-code/dependency review → Static scanning → Isolated behavioral/dynamic analysis → Evidence synthesis → Approve / Restrict / Reject
The pipeline is deliberately staged. A suspicious filename should not trigger an improvised verdict, and a clean scanner result should not skip provenance or isolation. At each stage record the evidence, analyst, tool version, decision, and unresolved questions.
Repository and artifact taxonomy
A useful triage begins by separating concerns that are often collapsed into “malicious weights.”
| Concern | What may be affected | Primary question |
|---|---|---|
| Unsafe serialized object | Pickle-like checkpoints or nested archives | Can the loader reconstruct objects or run imports while loading? |
| Malicious custom code | Python modules, plugins, custom operators, remote-code hooks | What code executes, under which identity, and with what permissions? |
| Dependency or install tampering | Requirements, setup scripts, build hooks, binary wheels | Does installation change the host or introduce an unreviewed dependency? |
| Backdoored or poisoned weights | Tensor values and learned behavior | Does a trigger or targeted input produce an unexpected security-relevant result? |
| Unexpected tokenizer/config behavior | Tokenizer files, config, pre/post-processing | Can parsing, paths, limits, or runtime options alter the trust boundary? |
| Repository impersonation or substitution | Account, fork, mirror, tag, mutable “latest” reference | Is this the intended publisher and exact revision? |
| Social-engineering content | README, model card, notebooks, issue comments | Is an operator being urged to weaken controls or disclose secrets? |
These categories can overlap. SafeTensors may reduce arbitrary-object deserialization risk, but it does not prove that weights are benign, that custom code is absent, or that the repository is authentic.
Provenance and publisher identity
Ask who published the repository and whether that identity is established. Distinguish an official organization from an individual account, fork, mirror, re-upload, or similarly named project. Review the account and commit history for unexplained ownership changes, sudden binary replacement, unusual release timing, or references that do not match the project’s normal channels. Pin the exact commit or immutable digest; a familiar name and a mutable branch are not provenance.
A cryptographic hash answers “are these bytes the same bytes?” It does not answer “are these bytes safe?” A malicious artifact can have a valid, reproducible hash. Signatures and Sigstore-style attestations can bind an artifact to a signer, workflow, or transparency log according to the signing setup. Verify the identity, repository, workflow, and policy claims, and keep the verification bundle. Do not turn a valid signature into a claim about security quality or absence of backdoors.
Registry scanning and static inspection
Use registry controls as evidence, not as a verdict. Hugging Face documents malware and pickle scanning, commit signatures, access controls, and third-party scanners; its pickle guidance explicitly warns that import scanning is best-effort and that users remain responsible for review. A scanner PASS can mean that known checks found nothing; it does not prove every file, parser, dependency, or behavior is safe. A FAIL or warning should preserve the finding and move the artifact to investigation or rejection until explained.
Static inspection should inventory every file, archive layer, serialization format, dependency manifest, declared entry point, script, notebook, native library, custom operator, and remote-code option. Inspect metadata without importing application modules. Tools such as Protect AI ModelScan can add model-file scanning evidence when their current documented coverage matches the format. Record scanner version, signatures, and limitations. Never claim that ModelScan or any scanner “proves a model safe.”
Custom code and dependencies
Treat trust_remote_code, custom Python, plugins, and custom operators as separate approval decisions from weight format. Disable implicit remote code for untrusted repositories. If code is required, review the pinned source, build it in CI from a reviewed revision, and run it in a least-privilege worker. Keep dependency installation inside a controlled build boundary; do not execute setup hooks or system-package commands from a model card.
Inspect tokenizers and configuration as code-adjacent inputs. A tokenizer can influence parsing and resource consumption; a configuration can select a loader, external path, plugin, or post-processing routine. Auxiliary files may be more security-relevant than a large weight file because they decide which code path is activated.
Loading risk, behavior, and supply-chain risk
Maintain three separate hypotheses:
- LOADING/EXECUTION RISK — the format, parser, or custom code can execute, access files, or exhaust resources while loading.
- BEHAVIORAL BACKDOOR / POISONING — a safely loaded model produces targeted, unsafe, or policy-bypassing behavior for selected inputs.
- REPOSITORY SUPPLY-CHAIN RISK — publisher identity, dependencies, build process, or delivery path has been substituted or tampered with.
Evidence for one hypothesis does not prove the others. A clean static scan does not establish behavioral safety; a suspicious behavior does not by itself identify a compromised repository. Track each claim with its own test and confidence.
Isolated behavioral and dynamic analysis
If behavioral validation or detonation is justified, use a disposable worker with no production API keys, cloud credentials, SSH keys, developer tokens, or shared secrets. Run as a low-privilege identity with bounded CPU, memory, storage, process count, and lifetime. Use default-deny or tightly proxied egress; do not permit access to metadata services, internal APIs, credential stores, or production registries. Instrument filesystem, process, network, and child-process activity where the environment supports it, and destroy the worker after analysis.
Compare outputs with a trusted baseline across known task slices, safety checks, tokenizer behavior, and expected configuration. Use synthetic accounts and non-sensitive data. Dynamic testing can reveal a path or behavior, but it cannot prove the absence of hidden triggers. Treat unexpected outbound access, file writes, child processes, or configuration changes as evidence requiring containment and review, not as an invitation to explore further on a production host.
MODEL REPOSITORY TRUST MATRIX
| Signal | What it proves | What it does NOT prove | Action |
|---|---|---|---|
| Publisher identity | A claimed account or organization controls the repository | That every contributor or release is benign | Verify ownership, history, and scope |
| Hash / digest | Exact bytes match a recorded artifact | Safety or benign intent | Pin and preserve it as evidence |
| Signature or attestation | Integrity and signer/workflow claims under the stated policy | Secure code, safe behavior, or absence of compromise | Verify identity and policy; retain bundle |
| Registry scanner | Specific checks completed with a result | Complete coverage or behavioral safety | Record tool/version; investigate warnings |
| Serialization format | Loader capability or reduced object-deserialization surface | Repository authenticity or safe weights | Select constrained loader and isolate |
| Custom code | Code paths and permissions required by the artifact | That reviewed code is free of defects | Disable by default; review and sandbox |
| Dependency manifest | Declared packages and versions | Actual transitive or runtime behavior | Reconcile lockfile, image, and build evidence |
| Behavioral tests | Observed results for tested slices | Absence of unknown triggers | Compare baseline; document coverage and limits |
Approval states and evidence
Use disciplined states: APPROVED; APPROVED WITH RESTRICTIONS; QUARANTINED; REJECTED; or UNKNOWN / NEEDS MORE EVIDENCE. Approval should name the exact digest, permitted loader, execution identity, network policy, intended workload, reviewer, and expiry or re-review trigger. Restrictions may include no custom code, read-only serving, a dedicated tenant, or offline inference.
ARTIFACT APPROVAL CHECKLIST
- Source, publisher, exact revision, digest, and acquisition time are recorded.
- File inventory includes weights, config, tokenizer, scripts, notebooks, adapters, native files, and dependencies.
- Format and loader options are documented; unsafe object loading and implicit remote code are disabled.
- Signatures, attestations, registry scans, and static-analysis results are verified with limitations noted.
- Dependencies and build steps are reviewed and reproducible; no untrusted install command ran on a privileged host.
- Behavioral tests use isolated, synthetic data and state their coverage and uncertainty.
- Approved execution has no production credentials, bounded resources, and controlled egress.
- Promotion, rollback, revocation, logging, owner, and re-review conditions are defined.
Incident handling
If evidence suggests tampering or malicious behavior, stop promotion and isolate every worker that consumed the artifact. Preserve hashes, logs, registry events, and deployment references. Identify affected revisions and downstream outputs, revoke credentials exposed to analysis, and compare deployed bytes with the approved digest. Notify the registry owner, vendor, or PSIRT through its security channel. Do not delete the only evidence before preserving it, and do not publish harmful details while coordination is in progress.