At a glance
- Who should read this
- ML engineers, platform teams and security teams that download, fine-tune or deploy third-party models and datasets.
- The risk
- A model file is software. Some serialization formats can execute code when loaded, repositories can be impersonated or tampered with, and a model's behavior can be deliberately altered in ways standard evaluation will not reveal.
- Priority actions
- Prefer weight formats that cannot execute code, such as safetensors, and never load untrusted pickle-based files outside a sandbox.
- Pin models to an exact revision and verify hashes or signatures before deployment.
- Record every model in an inventory with its source, version and approval.
Summary
Organizations increasingly build on open-weight models downloaded from public hubs. That brings the familiar risks of the software supply chain, plus some that are specific to models. A model can carry executable code in its file format, be published under a misleading name, change silently between downloads, or contain behavior planted during training that only appears under specific conditions.
The same discipline that protects software dependencies applies: know where each artifact came from, verify that it has not changed, limit what it can do when loaded, and keep an inventory.
Risks
| Risk | Description |
|---|---|
| Code execution on load | Formats based on Python's pickle, used by many PyTorch checkpoints, can run arbitrary code when deserialized. Recent PyTorch versions default to a restricted weights-only loader, which reduces but does not remove this risk; keep loaders patched and do not rely on it alone. Security researchers have reported malicious models on public hubs that use this to run payloads on load. |
| Impersonation | Repositories with names similar to popular models or organizations, or reused names after an account is deleted, can trick users into downloading the wrong artifact. |
| Silent change | Referencing a model by name rather than an exact revision means a later download can retrieve different weights. |
| Malicious custom code | Some models require loading repository code, for example by enabling remote code execution in a loading library. That code runs with the permissions of your environment. |
| Backdoored behavior | Weights or fine-tuning data can be manipulated so the model behaves normally except when a trigger appears. Such behavior is hard to find by testing alone. |
| Unclear license or data provenance | Missing documentation about training data and license terms creates legal, compliance and quality risk. |
Checks before production use
1. Source
- Download only from the verified account of the original publisher, and confirm the account through the publisher's own website or documentation.
- Review the model card for training data, intended use, license and known limitations.
2. Format
- Prefer safetensors or other formats that store only tensors and cannot execute code.
- Scan pickle-based files with a dedicated scanner, and load them only in an isolated sandbox with no credentials or network access.
- Do not enable remote code execution in loading libraries for models you have not reviewed.
3. Integrity
- Pin each model to an exact commit or revision, not a branch or tag that can move.
- Record cryptographic hashes of the files you approved, and verify them on every deployment.
- Where the publisher signs models, for example with the OpenSSF model signing specification, verify signatures as part of the pipeline.
4. Behavior
- Evaluate the model on your own tasks and safety tests before deployment, not only on published benchmarks.
- Include tests for unexpected behavior on unusual inputs, and compare outputs with a trusted reference model where possible.
- Accept that testing cannot prove the absence of a backdoor, and limit what the model can access and do accordingly.
5. Record
- Store approved models in an internal registry, and deploy only from there.
- Keep an inventory entry for each model: source, revision, hashes, license, owner, evaluation results and approval date.
- Apply the same controls to datasets and adapters used for fine-tuning.
Limit the impact at runtime
Because no pre-deployment check is complete, design deployments so that a compromised model has limited reach. Run inference with the least privilege it needs, without access to secrets or broad networks, separate model-serving infrastructure from sensitive systems, and monitor for unexpected outbound connections or file access by model-serving processes.
Governance
Make model intake a defined process with an owner, rather than an individual developer's choice. Approval criteria, an exception route and periodic review of the inventory bring models into the same assurance processes as other third-party software, and give governance frameworks such as the NIST AI RMF and ISO/IEC 42001 the inventory they depend on.
References
- Pickle Scanning, Hugging Face Hub documentation
- Safetensors, Hugging Face
- OpenSSF Model Signing (OMS) Specification, OpenSSF AI/ML Working Group, 2025
- model-transparency: model signing implementation, Sigstore
- Supply-chain Levels for Software Artifacts (SLSA), OpenSSF
- LLM03:2025 Supply Chain, OWASP Gen AI Security Project
This advisory draws on public research, standards and official guidance. It is general guidance, not advice for a specific environment. Related training: AI Security Fundamentals, LLM and Agentic Application Security.