Open-Source Model Risk
Risk Profile
Open-source models (Llama, Mistral, Mixtral, Falcon, etc.) offer control and cost advantages but introduce supply chain and operational risks.
Key Risks
Model Integrity
- Pickle deserialization: Many model formats execute arbitrary code on load
- Backdoored weights: Malicious models uploaded to public hubs pass benchmarks but contain hidden behaviors
- Fine-tune poisoning: Community fine-tunes may include harmful training data
Operational Risk
- No vendor support: You own the entire stack — inference, monitoring, patching
- Security patches lag: Vulnerabilities in model serving software may not have rapid fixes
- Talent dependency: Requires ML engineering expertise to operate
Compliance Risk
- License confusion: Some "open" models have restrictive licenses (Llama's acceptable use policy)
- Training data provenance: You may not know what data the model was trained on
- Liability: No vendor to share liability if the model causes harm
Controls
| Control | Implementation |
|---|---|
| Safetensors only | Only load models in safetensors format — no pickle execution risk |
| Hash verification | Verify model file hashes against published checksums |
| Model scanning | Scan model files for malicious payloads before loading |
| Sandboxed inference | Run models in isolated containers with no network access to sensitive systems |
| License review | Legal review of model license before deployment |
| Provenance documentation | Document model source, version, and modification history |
| Safety evaluation | Run safety benchmarks before production deployment |
| Update process | Defined process for updating model versions with testing gates |