Training & Fine-Tuning Security

Best time to learn: After Module 1 and after you understand model evaluation. Place it after the model lifecycle and evaluation portions of the existing roadmap, not before you know how to establish a clean baseline.

Prerequisites - Basic supervised learning and fine-tuning concepts. - Train/validation/test separation. - Model evaluation and reproducibility. - Module 1 artifact provenance.

### Threat model

Training-time attacks influence model behavior before inference. They are distinct from RAG poisoning and runtime prompt injection.

Study: - Pretraining and fine-tuning data poisoning. - SFT and preference-data poisoning. - Trigger-based backdoors and targeted behavior changes. - Label manipulation and data provenance failures. - Reward-model and preference optimization manipulation at a conceptual level. - Evaluation-set contamination and benchmark leakage. - Model checkpoint substitution. - Data lineage, deduplication, outlier review, and dataset versioning. - Backdoor persistence, transfer, and limitations of simple detection. - Why cleaning data does not guarantee a backdoor has been removed from an already trained model.

### Labs

• Clean baseline (1h): Use a small model or toy classifier and a synthetic, harmless task. Save model version, dataset version, seed, and baseline metrics.

• Synthetic poisoning (3h): Add a small controlled set of trigger-associated examples to a toy fine-tuning dataset. Keep the trigger and task benign and confined to your local experiment.

• Behavioral evaluation (2h): Compare clean inputs, trigger inputs, and unrelated inputs. Report false positives, false negatives, and changes in normal task performance.

• Mitigation comparison (2h): Compare data filtering, provenance checks, retraining from a clean checkpoint, and targeted evaluation. Explain why no single mitigation is guaranteed.

• Dataset governance (2h): Add dataset versioning, source tracking, review status, and a poisoning-test report to the project documentation.

### Project integration

• Project 1: Add a model provenance and behavior-regression record, even if you do not fine-tune its local model.

• Project 2: Treat embedding model, reranker, answer model, evaluation data, and source documents as separately versioned assets.

• Project 3: Ensure an agent's policy or behavior configuration cannot be silently replaced by an unreviewed fine-tuned checkpoint.

### Measurable completion criteria

• You can explain the difference between retrieval poisoning, training-data poisoning, and inference-time prompt injection.

• You have a reproducible synthetic poisoning experiment with a clean baseline.

• You report trigger behavior and collateral impact on normal task performance.

• You can describe at least three mitigations and their limitations.

• Dataset and checkpoint provenance are represented in your project documentation.

Do not overreach: Do not try to reproduce large-scale foundation-model poisoning or claim that a small toy experiment predicts real-world backdoor prevalence.

------------------------------------------------------------------------