stage S16h
AI Attack Surfaces & Threat Modeling
Map every trust boundary (`user -> API -> database/vector store -> model -> tools/MCP -> external APIs -> human approval`). Model the agent as a **confused deputy** that has legitimate credentials but can be tricked into misusing them.
Stage detail
When: Weeks 1--2, then revisit when RAG and agents are introduced.
- Map user input, system/developer instructions, conversation history, retrieved documents, model output, tool calls, external APIs, databases, logs, and human approvals.
- Learn the OWASP Top 10 for LLM applications as a threat taxonomy; check the current edition because categories evolve.
- Use STRIDE to prompt questions about spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege.
- Understand the confused deputy: a component with legitimate authority is tricked into using it for a caller who should not have it.
- Threat-model assets, identities, trust boundaries, entry points, data flows, threats, controls, tests, and residual risk.
Practice: threat-model the email-to-CRM agent from the supplied outline using mock email and a local/mock CRM. Identify indirect injection in email bodies/attachments, unauthorized CRM reads/writes, token exposure, data exfiltration, approval bypass, and audit gaps.
Do not learn yet: heavyweight enterprise threat-modeling bureaucracy. One accurate, testable threat model is more useful than a diagram nobody updates.
Study days for this stage
6- 1Set up the development environmentW1S0S1M1R06 R39
- 4LLM Internals & Token BudgetsW1S1S7M1R01 R04 R07 R11
- 8Study confused-deputy risks and least privilegeW2S1S3R02 R10
- 19Review the retrieval pipeline as an attack surfaceW4S0S1R10 R39
- 74End-to-End Cross-Project Attack ChainW13S1S11R05 R31 R39
- 89Create a unified attack taxonomy across projectsW15S1S8D1D2D3D4D5D6D7D8D9D10D11D12R05 R39 R41