Misuse Threat Modeling

1. 01

   Concepts & system design

   60 min· read · map the mechanismtimer

   R31 — NIST AI RMF Map & Measure

   R42 — NIST GenAI Profile harmful use & confabulation risks

   Read the assigned sections; map the mechanism, threat model and trade-offs.

2. 02

   Project implementation

   120 min· build · integrate · debugtimer

   Build a product-specific misuse threat model for P1, P2, and P3. Clearly separate external security attacks (e.g., BOLA, injection) from harmful/dual-use misuse by an authorized user.

   Create a versioned safety evaluation dataset containing allowed benign requests, borderline/ambiguous requests, disallowed harmful requests, and adversarial framing. Measure both false negatives (unsafe compliance) and false positives (over-refusal on benign tasks).

   Build, integrate, debug and improve the increment in the active project.

3. 03

   Security lab

   60 min· attack your own systemtimer

   No separate task text for this block on day 64 — the sourced work sits in the block above. Keep the time box.

   Test the abuse case, failure path, authorization bypass or resource limit.

4. 04

   Verification & evidence

   30 min· test · record evidencetimer

   Per-project misuse threat models + safety evaluation baseline reporting false-positive and false-negative rates.

   Add or run regression tests; record the 9-point evidence record.

5. 05

   Deliberate practice & Socratic review

   30 min· explain aloud · update gapstimer

   Answer the 8 Socratic questions without notes: What? Why? How? What if? Why not? How can I prove it? Where can it break? What have I learned?

   Re-explain the hardest concept aloud, inspect one edge case, update the gap log.

6. total5 h

• R31NIST AI RMF 1.0↗ nist.gov

  Govern, Map, Measure, and Manage functions; risk ownership, measurement, and lifecycle documentation.

• R42NIST AI RMF GenAI Profile↗ doi.org

  NIST AI 600-1: generative AI risks and suggested actions for confabulation, privacy, information integrity, and value chain.

Read only the named sections needed for today's task. Reading sits inside the 60-minute concepts block; it is not extra time.

• docsNIST · reference
  AI Risk Management Framework | NIST

  Provides the Govern, Map, Measure and Manage functions, the structure you use to assign risk ownership and record measurements for the misuse threat model across P1, P2 and P3.

  R31checked 2026-11-11
• docsNIST (NIST AI 600-1) · PDF, ~50 pages
  Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile

  Lists GenAI-specific risks such as confabulation and information integrity with suggested actions, which lets you separate security attacks like BOLA from authorized-user misuse in the model.

  R42checked 2026-11-11
• videoGoogle DeepMind Safety Research · 6 min
  Part 3: 3. Dangerous capability evaluations

  Explains evaluating models for harmful capabilities rather than attack vulnerabilities, the dual-use lens that distinguishes misuse threat modeling from classic security threat modeling.

  R42checked 2026-11-11
• videoNetsec Explained · 22 min
  STRIDE Threat Modeling for Beginners - In 20 Minutes

  Walks the STRIDE categories over a data-flow diagram, giving you a repeatable method to enumerate external attacks before you layer the separate misuse/dual-use analysis on top.

  R31checked 2026-11-11
• repoGitHub (Paul Röttger) · dataset + eval code
  GitHub - paul-rottger/xstest: Röttger et al. (NAACL 2024): "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models"

  Supplies paired safe and unsafe prompts built to expose over-refusal, the template you extend for measuring false positives on benign requests alongside false negatives on harmful ones.

  R42checked 2026-11-11

A lab is complete only when it produces evidence. Installing PyRIT, Garak, Presidio or TransformerLens is not completion. Saved on this device only.