Run Project 2 release candidate suite

1. 01

   Concepts & system design

   60 min· read · map the mechanismtimer

   R14 — Ragas

   R15 — BEIR

   R28 — PyRIT

   Read the assigned sections; map the mechanism, threat model and trade-offs.

2. 02

   Project implementation

   120 min· build · integrate · debugtimer

   Execute the complete P2 Release Candidate suite: 30+ QA pairs, single-hop and C2 multi-hop retrieval/citation metrics, abstention accuracy, tenant ACLs, C3/C7 ingestion/extraction tests, D5/D6/D10 security regressions, D8 PyRIT CI gate, D12 SBOM, and Docker Compose deployment check.

   Build, integrate, debug and improve the increment in the active project.

3. 03

   Security lab

   60 min· attack your own systemtimer

   No separate task text for this block on day 84 — the sourced work sits in the block above. Keep the time box.

   Test the abuse case, failure path, authorization bypass or resource limit.

4. 04

   Verification & evidence

   30 min· test · record evidencetimer

   Freeze P2 RC package: architecture diagram, threat model, benchmark & security report, and residual-risk statement.

   Add or run regression tests; record the 9-point evidence record.

5. 05

   Deliberate practice & Socratic review

   30 min· explain aloud · update gapstimer

   Answer the 8 Socratic questions without notes: What? Why? How? What if? Why not? How can I prove it? Where can it break? What have I learned?

   Re-explain the hardest concept aloud, inspect one edge case, update the gap log.

6. total5 h

• R14Ragas Documentation↗ docs.ragas.io

  RAG evaluation metrics: context precision, context recall, faithfulness, and answer relevance.

• R15BEIR Benchmark↗ github.com

  Retrieval evaluation methodology (nDCG@k, Recall@k) and why a single metric is insufficient.

• R28Microsoft PyRIT↗ github.com

  Python Risk Identification Tool: targets, orchestrators, converters, scorers, and multi-turn attack automation.

Read only the named sections needed for today's task. Reading sits inside the 60-minute concepts block; it is not extra time.

• docsRagas · docs
  Context Recall

  Defines context recall as the share of reference claims attributable to retrieved context, with LLM-based, non-LLM, and ID-based variants — the metric that separates retrieval failure from generation failure in the RC run.

  R14checked 2026-11-11
• repoBEIR / UKP Lab (GitHub) · benchmark framework
  BEIR

  Provides EvaluateRetrieval with NDCG@k, MAP@k, Recall@k, and Precision@k over heterogeneous datasets, so the 30+ QA pairs are scored with standard IR metrics instead of a single similarity number.

  R15checked 2026-11-11
• videoShrijayan · 20:23
  RAG Evaluation Metrics Explained: Context Precision, Recall, Relevancy & Faithfulness

  Walks each Ragas metric with worked examples and a metrics mindmap, which is the interpretation key for reading a failing context-recall score as a missing chunk rather than a bad prompt.

  R14checked 2026-11-11
• videoAnalytics Vidhya
  RAG Evaluation Metrics Tutorial

  Runs a RAGAS evaluation end to end in Python and reads the four scores, giving the mechanics for wiring the 30+ QA pairs and C2 multi-hop cases into one repeatable run.

  R14checked 2026-11-11
• articleNeurIPS Datasets and Benchmarks 2021 · paper
  BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models

  Shows BM25 beating early dense retrievers on several of 18 tasks and that no single retriever wins everywhere, which is the evidence for reporting several metrics across task types instead of one aggregate score.

  R15checked 2026-11-11

A lab is complete only when it produces evidence. Installing PyRIT, Garak, Presidio or TransformerLens is not completion. Saved on this device only.