Audio/Transcript Ingestion Boundary
30-Pair RAG QA Evaluation Suite
Jump & context
Week 4: Grounded RAG, Citation Verification, Poisoning, Multimodal Ingestion & RAG Evaluation
The five hours
- 01
Concepts & system design
60 min· read · map the mechanismR04 — LLM Top 10
R45 — NIST Adversarial ML
R14 — Ragas evaluation metrics
R15 — BEIR benchmark methodology
Read the assigned sections; map the mechanism, threat model and trade-offs.
- 02
Project implementation
120 min· build · integrate · debugThreat-model audio/speech-to-text ingestion (
source_modality=audio_transcript). *Lab:* Test transcription uncertainty, speaker ambiguity, and spoken instructions attempting to trigger unauthorized actions.Build a repeatable 30-question RAG evaluation dataset. Measure retrieval recall/precision, answer correctness, citation accuracy, and abstention separately. (Include structure-aware/vectorless outline comparison experiment).
Build, integrate, debug and improve the increment in the active project.
- 03
Security lab
60 min· attack your own systemNo separate task text for this block on day 24 — the sourced work sits in the block above. Keep the time box.
Test the abuse case, failure path, authorization bypass or resource limit.
- 04
Verification & evidence
30 min· test · record evidenceAudio modality policy tests + baseline 30-pair RAG evaluation report.
Add or run regression tests; record the 9-point evidence record.
- 05
Deliberate practice & Socratic review
30 min· explain aloud · update gapsAnswer the 8 Socratic questions without notes: What? Why? How? What if? Why not? How can I prove it? Where can it break? What have I learned?
Re-explain the hardest concept aloud, inspect one edge case, update the gap log.
- total5 h
Read today · inside the concepts block
R04 R14 R15 R45Prompt injection, sensitive info disclosure, improper output handling, excessive agency, vector weaknesses, unbounded consumption.
RAG evaluation metrics: context precision, context recall, faithfulness, and answer relevance.
Retrieval evaluation methodology (nDCG@k, Recall@k) and why a single metric is insufficient.
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (evasion, poisoning, privacy, abuse).
Read only the named sections needed for today's task. Reading sits inside the 60-minute concepts block; it is not extra time.
Free resources for this day
5 curated- docsRagas Documentation · tutorialEvaluate a simple RAG system
Shows building an evaluation dataset and scoring retrieval versus generation separately, the structure for your 30-question set with distinct recall, correctness, citation and abstention metrics.
R14checked 2026-11-11 - videoWelcomeAIOverlords · 15 minHow to use LLMs to build an eval dataset for your RAG system (ep 1)
Covers generating question/answer pairs from your own corpus and why dataset quality bounds the whole evaluation, directly applicable to assembling 30 labeled pairs.
R14checked 2026-11-11 - videoEvidently AI · 7 min6.1 How to evaluate a RAG system: methods and metrics
Separates retrieval quality from generation quality and introduces ground-truth-based evaluation, the split you need to report recall/precision apart from answer correctness.
R14R15checked 2026-11-11 - articleCheck Point Blog · research articleAudio Prompt Injection: What Voice Agents Hear That Text Guardrails Miss
Shows why checking the transcript alone misses signal-level audio injection, the threat-model gap in treating a speech-to-text transcript as trustworthy text.
R04checked 2026-11-11 - repoOpenAI (GitHub) · model + CLIopenai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision
Local speech-to-text to generate source_modality=audio_transcript chunks, letting you test transcription uncertainty and speaker ambiguity hands-on.
R04checked 2026-11-11
Evidence record · 9-point standard
A lab is complete only when it produces evidence. Installing PyRIT, Garak, Presidio or TransformerLens is not completion. Saved on this device only.
Day notes
emptyThe two source formulations
verbatim · nothing summarisedProject & Stage Target
P2 | M3, Core Evaluation
Focus 1 (Audio/Transcript Ingestion Boundary — Orig. Day 30)
Threat-model audio/speech-to-text ingestion (source_modality=audio_transcript). *Lab:* Test transcription uncertainty, speaker ambiguity, and spoken instructions attempting to trigger unauthorized actions.
Focus 2 (30-Pair RAG QA Evaluation Suite — Orig. Day 31)
Build a repeatable 30-question RAG evaluation dataset. Measure retrieval recall/precision, answer correctness, citation accuracy, and abstention separately. (Include structure-aware/vectorless outline comparison experiment).
Verification & Evidence
Audio modality policy tests + baseline 30-pair RAG evaluation report.
Read Today (60m Concepts Block)
R04 (LLM Top 10), R45 (NIST Adversarial ML), R14 (Ragas evaluation metrics), R15 (BEIR benchmark methodology).
---