Add new chapter
Harness Engineering
“Test infrastructure” for AI systems: make results stable, automate regressions, and quantify quality.
Core components
Eval Harness (evaluation infrastructure)
- Define the input set (golden set) and labels
- Define output judgment (rule-based + LLM judge)
- Offline regression: run it after every change
Quality Gate
- Failure threshold (e.g., pass-rate ≥ 95%)
- Critical use cases must all be green (critical path)
- Drift monitoring and alerts (prompt/model/data)
Observability (Observability)
- Record traceId + prompt/version + latency
- Save sampled input/output for review (anonymized)
- Production regression: shadow traffic / canary
Minimum viable product (MVP)
- Select 20 key use cases (covering core flows + high-frequency failure points).
- Write one repeatable runner (usable locally and in CI).
- Define 2 metrics: pass rate, P95 latency; set gate thresholds.
- Every time you change the prompt/model/retrieval, you must run regression tests and record the results.