AI Evals in Practice: For Engineers & PMs

Build trustworthy AI evaluation systems with LLM judges, adversarial testing, RAG metrics, agent evaluation, CI/CD gates, and production monitoring. A live, hands-on ByteByteGo course for engineers and product managers, taught by Manjeet Singh, Senior Director at Salesforce.

What you will learn

• How AI evals work: why unit tests and traditional QA break on non-deterministic systems, and how to measure accuracy, safety, reliability, cost and latency
• Error analysis: turn real failures into your first evals and find where a multi-step agent trace went wrong in minutes, not days
• LLM judges you can trust: grade outputs automatically with LLM-as-a-Judge, write pass/fail rubrics that behave like assertions, and validate every judge against human review
• Red teaming: attack your own agent with prompt injection and jailbreaks, automate red teaming with Promptfoo, and add guardrails that block attacks without blocking real users
• Eval datasets: build golden datasets from real traffic plus validated synthetic data, version them like code, and score RAG pipelines with the RAGAS metrics
• Agent evals: test whether agents call the right tools with the right arguments, judge the full trajectory and final answer, and catch failures hiding in multi-agent handoffs
• Evals in production: wire evals into CI/CD so regressions never merge, monitor live traffic, and catch model and data drift
• A complete capstone: take an untested agent to a gated, monitored pipeline, pick the cheapest model that still meets your quality bar, and turn every incident into a permanent regression test

Course outline

• Eval foundations and error analysis
• Designing reliable LLM judges
• Safety and adversarial testing
• Datasets, RAG and agent evaluation
• Evals in production and the capstone