Measure quality, reliability, and cost
Build an evaluation that reflects the task instead of relying on a memorable demo.
In plain language
An evaluation defines examples, expected behavior and criteria for judging results. Start with the user’s task: an answer might need to be accurate, supported by evidence, complete and appropriately uncertain. Include typical inputs, difficult cases and requests the system should decline or leave unanswered.
For applications, also measure latency, cost and failure rates. A response that passes one check can fail another. Separate retrieval problems from generation problems, and compare changes against a baseline on the same test cases.
A search assistant passes nine easy questions but invents an answer for an unavailable policy. Add no-answer cases and check citation support before calling the update an improvement.
Try it yourself
Write three checks for a support assistant: one answerable question, one unsupported question and one that must not reveal private information.
Go deeper · for developers and architects
Keep a held-out set and track model, prompt, dataset and tool versions. Inspect subgroup outcomes and false positives/negatives. Automated judges can help at scale but need calibration against appropriate human review.
Check your understanding
2 related questions, with an explanation for each answer.
Explore the original sources
Google evaluation metrics ↗Report an issue
Tell the editors what needs correcting. Reports are reviewed and do not automatically remove content. Do not include personal or confidential information.
Finished this lesson?
Completion records your choice, not a measured skill level.