Structured Output and Evaluations — Detail & Examples

Part of Prompt Engineering · overview: domain-4.html

In this page

Closing a schema ambiguity for a missing field, what a useful eval set actually needs, and a traced automated grading run that catches a regression.

4.3aClosing a schema gap

WORKED EXAMPLE · HANDLING A MISSING FIELD

BEFOREPrompt asks for JSON with fields: name, email, phone. No guidance on what happens if phone wasn't mentioned in the source text.

On most inputs, the model fills in all three fields correctly.
On an input where phone is genuinely absent, the model sometimes omits the field entirely, sometimes writes "N/A", sometimes guesses — three different behaviors for the same underlying case.
A downstream parser expecting a consistent shape breaks unpredictably depending on which behavior happened to occur.

Fix:State it explicitly: "If phone is not mentioned in the source text, set the field to null. Never guess or omit the field." — now there's exactly one behavior for the missing case.

4.3bBuilding a small eval set

What a useful eval set needs

A handful of representative inputs, including at least one edge case (like the missing-field example above), not just typical cases.
A known-good output or a clear grading rubric for each input, so "did this get better" is a measurable comparison, not a feeling.
Re-run the same set after any prompt change — that's what turns a subjective "this looks better" into evidence a change actually improved things without breaking something else.

4.3cAutomated grading in practice

INPUTRun prompt v2 against all 20 eval cases.

GRADER (a separate Claude call)Score each output against the rubric: does it match the required schema? Is the summary accurate to the source?

RESULT19/20 pass; case 14 (a missing-field edge case) now fails where it passed under prompt v1 — a regression caught before shipping.

← 4.2 Advanced Prompt Techniques Prompt Engineering Overview

Claude Certified Architect — Study NotesDWG 4.3