Confidence, Sampling, and Provenance — Detail & Examples

Part of Context Management · overview: domain-5.html

In this page

Why confident-sounding output isn't the same as verified output, a worked temperature choice for two different task types, and provenance shown in practice.

5.4aConfidence-sounding isn't verified

Two different things that look similar

A model can state an answer fluently and confidently while still being wrong — fluency is not evidence of correctness.
Calibration (does stated confidence track actual accuracy) tends to be weaker on questions far outside common patterns in training data — exactly where verification matters most.
System design should treat a confident-sounding claim as a claim to check, not as ground truth on its own, especially for high-stakes outputs.

5.4bSampling temperature, by task type

WORKED EXAMPLE · PICKING A TEMPERATURE

TASK AExtract structured fields (name, date, amount) from an invoice for a database.

This task has one correct answer per invoice — there's no benefit to variety.
A low temperature setting favors the most likely, most consistent output every time the same invoice is processed.

Choice:Low temperature — consistency and repeatability matter more than variety for structured extraction.

Contrast with a brainstorming task ("give me 5 different taglines for this product") where a higher temperature is the right call, because variety across the 5 outputs is the actual goal.

5.4cProvenance makes output auditable

OUTPUT (no provenance)"The competitor's price increased 15% this year." — if this is wrong, there's no way to trace why.

OUTPUT (with provenance)"The competitor's price increased 15% this year [source: press release, March 2026, retrieved via web search]." — a wrong claim can now be traced back to its source and checked or corrected directly.

← 5.3 Cross-Session Knowledge and Coherence Context Management Overview

Claude Certified Architect — Study NotesDWG 5.4