Pydantic AI structured outputs and evals on Bedrock

When an LLM first breaks in production it’s rarely dramatic. It returns valid JSON, your parser is happy, and then three services downstream something falls over because passengers came back as the string "two" instead of the integer 2. Nothing threw and nothing logged an error. The shape was right and the meaning was wrong, and you spend an afternoon working out which one of those it was.

So I’ve ended up trusting a model in stages. You lock down the shape of what comes back first. Then you ask whether the content inside that shape was any good. Then you find a way to judge the open-ended parts that no exact-match assertion is ever going to cover. Pydantic AI and Pydantic Evals cover that whole span, and a lot of the moving parts have landed or matured in the last few months, so it’s worth walking end to end.

flowchart LR
    shape["1. shape<br/>structured outputs<br/>(constrained decoding)<br/>is it the right shape?"] --> content["2. content<br/>Pydantic Evals<br/>cases + evaluators<br/>is the content correct?"] --> judge["3. judgement<br/>LLM-as-judge<br/>calibrated against your own labels<br/>was the open text any good?"]

Locking the shape with structured outputs #

Getting structured data out of a model used to mean one of a few hacks, and all of them leak.

The first is prompt-and-pray. You ask for JSON and you get back valid-looking JSON wrapped in a markdown fence, or with a cheerful sentence of preamble in front of it, or with a trailing comma that kills your parser. So you write a parse step, then a retry loop, then a little function that strips fences, and now you own a pile of glue code just to keep the output parseable.

JSON mode is the next step up. It guarantees the output is syntactically valid JSON, which is better, but it’s a low bar. Nothing in JSON mode stops the model returning the wrong fields, the wrong types, or a number where you asked for a string. You still need validation and you still get failures, just one step further downstream.

Then there’s the tool-calling trick: you define a function whose parameters are your schema and let the model “call” it. It works, but it’s a semantic mismatch, because you’re bending tool-invocation plumbing into a data-extraction job and carrying the tool-calling overhead for something that was never a tool.

Structured outputs fix this at the decoder, where it should be fixed. The provider compiles a grammar from your schema and constrains token generation to it, so the output is guaranteed to parse into the schema rather than usually parsing into it. That takes out the retry loops and the fence-stripping glue, along with the whole class of bug where a downstream system trips over a field that came back as null instead of 0.

Pydantic AI is how I wire that into Python. Give it a BaseModel as the output_type and it derives the JSON schema, uses the provider’s structured-output mode, then validates the response into a typed object. You get IDE autocomplete, type checking and your own field validators on top of the provider’s decoder-level guarantee. There is no hand-written parsing layer in between.

If you’ve spent any time hand-rolling try/except json.loads with a retry counter, all of that boilerplate just goes away. Here’s the before, a version I’ve written more than once:

raw = call_model(prompt)
try:
    booking = Booking(**json.loads(strip_fences(raw)))
except (json.JSONDecodeError, ValidationError):
    raw = call_model(prompt + "\n\nReturn ONLY valid JSON.")  # ask nicely, again
    booking = Booking(**json.loads(strip_fences(raw)))         # and hope

And the after:

agent = Agent("bedrock:anthropic.claude-sonnet-4-5-20250929-v1:0", output_type=Booking)
booking = agent.run_sync(prompt).output  # already a typed Booking, guaranteed

All of that glue is gone, and the guarantee lives in the decoder now.

If you’re AWS-native like I am, this stopped being a client-side trick on Bedrock.

AWS shipped structured outputs natively on Amazon Bedrock on the 4th of February 2026, and the mechanism is the one I described above: constrained decoding for schema compliance, with no prompt engineering or extra checks bolted on the side. There are two ways in. You define a JSON schema for the response format, or you use strict tool definitions so the model’s tool calls match your spec. It went generally available for Anthropic’s Claude 4.5 models and a set of open-weight models, across the Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream APIs, and reached GovCloud (US) on the 1st of April 2026.

Before this the Converse API left you reaching for function calling as the workaround, the same semantic mismatch as everywhere else. Now the guarantee comes from Bedrock itself. For a Pydantic AI stack running on Bedrock, the constrained decoding happens server-side and Pydantic AI just validates what comes back, with nothing faked in between.

Checking the content with Pydantic Evals #

A guaranteed schema tells you the response is well-formed. It tells you nothing about whether the content is correct, or even true. The model can hand you a beautifully typed object full of confident nonsense and your validators will wave it straight through, because every field is exactly the type you asked for.

That’s the gap evals fill, and Pydantic Evals is the harness for it.

The framework rests on three abstractions. A Case is one test: an input, an optional expected output, and some metadata. Think of it as a single row in a test table. A Dataset is a collection of Cases you run together. An Evaluator is the thing that looks at an output and decides pass or fail, or hands back a score.

You write a task function that wraps your agent, call dataset.evaluate(task), and it runs every Case, applies the evaluators, and gives you back a report with pass rates and scores. If you’ve written a parametrised pytest before, none of this will throw you.

A handful of evaluators ship in the box, and they cover more ground than you’d expect:

• EqualsExpected, Equals, Contains for exact and substring matching, for the cases where the answer really is deterministic.
• IsInstance checks the output is the right Pydantic model or type, so it slots straight into a stack that’s already validating shape.
• MaxDuration gives you a latency budget. A correct answer that takes nine seconds is still a failure in production.
• LLMJudge takes a rubric and grades the output with an LLM, either binary pass/fail or a 0 to 1 score, with a swappable judge model.

Custom evaluators are just a class that returns a bool, an int or float, or a string. The bool is an assertion, the number is a score, the string is a label. That’s the entire contract, and it’s enough to build whatever the built-ins don’t cover.

Put together, a small suite reads about how you’d hope:

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance, LLMJudge

dataset = Dataset(
    cases=[
        Case(
            name="refund_within_window",
            inputs="bought this yesterday, I want my money back",
        ),
    ],
    evaluators=[
        IsInstance(type_name="SupportReply"),
        LLMJudge(rubric="Reply approves the refund and stays civil about it"),
    ],
)

report = dataset.evaluate_sync(handle_ticket)
report.print(include_input=True, include_output=True)

The two checks answer different questions. IsInstance confirms that the response is the right kind of object; LLMJudge asks whether the answer inside it was any good.

There’s also non-determinism to handle. Run the same prompt twice and you can get two different answers even at temperature zero, because the inference stack isn’t batch-invariant and you don’t control the batch size on a hosted API. A single pass or fail tells you almost nothing.

Pydantic Evals handles this with a repeat parameter. Set repeat=5 and each Case runs five times, and you get back a pass rate instead of a flaky single result. There’s also Tenacity-based retry handling for transient failures, so a rate limit doesn’t take down the whole run.

Pick your pass-rate threshold deliberately. You almost never want 100%. You want a number that reflects how much variation your users will tolerate and how bad a failure actually is. That’s a product call as much as a technical one.

Judging the open-ended parts #

LLMJudge is where this gets useful, because most real outputs can’t be graded by exact match. Was the summary faithful to the source? Did it answer the question that was asked? That needs judgement, and a second model can supply it: the LLM-as-judge pattern.

An LLM judge is worth nothing until you’ve checked it agrees with a human. Prefer binary pass/fail over a 1 to 5 scale, because nobody can tell you what separates a 3 from a 4 and your reviewers won’t agree on it either. Use a strong judge model, ideally from a different family than the one you’re grading, so you’re not just measuring a model’s fondness for its own output. And calibrate against your own labels before you trust a single number it produces. If the judge and a human disagree half the time, the score tells you almost nothing about quality.

Used well, it gives you a quality gate for the parts structured outputs can’t reach. The framework won’t calibrate the judge for you, though. I learnt that the long way round while building lgtmaybe, where an uncalibrated reviewer produced confident findings nobody could trust.

Evals don’t have to stop at the final output, either. With the Logfire extra installed, Pydantic Evals captures OpenTelemetry spans while your task runs. HasMatchingSpan then lets you assert against the trace itself: which tools were called, in what order, with what arguments. That checks whether the agent took a sane path to get there, as well as whether the final answer was right.

flowchart TD
    input["Case input"] --> task["task function"] --> run["Agent run"]
    run --> t1["tool call: search"]
    run --> t2["tool call: fetch"]
    run --> out["final output"]
    out --> oe["output evaluators:<br/>IsInstance, LLMJudge"]
    t1 --> se["span evaluators:<br/>HasMatchingSpan"]
    t2 --> se
    oe --> report["EvaluationReport"]
    se --> report

For anything agentic, MCP tool calls or multi-step retrieval, output-only evals miss too much. An agent can stumble onto the right answer through a completely broken trajectory, and you won’t know until it hits an input where the luck runs out. Span evaluation catches that class of failure, and I add it to any agent I’d actually ship.

What it doesn’t do #

It’s a regression harness, and the gaps are worth naming so you don’t go looking for things that aren’t there.

There’s no built-in metric library, no G-Eval, no RAGAS-style faithfulness scoring out of the box. No red-teaming. No pass@k, confidence intervals, or statistical significance testing baked in. Judge bias mitigation is manual. The docs are honest that the LLM judge carries length and style biases, and the mitigation is yours to run: temperature zero, multiple judges, and validating against human labels.

I don’t hold any of that against it. It’s code-first, type-safe, Pydantic-native, backend-agnostic, and the library itself is free. Logfire is the optional paid backend if you want somewhere for the traces to land. If you need a wider metric library or a production dashboard, pair it with something like DeepEval or Langfuse rather than expecting Pydantic Evals to be those things. It does one job, turning stochastic output into a number you can gate a merge on, and it does it without adding a platform to your repo.

Structured outputs, now native on Bedrock, guarantee the shape. Pydantic AI turns the result into a typed object. Evals check the content, and a calibrated LLM judge can score the open-ended parts that exact matching will never cover.

If you’re already on Pydantic AI with validators in place, none of this is a rewrite. It’s the next few steps on the same path, and most of them are a few lines of code each.