Pydantic AI structured outputs and evals on Bedrock

When an LLM first breaks in production it’s rarely dramatic. It returns valid JSON, your parser is happy, and then three services downstream something falls over because passengers came back as the string "two" instead of the integer 2. Nothing threw. Nothing logged an error. The shape was right and the meaning was wrong, and you spend an afternoon working out which one of those it was.

I’ve lost that afternoon more than once, which is roughly why I now think about trusting a model in stages. You lock down the shape of what comes back first. Then you ask whether the content inside that shape was any good. Then you find a way to judge the open-ended parts that no exact-match assertion is ever going to cover. Pydantic AI and Pydantic Evals cover that whole span, and a lot of the moving parts have landed or matured in the last few months, so it’s worth walking end to end.

1. shape                 2. content              3. judgement
   structured    ──►         Pydantic Evals ──►      LLM-as-judge
   outputs                   cases +                 calibrated
   (constrained              evaluators              against your
    decoding)                                        own labels
   ───────────               ──────────────          ─────────────
   is it the                 is the content          was the open
   right shape?              correct?                text any good?

Why structured outputs beat raw JSON #

Getting structured data out of a model used to mean one of a few hacks, and all of them leak.

The first is prompt-and-pray. You ask for JSON and you get back valid-looking JSON wrapped in a markdown fence, or with a cheerful sentence of preamble in front of it, or with a trailing comma that kills your parser. So you write a parse step, then a retry loop, then a little function that strips fences, and now you own a pile of glue code whose only job is to apologise for the model.

JSON mode is the next step up. It guarantees the output is syntactically valid JSON, which is better, but it’s a low bar. Nothing in JSON mode stops the model returning the wrong fields, the wrong types, or a number where you asked for a string. You still need validation, you still get failures, you’ve just pushed them one step downstream.

Then there’s the tool-calling trick: you define a function whose parameters are your schema and let the model “call” it. It works, but it’s a semantic mismatch, because you’re bending tool-invocation plumbing into a data-extraction job and carrying the tool-calling overhead for something that was never a tool.

Structured outputs fix this at the decoder, where it should be fixed. The provider compiles a grammar from your schema and constrains token generation to it, so the output is guaranteed to parse into the schema rather than usually parsing into it. That takes out the retry loops and the fence-stripping glue, along with the whole class of bug where a downstream system trips over a field that came back as null instead of 0. You ask for a shape and that’s what you get.

How Pydantic AI fits in #

You hand it a BaseModel as the output_type, it derives the JSON schema, drives the provider’s structured-output mode, and validates the response straight back into the typed object. You get the decoder-level guarantee and a real Python object: type safety, IDE autocomplete, and your own field validators sitting there as a second line of defence. The model gives you the right shape, Pydantic gives you the right object, and the seam between the two more or less disappears.

If you’ve spent any time hand-rolling try/except json.loads with a retry counter, the difference is hard to oversell. The boilerplate doesn’t shrink, it stops existing. Here’s the before, a version I’ve written more than once:

raw = call_model(prompt)
try:
    booking = Booking(**json.loads(strip_fences(raw)))
except (json.JSONDecodeError, ValidationError):
    raw = call_model(prompt + "\n\nReturn ONLY valid JSON.")  # ask nicely, again
    booking = Booking(**json.loads(strip_fences(raw)))         # and hope

And the after:

agent = Agent("bedrock:anthropic.claude-sonnet-4-5-20250929-v1:0", output_type=Booking)
booking = agent.run_sync(prompt).output  # already a typed Booking, guaranteed

All of that glue is gone, and the guarantee moved into the decoder where it belongs.

Structured outputs on Bedrock #

If you’re AWS-native like I am, the development worth knowing about is that this stopped being a client-side trick on Bedrock.

AWS shipped structured outputs natively on Amazon Bedrock on the 4th of February 2026, and the mechanism is the one I described above: constrained decoding for schema compliance, with no prompt engineering or extra checks bolted on the side. There are two ways in. You define a JSON schema for the response format, or you use strict tool definitions so the model’s tool calls match your spec. It went generally available for Anthropic’s Claude 4.5 models and a set of open-weight models, across the Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream APIs, and reached GovCloud (US) on the 1st of April 2026.

Before this the Converse API left you reaching for function calling as the workaround, the same semantic mismatch as everywhere else. Now the guarantee comes from Bedrock itself. For a Pydantic AI stack running on Bedrock, the constrained decoding happens server-side and Pydantic AI just validates what comes back, with nothing faked in between.

A valid shape doesn’t mean good content #

A guaranteed schema tells you the response is well-formed. It tells you nothing about whether the content is correct, relevant, or actually true. The model can hand you a beautifully typed object full of confident nonsense and your validators will wave it straight through, because every field is exactly the type you asked for.

That’s the gap evals fill, and Pydantic Evals is the harness for it.

The three core abstractions #

The framework rests on three abstractions, and you’ll have them memorised in about five minutes.

A Case is one test: an input, an optional expected output, and some metadata. Think of it as a single row in a test table. A Dataset is a collection of Cases you run together. An Evaluator is the thing that looks at an output and decides pass or fail, or hands back a score.

You write a task function that wraps your agent, call dataset.evaluate(task), and it runs every Case, applies the evaluators, and gives you back a report with pass rates and scores. If you’ve written a parametrised pytest before, none of this will throw you, and that familiarity is the best thing about it.

The evaluators you get for free #

A handful ship in the box and they cover more ground than you’d expect:

• EqualsExpected, Equals, Contains for exact and substring matching, for the cases where the answer really is deterministic.
• IsInstance checks the output is the right Pydantic model or type, so it slots straight into a stack that’s already validating shape.
• MaxDuration gives you a latency budget. A correct answer that takes nine seconds is still a failure in production.
• LLMJudge takes a rubric and grades the output with an LLM, either binary pass/fail or a 0 to 1 score, with a swappable judge model.

Custom evaluators are just a class that returns a bool, an int or float, or a string. The bool is an assertion, the number is a score, the string is a label. That’s the entire contract, and it’s enough to build whatever the built-ins don’t cover.

Put together, a small suite reads about how you’d hope:

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import IsInstance, LLMJudge

dataset = Dataset(
    cases=[
        Case(
            name="refund_within_window",
            inputs="bought this yesterday, I want my money back",
        ),
    ],
    evaluators=[
        IsInstance(type_name="SupportReply"),
        LLMJudge(rubric="Reply approves the refund and stays civil about it"),
    ],
)

report = dataset.evaluate_sync(handle_ticket)
report.print(include_input=True, include_output=True)

The shape check and the judgement check sit side by side, which is the whole point. One asserts the response is the right thing. The other asks whether it was any good.

Handling non-determinism #

Run the same prompt twice and you can get two different answers even at temperature zero, because the inference stack isn’t batch-invariant and you don’t control the batch size on a hosted API. A single pass or fail tells you almost nothing.

Pydantic Evals handles this with a repeat parameter. Set repeat=5 and each Case runs five times, and you get back a pass rate instead of a flaky single result. That one flag is the difference between a suite you trust and one that goes red on a Tuesday for reasons nobody can reconstruct. There’s also Tenacity-based retry handling for transient failures, so a rate limit doesn’t take down the whole run.

Pick your pass-rate threshold deliberately. You almost never want 100%. You want a number that reflects how much variation your users will tolerate and how bad a failure actually is. That’s a product call as much as a technical one, and it deserves more thought than just reaching for the round number.

LLM-as-judge #

LLMJudge is where this gets useful, because most real outputs can’t be graded by exact match. Was the summary faithful to the source? Was the tone right? Did it answer the question that was asked? That needs judgement, and a second model can supply it. This is the LLM-as-judge pattern, and it’s the natural next step after structured outputs and basic evals.

An LLM judge is worth nothing until you’ve checked it agrees with a human. Prefer binary pass/fail over a 1 to 5 scale, because nobody can tell you what separates a 3 from a 4 and your reviewers won’t agree on it either. Use a strong judge model, ideally from a different family than the one you’re grading, so you’re not just measuring a model’s fondness for its own output. And calibrate against your own labels before you trust a single number it produces. If the judge and a human disagree half the time, the score tells you almost nothing about quality.

Used well, it’s a quality gate for the parts of the response structured outputs can’t reach. Used lazily, it’s theatre, and the framework can’t tell the difference. It gives you the mechanism, and the calibration is on you. I went the long way round on that building lgtmaybe, where an uncalibrated reviewer produced confident findings nobody could trust.

Asserting on the trace #

With the Logfire extra installed, Pydantic Evals captures OpenTelemetry spans while your task runs. HasMatchingSpan then lets you assert against the trace itself: which tools were called, in what order, with what arguments. Not just whether the final answer was right, but whether the agent took a sane path to get there.

Case input
    │
    ▼
task function ──► Agent run ──┬──► tool call: search ──┐
                             ├──► tool call: fetch  ──┤
                             └──► final output         │
                                     │                 │
                    ┌────────────────┘                 │
                    ▼                                   ▼
          output evaluators                    span evaluators
          IsInstance, LLMJudge                 HasMatchingSpan
                    │                                   │
                    └─────────────────┬─────────────────┘
                                      ▼
                              EvaluationReport

For anything agentic, MCP tool calls or multi-step retrieval, output-only evals lie to you constantly. An agent can stumble onto the right answer through a completely broken trajectory, and you won’t know until it hits an input where the luck runs out. Span evaluation catches that class of failure, and I add it to any agent I’d actually ship.

What it doesn’t do #

It’s a regression harness, not a platform, and the gaps are worth naming so you don’t go looking for things that aren’t there.

There’s no built-in metric library, no G-Eval, no RAGAS-style faithfulness scoring out of the box. No red-teaming. No pass@k, confidence intervals, or statistical significance testing baked in. Judge bias mitigation is manual. The docs are honest that the LLM judge carries length and style biases, and the mitigation is yours to run: temperature zero, multiple judges, and validating against human labels.

I don’t hold any of that against it. It’s code-first, type-safe, Pydantic-native, backend-agnostic, and the library itself is free. Logfire is the optional paid backend if you want somewhere for the traces to land. If you need a wider metric library or a production dashboard, pair it with something like DeepEval or Langfuse rather than expecting Pydantic Evals to be those things. It does one job, turning stochastic output into a number you can gate a merge on, and it does it without dragging a platform into your repo.

Where this leaves you #

So the picture holds together. Structured outputs, now native on Bedrock, guarantee the shape. Pydantic AI turns that shape into a typed object you can actually work with. Evals tell you whether the content inside the shape was any good. And LLM-as-judge, once you’ve calibrated it, scores the open-ended parts exact match was never going to reach.

If you’re already on Pydantic AI with validators in place, none of this is a rewrite. It’s the next few steps on the same path, and most of them are a few lines of code each.