Structured outputs for LLMs: a developer guide (2026)

Getting a language model to return prose is easy. Getting it to reliably return {"status": "approved", "confidence": 0.87} every single time, with no stray text, no missing fields, and no invalid types, used to be the hard part of shipping LLM features to production.

By 2026 this problem is mostly solved at the API level. Most frontier providers support schema-constrained generation directly, meaning the model is not just asked nicely to return JSON, it is structurally prevented from returning anything else. That changes how you should architect the parsing, validation, and error handling in your app.

This guide covers the current state of structured outputs, how the major approaches differ, and how to pick the right one for a given feature.

What "structured output" actually means

Structured output is any technique that constrains an LLM's response to a predictable, machine-parseable shape, typically JSON that matches a schema you define, rather than free-form text.

There are three broad approaches in production today, and they are not interchangeable:

1. Prompted JSON: you ask the model to return JSON in the prompt. The model tries its best. Nothing enforces the shape.
2. Constrained decoding / JSON mode: the provider enforces valid JSON syntax, and in the strict variants, enforces your exact schema, at the token-sampling level.
3. Tool or function calling: the model chooses to invoke a defined function with typed arguments, and the provider validates the arguments against your schema before returning them.

Each solves a different part of the problem, and the differences matter once you are past a demo.

Approach 1: Prompted JSON (the fallback, not the plan)

This is "please respond only in JSON" in your system prompt, with no provider-side enforcement.

It works most of the time with capable models. It also occasionally returns:

• JSON wrapped in a markdown code fence
• A sentence of preamble before the JSON ("Sure, here's the JSON you requested:")
• Missing fields the model decided were not important
• Valid JSON with the wrong types (a number as a string, a null where you expected an array)

None of these failures are common with strong models in 2026, but "occasionally" at scale means real requests failing in production. Prompted JSON should be treated as a fallback path for providers or models that do not support stronger guarantees, not the primary strategy for anything customer-facing.

If you must use it, always wrap parsing in a try/catch, strip markdown fences defensively, and validate the parsed object against a schema before trusting it.

Approach 2: Constrained decoding and JSON mode

Constrained decoding works at a lower level than the prompt. Instead of hoping the model produces valid JSON, the provider restricts which tokens the model is allowed to sample at each step, so it is structurally impossible to produce invalid syntax.

Two tiers exist depending on the provider and model:

• JSON mode: guarantees syntactically valid JSON. Does not guarantee it matches your schema. You can still get {} or a JSON object with the wrong keys.
• Schema-constrained JSON (strict mode): guarantees both valid JSON and conformance to a schema you supply, typically JSON Schema. Required fields will be present, types will match, and enums will be respected.

Strict schema-constrained generation is the strongest guarantee available without a second validation pass, because the constraint happens during generation, not after. As of 2026, this is broadly supported across frontier model APIs (OpenAI's Structured Outputs, Anthropic's tool-based schema enforcement, and Google's response schema parameter on Gemini), though exact feature names and support levels vary by model tier, so check current provider docs before assuming parity across every model in a family.

When to use it

Use schema-constrained generation whenever the output feeds directly into your application logic without a human reading it first: classification results, extracted entities, form-filling, routing decisions, or anything stored directly in a database column with a fixed type.

Approach 3: Function and tool calling

Function calling frames the model's job differently: instead of "write JSON," the model is told "you have access to these functions, decide if and how to call them." The model returns a structured call with arguments matching your function's parameter schema.

This is the right model when:

• The model needs to decide whether to produce structured output at all (should it answer in text, or call a tool?)
• You are chaining multiple possible actions and want the model to pick one or several
• You want the same schema enforcement as strict JSON mode, but framed as an action rather than a data blob

Function calling and schema-constrained JSON solve overlapping problems from different angles. If your use case is "always produce this exact shape," plain schema-constrained output is simpler. If your use case is "decide what to do, then produce arguments for it," function calling is the better fit.

Comparing the approaches

Most production systems in 2026 use a mix: function calling for agent decision points, strict schema-constrained output for deterministic extraction and classification tasks, and prompted JSON only where a provider or model genuinely lacks stronger support.

Design your schema like an API contract

Whichever approach you use, the schema itself is doing real engineering work. Treat it with the same care as a public API contract, because the model is effectively an unreliable caller of that contract.

A few practices that consistently reduce failures:

• Keep required fields minimal. Every required field is a chance for the model to hallucinate a value when it does not actually have one. Make fields optional when "unknown" is a valid state.
• Use enums instead of free strings wherever possible. "status": "approved" | "rejected" | "pending" is far more reliable than an open string field you then have to normalize.
• Avoid deeply nested structures when you can flatten. Deep nesting increases the chance of a subtly wrong shape, even under strict constraints, and makes downstream parsing more fragile.
• Add a confidence or reasoning field for anything high-stakes. Even with a guaranteed shape, you often want a signal for when to route to human review versus auto-accept.
• Version your schemas. Treat schema changes like breaking API changes. If you add a required field, old cached completions or retries built against the old schema will fail validation.

Validate even when the provider guarantees the schema

Provider-side schema enforcement is strong, but it is not a substitute for validation in your own code, for a few concrete reasons:

• Provider outages or fallback models may not support the same guarantee tier. If your code silently trusts the shape without checking, a degraded fallback path can push malformed data downstream.
• Semantic correctness is not syntactic correctness. The model can return a syntactically perfect object with a nonsensical value, like a "confidence": 1.4 when your range is 0 to 1, or a date field with an internally inconsistent value.
• Multi-provider setups are common by 2026 for cost and reliability reasons. A schema that is strictly enforced on one provider may only be loosely enforced (or unsupported) on another you fail over to.

Run every structured response through a runtime validator (Zod, Pydantic, or your language's equivalent) before it touches business logic. This is a small amount of code that prevents an entire class of production incidents.

Handling failures gracefully

Even with strict schema enforcement, you should design for the failure case, because it will happen: a rate limit forces a fallback to a weaker model, a provider has an incident, or a genuinely ambiguous input produces a technically valid but useless response.

A reasonable failure strategy:

1. Validate the response against your schema on receipt.
2. On validation failure, retry once with a slightly more explicit prompt or a stricter enforcement mode if you were not already using one.
3. If it fails twice, fall back to a safe default or route to a human review queue rather than surfacing a broken state to the end user.
4. Log the raw response for failed validations. These logs are the fastest way to spot schema issues or model regressions before users report them.

Treat structured output failures the same way you would treat a flaky third-party API: expected occasionally, handled explicitly, never silently swallowed.

Structured outputs in agentic workflows

Agent systems lean on structured outputs even more heavily than single-turn features, because each step in a multi-step agent typically needs to produce a machine-readable decision that feeds the next step.

A few patterns specific to agents:

• Use function calling for the routing layer. Let the model choose which tool to call next, with arguments validated against each tool's schema.
• Keep intermediate structured outputs small. Agents that pass large structured blobs between steps accumulate context and cost quickly. Extract only what the next step needs.
• Log every structured decision in the chain. When an agent produces a wrong final answer, the structured intermediate outputs are what let you debug which step went wrong, rather than guessing from a final unstructured response.

A simple decision guide

If you are unsure which approach to reach for on a new feature, this order of questions helps:

1. Does the model need to decide whether to act, or just extract/classify? If it is deciding among actions, use function calling.
2. Is the output going straight into application logic or a database? If yes, use strict schema-constrained JSON, not prompted JSON.
3. Is this a low-stakes prototype or an experiment? Prompted JSON with a validation layer is fine to move fast, but plan to upgrade before shipping broadly.
4. Are you running on multiple providers or models? Confirm schema enforcement support on every model in your fallback chain, not just the primary one.

Frequently asked questions

Do I still need a JSON schema if the model rarely gets it wrong?

Yes. "Rarely" at scale is still a real failure rate, and the cost of an unvalidated malformed object reaching your database or a user-facing screen is usually far higher than the cost of adding a validation step.

Is function calling slower than plain JSON mode?

Latency differences are typically small and depend more on model choice and response length than on the mechanism itself. Benchmark your specific use case rather than assuming one approach is inherently faster.

Can I mix structured and unstructured output in one response?

Some providers support this (structured tool calls alongside free-text explanation), but it adds complexity to parsing. If you need both, it is often cleaner to make a separate call, or a separate field, for the explanation.

What happens if I add a new required field to my schema later?

Treat it as a breaking change. Any cached responses, retries, or logs generated against the old schema may not satisfy the new required field. Version your schemas or make new fields optional with sensible defaults.

Which provider has the best structured output support in 2026?

Support has converged considerably across major providers, and the practical answer depends on which models you are already using for quality and cost reasons. Check the current documentation for your specific provider and model tier, since support tiers still vary by model size.

The honest summary

Structured outputs stopped being a prompting trick and became an API-level guarantee. That is a genuine reliability upgrade, but it does not remove the need for good schema design, runtime validation, and explicit failure handling in your own code.

Use schema-constrained generation as the default for anything that feeds application logic. Use function calling when the model needs to decide what to do, not just how to format an answer. Validate everything anyway, log failures, and design a graceful fallback path. Do that, and structured outputs become one of the more boring, reliable parts of an LLM-powered feature, which is exactly what you want from infrastructure.

If you are building a developer tool or AI product and want early feedback plus a free, indexed profile, submit it to makers.page. Real usage, real feedback, and a dofollow link if it is a fit.

Related reading:

• How to get your first 100 users without a marketing budget
• Why dofollow backlinks still matter for indie startups