Constrained Decoding Models Explained: Encoders, Decoders and When to Use Them

Have you ever asked an LLM for JSON and got back "Sure! Here is your JSON:" followed by the JSON? Your parser fails, your pipeline stops, and someone adds a retry loop.

I have been there. For a long time, the fix was a polite prompt. "Reply with valid JSON only. No extra words." It works most days. On the other days, it breaks.

There is a better fix, and it does not rely on the model behaving well. It is called constrained decoding.

In this article, you will learn:

• What encoders and decoders are, and when to use each
• How constrained decoding works, in plain words
• Which tools do it today
• What research says about its strengths and limits
• How it compares with a plain LLM, and with new decision models like Jev and Laya
• How to use it safely inside an agent

A quick note on the name. In this article, CDM means constrained decoding model: a decoder LLM run together with a constraint engine. CDM is the label I use for this setup. It is not a product name.

Encoder and Decoder in Plain Words

Almost every modern language model is built from blocks called transformers. What changes is how the blocks are arranged. A survey of language models for software work describes three layouts: BERT as encoder-only, LLaMA as decoder-only, and CodeT5+ as encoder plus decoder (arXiv 2408.02479).

Here is the easy way to remember it.

An encoder is a reader. It looks at the whole message at once, and every word can see the words on both sides. The output is understanding, a set of numbers that capture meaning. That is why encoders are good at jobs like classifying, routing, searching, and filtering. Researchers who built small safety checkers note that encoder models are efficient at narrow jobs such as classification, name tagging, and sentiment (arXiv 2410.08442).

A decoder is a writer. It produces one small piece of text, called a token, at a time. Each new piece depends on what it already wrote. Chat models like Llama work this way. Decoders became popular because they are flexible: almost any task can be turned into "write the answer."

An encoder plus decoder reads one thing and writes another, like a translation or a summary. T5 and CodeT5+ are examples.

A CDM sits in the fourth row. Jev and Laya are closer to the encoder idea. Laya, for example, is built on a ModernBERT encoder with a decision layer on top (Laya model card).

What Is Constrained Decoding?

A decoder picks its next token by scoring every token it knows. Constrained decoding adds a step. Before the model picks, a rule engine blocks the tokens that would break your rules. The XGrammar paper describes this as restricting the available tokens at each step (arXiv 2411.15100).

The loop has five steps:

1. Score: the model scores every possible next token.
2. Check: the rule engine works out which tokens are legal at this spot.
3. Mask: illegal tokens are blocked.
4. Pick: the model chooses from the legal tokens only.
5. Repeat: the token is added, and the loop runs again until the answer is complete.

The values in the close-up part of the picture are made up to show the idea. What matters is the shape of the process: blocked tokens disappear, and the remaining choices share the probability.

What Kinds of Rules Can You Use?

The common ones are a JSON schema, a regular expression, a fixed list of choices, or a grammar. vLLM, a popular serving engine, lists all four (vLLM docs).

The Outlines library shows how simple this can be. For a yes-or-no answer, you use a Literal["Yes", "No"] type. For a sentiment label, you use Literal["Positive", "Negative", "Neutral"] (Outlines on GitHub). The model can no longer return anything else.

That means a choice constraint turns a general LLM into a classifier that can never invent a label.

One Thing to Remember

Masking controls what the model is allowed to write. It does not change what the model understands. Everything in the research section below comes back to this point.

Tools You Can Use Today

The tools fall into three groups: engines that compute the mask, servers that run them, and hosted APIs that hide them.

Two things to watch for.

Hosted APIs support a part of JSON Schema, not all of it. For example, Google's documentation for Claude on Vertex AI says some constraints, such as numeric and string-length rules, are not supported (Vertex AI docs). If a rule is not enforced by the mask, check it in your own code.

Your backend can change under you. vLLM's configuration notes say the automatic backend choice can change between releases (vLLM config). If you need repeatable behavior, set the backend yourself and pin the version.

What the Research Says

I read the main papers so you do not have to. Several 2026 papers are preprints, which means they have not been peer-reviewed yet. Here are the five points that matter most.

1. A valid format becomes a guarantee. A 2026 preprint on small open models found that constrained decoding removed format failures such as wrong types and missing fields (arXiv 2609.23742). If a program must parse the output, this is the main reason to use it.

2. Valid is not the same as right. The same paper found that the content can still be wrong or thin. A model that skips a required step still passes the schema if the schema allows it. The authors describe rescued answers that look complete but carry little meaning. The short version: constraints fix form, not understanding.

3. Real schemas are harder than demos. JSONSchemaBench tested several engines, including Guidance, Outlines, llama.cpp, XGrammar, OpenAI, and Gemini, on a large set of real-world schemas. Engines differ in which schema features they support (arXiv 2501.10868). That benchmark is from early in this wave of tools, and engines have improved, so test your own schemas on the engine you plan to use.

4. Strict formats can hurt reasoning. The paper "Let Me Speak Freely?" found that format restrictions can weaken reasoning, and stricter formats can do more harm. It also found that constraints can help classification (arXiv 2408.02442). In reasoning tasks, JSON mode often did not keep the order of "think first, answer second." So if the model must think, let it think first.

5. Masking can nudge what the model prefers. Research on grammar-aligned decoding shows that blocking tokens can distort the model's own probabilities, so some outputs are valid but lower in quality (arXiv 2405.21047). A 2026 preprint on draft-conditioned decoding targets a related problem, where masking pushes the model down a path that is valid but wrong (arXiv 2603.03305).

CDM vs. LLM vs. Jev and Laya

Now we can compare the designs. All three can give you a valid answer from a fixed list. They get there in different ways.

What Are Jev and Laya?

They are decision models, sometimes called System One models, after the fast, gut-level mode of thinking. You give them an input, a question, and the allowed answers. They return probabilities for each answer in one pass. They do not write text.

Jev comes from TypeSafe AI and is offered as a hosted service. Its evaluation paper describes typed answers: a choice, a position on a rubric, or a true-or-false probability (arXiv 2609.37647). Laya comes from Convai Innovations and is open, so you can host and fine-tune it yourself. Its model card explains that it is trained so that its probabilities are honest, a method it calls RLCD (Laya model card). I covered both in Classification Never Left. It Just Got a New Home in LLMs.

How Do They Differ From a CDM?

Think of a quiz with four answers.

A CDM is a writer who is only allowed to write one of the four answers. It still goes through the writing loop, and it can also handle open-ended formats like SQL.

A decision model reads the question once and hands you a score for each of the four answers. It cannot write anything else, and it is built for calibrated scores.

ScaleDown's CDMs, covered in the next section, are a third variant. They are small models trained for one task, so the limit on what they can return is built in during training instead of added by a mask at run time.

What About a Head-to-Head Test?

I looked for a direct comparison of a constrained LLM against Jev or Laya on the same data and did not find one. So I will not name a winner. If you are choosing between them, test both on a few hundred of your own inputs.

One weakness is shared. A closed list forces an answer. If a message fits none of your categories, both a decision model and a constrained LLM will still pick one. This follows from how the list works, and it is why you should always add an "other" option and test it on purpose.

When to Use What

Walk the questions from left to right and stop at the first yes.

Two rules of thumb:

1. If the model must think, let it think first. Give it a free reasoning step, then constrain only the final answer. If you must use one schema, put the reasoning field before the answer field.
2. If the answer is a pick, constrain it tightly. Picks are where constraints help most.

How ScaleDown Brought CDMs Into Agents

A recent post from LangChain shows where this idea is heading. Long-running agents spend a lot of effort on context management: summarizing old history, shrinking large tool results, and pulling the right values out of long inputs. Each of those is a small decision, and making them with a large LLM on every turn is slow and costly.

ScaleDown's answer is a set of CDMs. Instead of writing any string one token at a time, each model is trained for one job and only returns answers that job allows, such as a label from your list or pieces of the input text. ScaleDown offers models to summarize, compress, extract, and classify, and the langchain-scaledown package wraps them as middleware for LangChain and Deep Agents. The big LLM keeps the planning and writing, and the CDMs handle the high-volume steps around it (LangChain blog).

Its classify model will feel familiar if you have read about Jev. Both return a typed answer with a probability, and the post says ScaleDown also adds a short reason, which helps when you debug why an agent stopped or routed a task. The post also notes that ScaleDown is now available through the LangSmith LLM Gateway, so its calls are traced like your other model calls.

One caution. This is a vendor-written post, and the speed and cost claims come from ScaleDown's own benchmarks, so test them on your own workload before you rely on them.

Using a CDM Inside an Agent

The two designs work well together. A CDM keeps the agent's tool call valid. A decision model, or a second choice constraint, decides whether that call is safe to run.

Here is how it flows:

1. A user or system sends a request.
2. The LLM agent plans the next step and proposes a tool call.
3. Constrained decoding makes sure the call matches the schema, so it always parses.
4. A decision gate scores the call as allow, review, or block.
5. Allowed calls run, unsure calls go to a person, and risky calls are rejected and logged.

Three habits keep this honest:

1. Test two things. Check that the output is valid, and check that it is right. They are not the same.
2. Log every step. Keep the proposed call, the schema it matched, the gate's scores, and who approved it.
3. Treat the gate as one control. Give the agent only the access it needs, and require a human for high-impact actions.

I wrote more about these controls in AI-Era Cybersecurity, Part 2, and about the tools that sit beside them in Part 1.

A Simple Checklist

Limits of This Article

• Some of the newest evidence is in preprints and small studies. Treat it as a signal, not a final answer.
• Hosted APIs do not publish their full mask engines, so I rely on their documentation.
• I found no same-data comparison between a constrained LLM and a decision model, so the comparison here rests on design differences and separate sources.
• Tools in this area change quickly. Check each official page before you decide.

Related Articles From Me

Conclusion

A CDM is a decoder with guardrails. It gives you a promise that is easy to state and easy to test: the output will always fit the rules. It does not promise that the answer is right, and strict rules can cost you reasoning if you apply them too early.

Decision models like Jev and Laya make a different trade. They give up writing in return for fast, one-pass scoring with probabilities built to be honest. They have their own risks, such as confident wrong picks and forced answers.

So match the tool to the shape of the job. Constrain tightly when the output must parse, or the answer is a pick. Leave the model free when it needs to think. And test both the format and the meaning on your own data before you trust either one.

If you try this in your own system, I would like to hear which part you tried first. Share it in the comments.

References

Papers

• Efficient Guided Generation for Large Language Models (Outlines)
• XGrammar: Flexible and Efficient Structured Generation Engine
• XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs
• JSONSchemaBench
• Let Me Speak Freely? Format Restrictions and LLM Performance
• Grammar-Aligned Decoding
• The Hidden Cost of Structured Generation: Draft-Conditioned Constrained Decoding
• Constrained Decoding Eliminates Structural Failures in Small LLMs
• From LLMs to LLM-based Agents for Software Engineering: A Survey
• JurEE not Judges: Small, Specialized Encoder Ensembles
• Evaluating and Benchmarking the System One Model Jev

Tools and Documentation

• Outlines
• XGrammar
• llguidance
• llama.cpp grammars
• vLLM structured outputs
• NVIDIA NIM structured generation
• OpenAI Structured Outputs announcement
• Claude structured outputs
• Claude structured outputs on Vertex AI

ScaleDown and LangChain

• Building Long-Running Agents with ScaleDown and Deep Agents, LangChain blog
• ScaleDown

Decision Models

• Laya model card, Hugging Face
• Jev: Decision Models on Trial, Cisco Blogs