5 Reasons Deterministic Tooling for AI Agents Beats an LLM-Only Stack

An AI agent writes you a dbt model. The dbt model compiles, it runs and it returns numbers. None of those three steps tells you whether the numbers are right. Some parts of that question have exactly one right answer.

Three of those parts already have a tool that answers them exactly:

• A row-level diff tells you whether a rewritten query returns the same rows as the query it replaced.
• A pattern scan tells you whether a column holds email addresses.
• Column-level lineage names every model that breaks when you rename a field.

Each of those three tools gives the same answer on every run.

A language model can answer those same three questions, and its answer moves. Change the phrasing of the request and the answer can change. Deploy a new model version and the answer can change again. On a question with one fixed answer, an answer that moves is a defect, because you cannot tell which run to trust.

Deterministic tooling for AI agents means compiled code that returns the same answer for the same input, on every run. A data agent built on that code has two layers. Compiled checks own every question with one correct answer. The model owns every question where judgment decides.

The two layers of a data agent. Compiled logic runs the checks underneath, and the model supplies judgment on top.

The Two Layers Behind Deterministic Tooling for AI Agents

A working data agent runs two layers, and each layer fails differently.

The lower layer is compiled code. It:

• parses SQL,
• validates a schema,
• compares two result sets,
• scans for personal data, and
• walks a dependency graph.

Each of those five jobs has a correct answer that does not change with the phrasing, the temperature or the model version you deployed.

The upper layer is the model. Its job starts where the exact answers run out. It reads a request like "check whether my refactor broke anything" and works out which tools answer that request. Then it calls those tools in order and explains the result. No fixed rule could cover those steps, because the right tools depend on what the person meant.

A model asked to do the lower layer's job produces answers that are usually right. On a question with one correct answer, usually right still means wrong some of the time.

Every failure in the last column happens silently. Your pipeline already catches the loud failures, such as a query that does not compile, and a loud failure costs one retry. A silent failure looks like success. A model that summarizes a diff it never ran returns a clean report, and the migration ships with the row loss inside it.

Reason 1. Compiled Checks Cost Nothing Per Run

A model check costs tokens on every call, so the bill scales with how often you check. A compiled check costs the same on the thousandth run as on the first, and that cost is close to zero. The cost per run decides how often you can afford to check. How often you check decides how early you catch a mistake.

A lint you pay for per call runs once, at the end, against the final diff. That is the only placement a budget justifies. A free lint runs on every keystroke, on every save and inside the agent loop. It catches the problem while the author still remembers what they meant.

Altimate's Rust core runs fast enough that placement stops being a decision:

• Linting is free per run. A SQL lint costs 0.48 ms per query, so it runs on every keystroke without a budget conversation.
• Validation is nearly free too. A schema check costs 2 ms per model, so a full pass over a 600-model project finishes in 1.2 seconds.

Reason 2. Models Are Right on Average and Wrong in Ways You Cannot Predict

An accuracy number tells you how often a system is right. It does not tell you which cases the system gets wrong. Suppose a model classifies SQL anti-patterns correctly 95% of the time. That model is wrong on one query in twenty. You cannot predict which queries get the wrong answer, and the errors do not spread evenly across easy queries and hard ones.

Run that rate across a 600-model project and one pass misjudges thirty models. Neither the false alarms nor the silent passes announce themselves, so the reviewer has to check the checker.

A reviewer who finds bad SQL behind a green check stops trusting the check. A check nobody trusts gets skipped, and a skipped check catches nothing.

Altimate publishes 19 anti-pattern rules at 100% accuracy across 1,077 benchmark queries, with zero false positives. A rule set can reach 100% on a bounded problem because the rule is the definition of the pattern it looks for. A query either matches the definition or misses it. A model has no definition to match against. It estimates whether the query looks like the pattern, and an estimate is wrong some of the time.

The same unpredictability shows up when a model does the whole job. One agent ran 270 DataAgentBench trials on each of two models, Claude Sonnet 4.6 and DeepSeek v4 pro. The run reported stratified Pass@1, which is the share of tasks solved on the first attempt, balanced across task types.

The two Pass@1 scores sit 3.5 points apart, which reads as a close race. The score hides a second number, the count of trials where the model wrote nothing at all. DeepSeek v4 pro returned nothing on 79 of 270 trials. Claude Sonnet 4.6, the better model on the headline score, still returned nothing on 32 of 270 trials, which is an 11.9% miss. A headline score folds those empty runs into the failures and never reports them on their own.

For a gate, an empty run means the change went through unchecked. A gate built on the model layer alone would leave roughly one dbt model in eight unchecked. On the next run it would leave a different one in eight unchecked, so rerunning the gate does not close the gap.

A second benchmark, ADE-Bench, scores the whole agent against real dbt projects, with the compiled tools and the model together. That combined run reaches 74.4%. The model layer decides that ceiling. The compiled layer's job is to catch the quarter the model missed and report it.

The same system measured at both layers. The left panel shows no variance because the compiled layer has none.

Reason 3. Sub-Millisecond Checks Run While You Type

Latency decides where a check can run. A check taking two seconds cannot run while somebody types, so it moves to save, then commit, then CI. Each move puts distance between the mistake and the feedback.

Compiled validation at half a millisecond runs inside the editor while you write, so the underline appears at the cursor and the fix costs one keystroke. The same check through a model call returns after you move to the next file.

An agent that validates its own work between steps needs each check cheap in wall-clock time as well as in money. A two-second check after each of twenty steps adds forty seconds of waiting to one task, and nobody waits for that.

An execution plan is the same kind of exact answer, and our write-up on profiling a slow dbt query walks through reading one.

Reason 4. Deterministic Checks Produce a Trace You Can Defend

A regulated team has to answer one question after the fact. Why was this change allowed? A rule answers that question with a name and a version. Rule 12 passed and rule 14 failed, on this commit.

A model answers the same question with a different paragraph each time, because nothing pins the wording. A different paragraph is fine in a summary somebody reads once. It is useless as evidence. An auditor compares the record from today with an earlier one, and the two have to match.

Reason 5. Tools Compose Across Teams and Prompts Do Not

A prompt that works is a local artifact. Its behavior depends on the model version it was tuned against. It also depends on the exact phrasing somebody reached after repeated attempts, and often on examples from one team's project. Copy that prompt to another team and it keeps producing output. The output gets worse, and nothing reports the change.

A tool has a signature instead of a phrasing. It accepts typed arguments, it returns a structured result, and it behaves the same for every caller. That signature turns a check into shared infrastructure rather than one engineer's private prompt.

The Model Context Protocol gives tools a standard interface. One lineage lookup written against that interface serves a CLI agent, an editor agent and a chat assistant, with one implementation instead of three. Altimate ships 21 skills in its open-source repo behind that interface.

A standard interface does not make a tool trustworthy on its own. An annotation is a server's own description of what its tool does. The protocol tells a client not to believe an annotation, in its section on tool annotations:

For trust & safety and security, clients MUST consider tool annotations to be untrusted unless they come from trusted servers.

What a client can rely on instead is the tool's behavior. Run the tool against a known test input and compare the result with the answer you expected.

Ask Whether Two Engineers Could Disagree

One question sorts almost every case: could two competent engineers, given the same data, disagree?

If they could not disagree, the question belongs in the compiled layer. The row diff, the email scan and the lineage lookup from the table above all pass that test. Two engineers who disagree on any of those three are reading different data.

If they could reasonably disagree, the question belongs to the model or to a person. dbt model naming, grain and where a transformation should live are all questions of that kind. A rule that picks one side encodes somebody's preference as law. People argue with that rule, then disable it.

Three jobs belong to the model alone. Atlan's published analysis of agent failures reaches the same division of labor between the model and the compiled checks.

• The model reads a vague request and decides what the person actually wants.
• The model weighs a tradeoff where two valid answers exist and the choice depends on context.
• The model explains a technical result to somebody who did not ask a technical question.

A third case sits between the two. Some questions have an exact answer that costs too much to compute on every run. A full row-level diff of a billion-row table is one example. Those questions still belong in the compiled layer, run on a schedule or against a sample.

How to Wire the Layers in the Right Order

The order these layers run in decides the outcome more than the choice of components does. When the model calls the compiled checks before it answers, it sees its own errors and can fix them. When the compiled checks run after the model has produced its output, they can only report the errors.

Our agent modes reference is one implementation of that ordering.

A team that already owns a general-purpose agent is missing the compiled core and the tool interface underneath it. Ten ways AI coding agents fail at data engineering walks through that gap one failure at a time. Nine signs your data platform needs an agent-first overhaul covers the same gap at the platform level.

Start with one change. Pick the check your team runs by hand most often, and move it under the agent.