Listen to the full conversation on DataScienceWithSam, available on Apple Podcasts and Spotify.

Most data teams have lived some version of this same story: an AI agent runs a query, the query returns a result, the number looks reasonable, and the dashboard ships. Weeks later, someone traces a bad decision back to that query and finds a join that never should have executed the way it did. Nobody caught it in the moment because nothing about the failure looked like a failure.

On this episode of DataScienceWithSam, Pradnesh Patil, Co-Founder and CEO of Altimate AI argues that this is not a symptom of weak models. It is the predictable result of asking an agent to act without the infrastructure required to know what is true about the data. The gap is not in what the model can do, it is in what the model has been given to work with.

The core distinction: prompts vs. harnesses

A system prompt tells a model what to do. A harness tells it what is actually true. That distinction, Patil says, is the one most teams get wrong when their first AI data agent rollout stalls out or quietly produces bad numbers. Teams write increasingly elaborate instructions into the prompt, hoping better wording will compensate for the model having no grounded view of the schema, the lineage, or the cost of the action it is about to take. It rarely does.

Patil breaks the agentic data engineering harness into five components: context, governance, MCP tools, shared skills, and agent infrastructure. Each one answers a different question an agent needs answered before it acts responsibly. Context tells the agent what the data actually looks like today. Governance tells it what it is allowed to do. Tools give it the concrete means to act. Shared skills capture the institutional knowledge a team has already built up. Agent infrastructure ties all of it together into something that runs reliably at scale.

Miss any single one of those five components, Patil says, and the resulting failures are silent. The query runs, it returns a result, maybe the result looks correct to any human checking... but it is not, and nothing in the interface tells anyone that until the damage has already spread downstream into a report, a model, or a decision.

Between 27% and 33% of AI-generated queries reference tables that do not exist in the schema, phantom references that a grounded agent would never produce in the first place. A further 78% contain what Patil calls "silent wrong joins": joins that execute cleanly, return a syntactically valid result, and still return the wrong data. Neither statistic describes a model that is bad at SQL. Both describe what happens in the absence of infrastructure that keeps a model grounded in the real, current state of the data it is querying.

Harness beats model size

The instinct when an agent underperforms is to reach for a bigger model. Patil pushes back on that instinct directly, and points to Altimate Code's performance on ADE-Bench as the clearest evidence against it. Altimate Code topped the leaderboard running on Sonnet, while competing systems relied on Opus, a materially larger and more expensive model, to reach lower scores.

The takeaway is not that a smaller model can outperform a larger one under identical conditions, but that the system surrounding the model, the harness that grounds it in context and governs what it is allowed to do, determines outcomes more than raw model capability does. For data teams evaluating where to invest, that reframes the question. The more valuable investment is often not the next model upgrade. It is the infrastructure that makes any model reliable once it is put to work on real data.

Where LLMs should, and shouldn't, make decisions

A recurring theme here is the boundary between deterministic and probabilistic reasoning. Validation, cost checks, and query correctness are deterministic problems, Patil argues: they have a right answer that does not depend on interpretation, and they should never be handed to a model to reason about probabilistically. A query either references a table that exists or it does not. A join either matches the keys correctly or it does not. A cost estimate either falls inside budget or it does not. None of that needs a language model's judgment.

Hard rules should handle hard rules, and the model's judgment should be reserved for genuine ambiguity, the calls that actually require weighing context and making a reasoned choice. Building a harness that respects this boundary, rather than routing everything through the model, is what keeps an agent from confidently producing a wrong answer with the same tone of voice it uses for a right one.

Fixing context compaction for long-running tasks

Standard LLM context compaction was designed for general-purpose chat, where discarding older turns of a conversation carries little cost. Applied to a long-running data engineering task, the same compaction approach tends to discard exactly the schema and lineage context an agent needs to keep working correctly. An agent that loses that context mid-task does not usually stop. It keeps going, with a thinner and thinner grip on what is actually true, until its output drifts from correct to merely plausible.

Altimate's response is a compaction approach built specifically for this kind of task, one that treats schema and lineage as context to be preserved rather than as expendable conversational history. The distinction matters most on the tasks that take the longest: multi-step pipeline builds, large-scale migrations, and anything else where an agent is expected to stay grounded over many turns rather than a handful.

The $5,000 query bill

Governance is not an abstract concern here. Patil cites a real $5,000 Cortex AI query bill as the kind of outcome that permission-based governance helps avoid. The bill was not the result of malicious intent or a rare edge case. It was the predictable outcome of an agent operating without limits on what it was allowed to query and at what cost.

Without controls on scope and spend, Patil argues, agents can go rogue on cloud cost just as easily as they can go rogue on correctness, and the two failure modes compound each other. An ungoverned agent that is also ungrounded in the real schema is not just likely to produce a wrong answer. It is likely to produce a wrong and expensive answer, executed against the wrong tables, at a cost nobody approved in advance.

Where data engineering is headed

Patil's broader claim is that the era of hand-writing SQL is ending. The role is shifting toward building and governing the systems that let agents do that work reliably: defining the context that grounds them, the governance that constrains them, and the shared skills that let institutional knowledge outlive any one person's tenure on a team.

His advice reflects that same emphasis on infrastructure over individual features: Build open source, build cross-platform, so that the tools work across Snowflake, Databricks, and every other environment your data estate actually spans. Avoid siloed AI features that solve a narrow problem for one vendor's users without moving the broader field forward. For Patil, the teams that internalize this shift early, and start investing in the harness rather than the next model swap, are the ones that will find AI agents genuinely trustworthy in production, rather than merely fast and occasionally, silently, wrong.

Listen to the full conversation on DataScienceWithSam, available on Apple Podcasts and Spotify.

A coding agent enters the data warehouse

A data engineer drops a one-liner into Claude Code: "Add patient_visit_count to dim_patient." Twenty seconds later, Claude has read the project, scanned sources.yml, and written plausible-looking SQL. dbt compile is happy. The PR merges Friday evening.

At 2 AM Saturday, the scheduled job fires. The model references PATIENT_ID, but the actual warehouse column is SUBJECT_ID. The run fails in production.

Nothing about the SQL looked obviously wrong. Claude inferred the schema from surrounding code and naming patterns instead of verifying it against the actual warehouse.

Claude Code can write a dbt model. What it can't do, by itself, is run that SQL against your warehouse before you ship it.

This is one instance of a broader pattern. General coding agents like Claude, Cursor, and Copilot are designed for software where the source code is the truth. Data engineering inverts that: the warehouse is the truth, dbt YAML is its description, the agent's SQL is a guess. When the warehouse and the code disagree, only the warehouse is right, and the agent has no way to ask.

This isn't a hypothetical. In our first experiment, bare Claude shipped a dbt model with two hallucinated column names that looked right based on the surrounding code but didn't exist in the actual Snowflake table. dbt compile passed. The model would have failed on first real run.

The altimate-code plugin catches this because Claude calls it before generating, and it actually queries the warehouse for the real columns.

The altimate plugin gives Claude warehouse hands: schema introspection, cross-warehouse execution, dbt-idiomatic defaults. That's what turns a code generator into something that finishes the task.

We ran three categorically different experiments to find out where this matters and where it doesn't. The answer is more interesting than "always" or "never.", and reveals four common blind spots when deploying coding agents on data engineering tasks.

The three experiments

We deliberately picked task shapes that stress different parts of an AI data engineering workflow:

Each task answers a different question about where the plugin actually earns its keep. Taken together, they give you a framework to predict where it'll pay off on your workloads.

Blind spot #1: Schema hallucination on real warehouses

The task: A real Snowflake mart-layer refactor of a patient-360 model with HIPAA constraints, and dozens of column-level decisions.

What happened bare: Claude generated a working-looking model. Two column names were inferred from context, close to right but wrong. dbt compile was happy. Code review by a human would likely have missed them (they looked plausible).

What happened with the plugin: Claude delegated schema introspection to altimate-code's altimate-dbt columns tool, which hit Snowflake directly and returned the actual columns. The hallucinated names were caught before generation.

The cost was a wash. Plugin runs landed at near-parity per-token spend. So the correctness win is essentially free for any real-warehouse work.

If your Claude is touching real warehouse models, the (free) plugin pays for itself by not shipping broken column names.

Blind spot #2: Cross-warehouse work the agent can't reach

The task. Migrate a customer's project from MSSQL to Snowflake. Validate parity. This requires connecting to both warehouses, reading from one, comparing aggregates to the other.

What happened bare. Five attempts in a row, Claude crashed on the connectivity layer. No pyodbc driver. No pymssql. No idea how to find MSSQL credentials. Five runs produced zero working output.

What happened with the plugin. altimate-code's pre-baked warehouse adapters drove the cross-DB diff end-to-end across four internal sub-sessions, with 9+ real pyodbc.connect() calls and a working parity report at the end.

The honest cost: ~6× a bare-Claude run. But bare-Claude's run produced nothing. The right comparison isn't dollars-per-attempt, it's dollars-per-correct-answer, and bare-Claude's denominator is zero.

For cross-warehouse migrations, parity work, or source-of-truth reconciliation, the plugin isn't a nice-to-have. It's the only path that finishes.

Blind spot #3: Default SQL patterns aren't dbt-idiomatic

This was the surprise.

The benchmark. ADE-Bench: 24 dbt tasks running against local DuckDB, objectively scored. Crucially, Claude has everything it needs locally: no cross-warehouse, no missing drivers. This is the "where the plugin shouldn't matter" scenario.

The headline result. Of 5 hard tasks bare Claude failed, the plugin's passive presence (just being installed; no Skill tool invocation, no subprocess delegation) rescued 1. Specifically: an ADE-Bench task called intercom003 went from FAIL → PASS for an extra ~$0.13 in tokens.

Here's what changed in the output:

Bare Claude defended its wrong output as expected behavior. Plugin Claude wrote dbt-idiomatic SQL. The Skill tool never fired. The plugin's value here was purely the loaded skill descriptions sitting in Claude's context, biasing it toward correct patterns.

And the intercom003 task wasn't the only one we ran. We tested the full 5 hard ADE-Bench tasks across 4 configurations (passive, forced consultation, mandatory delegation) to map which mechanisms actually flipped failures:

Two rescues, two completely different mechanisms. The passive presence rescue (intercom003) was nearly free. The mandatory delegation rescue (another ADE-Bench task, analytics_engineering006) cost 18× a bare run but turned 0/7 tests into 7/7. Forcing the Skill tool to fire without demanding actual delegation rescued nothing. Claude reads the guidance, then proceeds to do the work itself.

Even if Claude never delegates to altimate-code, the plugin's presence is a free correctness prior on dbt-shaped code.

The cheapest mode of adoption (just enable the plugin and walk away) measurably improves dbt output quality.

Blind spot #4: Wasted exploration on tasks the agent could solve

The first three blind spots are about things Claude can't do at all without the plugin. This one is different: it's about tasks Claude can already solve and whether the plugin makes those cheaper or more expensive. The answer is surprising: just having the plugin loaded changes how Claude explores, even when no plugin tools ever fire.

The intercom003 task rescue hinted at something deeper than tool augmentation: the plugin was changing Claude's search behavior even when no plugin tools fired. So we asked whether that same passive context bias also reduced cost on tasks Claude already solved correctly.

We replayed three ADE-Bench medium tasks where bare Claude already produced correct, test-passing answers (f1006 semantic data fix, f1010 analysis gotchas, airbnb009 missing-days debug), this time with the plugin enabled. Both configurations passed all tests. We compared turns, tool calls, and spend.

(Note: airbnb009 above is a medium-complexity task, and different from airbnb005-shape, the simpler task class discussed in the next section of this article.)

The aggregate is a 18.6% cost reduction with the plugin enabled, on tasks where both produce identical, correct outputs. Two of three tasks were cheaper with the plugin; one was essentially flat. The biggest win, f1006, saw the plugin use 4 fewer turns and 15 fewer tool calls than bare.

Importantly, this isn't from the plugin's deterministic tools firing. In all three runs, altimate-dbt was never invoked. Claude used the same Read/Bash/Edit tools either way. The savings come from the skill descriptions in Claude's system prompt biasing exploration: plugin Claude reads fewer files, skips more dead-end debugging paths, converges to the answer with less noise.

Loaded plugin context is a prior, not a tax. It guides exploration toward the answer, even when the bundled tools never fire.

This generalizes the intercom003 task result: passive context bias improves both correctness and efficiency on real dbt work.

The decision framework

Here's a single chart you can use to decide, task by task, which mode of the plugin matters and what it'll cost:

Use this to predict where the plugin will pay off on your workloads.

Where the plugin doesn't help

It's not a free lunch everywhere. Two patterns to know about before you adopt:

• On the simplest model-creation tasks (airbnb005-shape), plugin presence costs +25% with no correctness gain. These are tasks bare Claude one-shots in a handful of turns; there's no exploration loop for the context bias to short-circuit. On a richer mix of dbt mediums (Blind spot #4) the aggregate flipped the other way to −18.6%, but the simplest end of the spectrum is a small tax.

• The Skill tool is advisory, not delegating. Even forcing it to "fire" via system-prompt nudges doesn't translate to Claude actually executing altimate-code. We tested this. Don't over-invest in prompt tricks. Just install the plugin and let the passive context bias do the work, then prompt invoke altimate-code for cross-warehouse tasks.

The pattern is consistent: the plugin shines on the work that's actually hard. For trivial tasks, it's a small tax. For warehouse-shaped, multi-DB, or idiom-sensitive work, it's the difference between a working answer and no answer.

Recommendation: Leave the plugin on by default. The +25% overhead on trivial tasks is small in absolute terms (these are cheap, short runs), and dynamically toggling introduces friction that costs more in engineering time than it saves in tokens. The aggregate across a realistic mixed workload still favors leaving it enabled. If you're running a batch of pure staging model scaffolding with no warehouse lookups, that's the one case worth disabling it — but it's the exception, not the workflow to optimize for.

How to adopt: — the 5-minute path

Step 1. Add to ~/.claude/settings.json:

{
  "enabledPlugins": {
    "data-engineering-skills": true
  }
}

Step 2. Verify it loaded:

claude --print "list available skills" | grep -i altimate

Step 3. For cross-warehouse and parity tasks, demand altimate-code explicitly in the prompt:

Use altimate-code to drive the cross-DB diff

Step 4. Pre-bake your warehouse credentials in altimate-code's connection registry once. Saves re-doing it per task and avoids credential leakage into prompts.

The bottom line

The altimate plugin doesn't make Claude smarter at SQL. It makes Claude competent at the parts of data engineering that aren't SQL: schema introspection, cross-warehouse connectivity, dbt-idiomatic defaults. And on the dbt work it already could do, it does it with fewer tokens.

For data teams running Claude on real warehouse work, that's the difference between a code generator and an agent that finishes the task.

Bare Claude is cheap when it works. When it doesn't (when the warehouse can't be reached, or when the column names are wrong), the cost is undefined. And even when it does work, plugin Claude usually gets there cheaper.

Try it. Altimate Data Engineering Skills Plugin for Claude Code

If you run it on your next migration or parity task, we'd love to hear what shape of work it helped, or didn't.

All three experiments used Claude Sonnet 4.6 uniformly. Sample size is small per task (1-2 runs per configuration); the task shapes are deliberately divergent, so treat the numbers as directional rather than precise.

With ever-changing models, new and better ones coming out every few months, it's great if we don't have to rely on them too heavily. The better your tooling, the less dependent you become on any single model. That's also why the deterministic harness matters: a correctness layer that lets you reproduce outputs and trace lineage regardless of which model you're running underneath. This is especially true during maintenance or extending the project, where verification is the real job.

The danger isn't only a crash or an error message, but a wrong number that didn't break. It might be a clean query, but it introduces duplicated rows.

In this article, we go through the three levels of AI agents in data engineering, how to structure projects so the AI delivers its best outcomes, and how dedicated agents with a deterministic core help us build higher-quality pipelines — ones we can actually trust. And we look at a practical example of how it works with a blast radius analysis.

The Three Levels of AI Agents in Data Engineering

Why should we use agents for data engineering? And at what levels can agents help us productively? As LLMs will always have some error tolerance, as humans do too, we need a way to be more confident in producing the code.

Chat-phase, Autonomous and Dedicated Tooling

There are different levels of confidence and levels on which the agents can help us.

1. The initial chat-phase: the development where we prompt Claude or ChatGPT. The model tries to understand the context based on what it has access to. It takes a decent amount of tokens, as it needs to scan everything from scratch.

2. The autonomous approach, where Claude Code or Codex also have access to the tools humans have, mostly the CLI on the terminal, making it possible to query Postgres with psql or read from S3 or Parquet with DuckDB to verify queries and data. A much higher quality outcome.

3. Dedicated agents for the task at hand. E.g., for data, the tools know dbt or know how to transpile SQL code deterministically, meaning not from training data only, but with an actual tool that does it much faster and more reliably. Built-in checks and features a "general" agent can't provide.

Showcasing the three levels of AI agents in data engineering

Ideally, we'd want to always use the dedicated tools, but there isn't always one.

Where in the DE Lifecycle Each Level Actually Helps

BI Dashboards vs. Plumbing the Data Pipelines, or Creating Source Ingestions, or Maintaining? For data engineering, the question is not only if there is dedicated agent tooling, but also on what part of the data engineering lifecycle AI agents can help data engineers and analysts the most, and potentially even domain experts?

The lifecycle contains the ingestion part, ETL, or understanding the business in great detail, or is it just to visualize the result? Or should it cover maintenance in case of overnight ETL errors, or the full data lifecycle?

In general, before we go into more details later, agents can help us on the full cycle, but it always depends on who you are and what role you play. Building from scratch with no knowledge or seniority is dangerous. Why? Because they can't verify if the produced code is correct. Okay for a side project or a proof of concept, but not for actual production.

What's the Engineering Discipline for Working with AI?

There's also a part that is less technical, a way of guiding the agents in the right direction. Especially if we want to safely use it in large projects or organizations, we can't just let it run without guidance.

For that we need:

1. clear project structure in which the agents can flourish. The more is given, the fewer tokens are used for this work, and it will be more aligned across the project. (Another reason a deterministic workflow such as uv init is best, because it will always be the same).

2. build with clear instructions (agentic skills, superpowers, etc.) on how the tools are used (basically providing CLIs and API documentation). This is the bulk of the work anyway. That's the data architecture, the brainstorming with fellow humans before you build something, instead of missing a key insight in the beginning and then letting the agent run down the wrong path. Also, be realistic: prompting "be correct" or "use state-of-the-art" won't make it more correct or more state-of-the-art than the model was trained on. So if it's a rather new architecture, it's a must that you provide these links and hints.

3. set up the project in a modular fashion, so the agents cannot break the whole project if they make a small change, so you don't end up in a scenario like dependency hell with everything dependent on each other.

4. use a declarative approach, with descriptive configuration that says the what and not the how, so that you can collaborate on these configs with the agents, version them, and easily revert or change something, as well as decouple the implementation logic from the actual business logic.

With these steps, you can get the best out of the agents of today. I'd say the model matters less, but the structure does, and as Mario says, so does the workflow approach. For example, extensively plan (the process before writing a single line) and correct the model before any implementation that could lead down the wrong path is written.

Also, don't overthink it. But this is only the workflow and learning the soft skills and discipline of working with agents. How does that look in a real-world project?

!Note:

The key is to get use out of AI, not to get more work. E.g., most developers used to think about the problem. Today, most drown in PRs. When the AI tooling gets better, AI can provide more quality code that is correct, that needs less review or fewer iterations, which means fewer PRs and less work for the developers to go through.

The Correctness Layer for Data Engineers

A key insight is that AI agents should support the "human in the loop" for correctness, or a correctness layer. And rather than making more work to verify more code, we should be confident in the process and know that the code it produces is verified and ultimately correct.

But how do we get more "correct" work and a layer in which we can verify it? The biggest argument is a deterministic-validation architecture in full. E.g., Altimate Code splits the agent into a probabilistic layer on top and a deterministic Rust/TS layer underneath that does the actual SQL ops such as parsing, validating, and equivalence checks, so that the agent itself never has to be trusted on those questions.

An example of how Altimate Code is built with its probabilistic agent, deterministic harness, and deterministic core | Image from the article The Correctness Layer: Why Data Agents Need Determinism

Altimate Code, for example, is built on a probabilistic agent, deterministic harness, and deterministic core. The probabilistic agent with the LLM does the creative work of reading intent, picking a strategy, drafting SQL, summarizing results, and recovering when something goes wrong.

Below the boundary sits the deterministic harness, a TypeScript layer that intercepts every tool call: a dispatcher checks hasNativeHandler before the call runs, and routes it either to a native, deterministic handler or back to the model. Those handlers don't reimplement logic themselves, they call into the deterministic core, a Rust engine (altimate-core) that exposes SQL operations as pure functions over ASTs and schemas, wired in via napi-rs bindings. Parsing, validating, transpiling, checking query equivalence, diffing schemas, extracting column lineage, diffing rows across warehouses — all of it runs sub-millisecond, and all of it returns the same answer on the same input, every time.

Like a compiler, the agent never decides whether two queries are equivalent or a column exists upstream. Instead, it calls a function that proves it against the parsed AST and the schema, the same way a type-checker proves a program compiles rather than guessing.

How the correctness layer adds additional verification

That's the distinction that makes the output easier to review, as factual checks have been run and the output is either correct, or there's a bug that it can fix directly. The rest a human can re-verify. On the dilemma of having stopped to hand-write code and approving it faster than humanly possible to check, you can also read more at You Are the Trust Layer.

!Note: There's another factor: being wrong

Bare agent use might be cheap, but only until they're wrong, and then the cost is unbounded.

Improvements for Better Usage of Tokens

Altimate, or data engineering agents that have deterministic functions and integrated understanding of how to work, can help you save tokens and be token lean (the opposite of tokenmaxxing, which is popular on Twitter/X, using as many tokens as possible and having an agent running at all times). Because in large enterprises, token costs are a real budget point.

To slow down the tokens, an easy trick is to instruct the model to use fewer tokens and words itself - caveman is a good example of that, but you can also add a singular prompt to your CLAUDE.md, Codex, or model of choice in combination with Altimate Code.

An example of Altimate Code showing a trace of data lineage and a web UI for it.

There's a second, less obvious cost: the token itself isn't a stable unit. When Anthropic shipped Opus 4.7, the same prompt that cost X tokens on 4.6 started costing roughly 1.4X (same input, same answer, more tokens, same price per token).

In The Great Token Heist of '26 , the Altimate team makes the case that "cost-per-token is the wrong number to optimize", since the meter itself can move with a vendor's next model update, and what we should track instead is cost-per-task. I fully agree, and this is where deterministic function calls work around that volatility by not using a model/tokens for every task, making it less expensive.

Typical Use Cases

In this chapter we go through typical AI agent use cases for data engineering.

There are many of them. You can use them to educate yourself or your team, build production data pipelines, build data apps, and visualize your data in new innovative ways (usually HTML web pages with React and other JavaScript frameworks). But in general, the use cases fit into these approaches:

1. Start a new project from scratch example: Building a data landscape with more open source.

2. Extending an existing project or data warehouse: Adding new data pipelines.

3. Maintaining current setup: Update and verify it still works when changes come in.

4. Migration: Migrate from one database or tooling to the next.

5. Finding the Blind Spots: Two similar-sounding IDs might be wrongly used for a join, or missing data in a column that got missed in a nightly load, or anything in between. If agents can do these checks, that would be super beneficial. With more access to CLI, Model Context Layer, and deterministic tooling, these things are truly possible.

Below we go through extending and changing an existing warehouse with a change of column, and using Altimate Code to give us a Blast-radius assessment.

Showcases: Blast-Radius Example

A Blast-radius refers to the potential extent of damage. For example, before you knock down a wall in your house, you want to know if there's plumbing behind it, electrical wiring within it, or if it's holding up the floor above.

The same is true for a data warehouse or a data project with lots of ETL. For example, if a data engineer cleans up the table fct_orders by joining orders to order_items and summing order_total. It compiles, the dbt tests pass, nothing errors. But the join changes the grain, so any order with several line items now gets counted once per item, and revenue quietly inflates.

It's best to know, before you rename a column or add a new join, the downstream (data that comes after the current task) dependencies to the dashboard — that's what the blast-radius report does.

With Altimate Code we can achieve this. Before any change goes through, it maps out the full impact automatically and produces a detailed blast-radius report with what will break, what's safe, what needs someone to sign off, and also performs the changes. Here is what this looks like:

Rename and Change Columns and Logic

As an example, in this prepared ecommerce repo with different DWH layers such as staging -> intermediate -> marts, I prompted this request to change unit from cent to dollars:

It recognized the dbt name and invoked dbt-analyze automatically:

It gave me a full Blast-radius report and the impact my changes would have on the project:

Including semantics only, to point out what's safe and what's not:

With a fixed order to address breaking changes, semantics and docs, and intentionally untouched:

Notice, I hadn't said anything about blast analysis or using dbt-analyze. It did it on its own, ran dbt, and analyzed it deterministically.

This shows how Altimate Code looks behind the walls of data engineering, just like blast radius analysis.

If you want to see another example and a full blog post on Blast Radius, check out Blast Radius Analysis Using Altimate Code, and what Altimate Code did as in the video. Or Altimate provides many more examples and Showcase on their website such as Migrate SQL Server to Snowflake with dbt or showing how to resolve An Upstream Schema Changed.

!Note: Connect a model to Altimate

Make sure to connect to a model with /connect and choose an existing subscription with API credits, or any other subscription. I used opencode zen for my example, which includes e.g. Opus 4.8.

Correctness Over Confidence

I hope you got a better understanding of why AI agents can be genuinely useful, especially when provided with the right tools and applied with the right discipline.

You've also seen how deterministic tooling, purpose-built for data engineering and analytics problems, gets you both better correctness and better token economics than general-purpose agents alone.

Coming back to where we started: not every task needs a level-three agent. A quick chat-phase agent is fine for exploring a dataset or drafting a query you'll review yourself. But the moment that output touches production or serious work, a dashboard, a nightly job, a number someone makes a decision on, you want the deterministic core underneath it, not just a model that sounds confident.

That's the gap Altimate Code is built to close. It runs on deterministic functions purpose-built for DE workloads, it's open-source via the OpenCode TUI, and for teams wanting more, there's Altimate Studio — a paid, multi-agent platform with extras like warehouse cost optimization, dbt development acceleration, and migration tooling.

Check out Altimate Code, it's free and open-source. Give them a star if you like them, and find more information on their docs and new website.

Sometime this year your job changed, and nobody sent the memo. You used to write code. Now you mostly approve it: you read a summary, it looks right, you hit accept, and you've moved on before you could have checked it even if you wanted to. The machine produces faster than any person can verify, and the distance between what it ships and what you actually confirmed grows every week.

So what does that make you? Not a tool's user. You are managing a labor force whose work you never read, and you are the only check between its confident output and what reaches production. You are the trust layer.

The defining problem of this era isn't that AI can't do the work. It's that it does the work far faster than anyone can check it.

I know because I measured it. I wrote a small proxy, pointed it at my own machine, and logged every request that crossed the wire for forty-eight hours. One developer, two days: 11,302 requests across 109 concurrent sessions and sub-agents. The last change one of those agents handed me had taken it six minutes and something close to an afternoon of human work. I looked at it for eleven seconds, then hit accept. I read what surfaced and trusted the rest. So do you.

The first problem with a trust layer made of one person is that it's uneven. How much gets caught depends entirely on who is holding the line. A senior catches a surprising amount by reflex: the function that's subtly wrong, the assumption that doesn't hold. A junior catches less, because spotting the bad answer is the same skill as writing the good one, and that skill is the thing they're still building. And the ten sub-agents your agent spawned that nobody opened? Those are checked by no one at all. Across a real team, the work is thin on checking, and thinnest exactly where no one is looking.

The second problem is worse, because it doesn't care how good you are. Generation is parallel and nearly free: one prompt fans out into ten agents. Your attention is serial and fixed: you read one thing at a time, and there are only so many hours. So as the volume climbs, the fraction any human can actually check falls, and not gently. It heads for zero. The senior's reflexes don't beat this. They just start from a higher number on the way down.

You might assume the part you don't check is mostly harmless, or you'd have noticed by now. I assumed that too, until I scored it. Almost everything the agents produced looked like a correct answer: clear, confident, well-formed. Barely a third of it actually was one, once I checked it against the code and the task in front of it. Looking right and being right turn out to be very different numbers, and the gap between them is exactly the part a quick skim waves through.

And the work you don't check doesn't disappear. It ships, things get built on top of it, and (this is what turns a backlog into debt) the next agent reads it as context and repeats it as fact. A wrong assumption from Tuesday is settled truth by Friday, cited in code none of you wrote by hand. You took the debt on to move fast. It accrues interest quietly, and it comes due later: the incident, the rollback, the week spent tracing where the data went wrong.

The person clicking accept is almost never the person who pays.

That's why you, personally, feel fine. The cost is displaced. It lands in time, weeks after the commit that caused it, and it lands on other people: the junior who trusts the senior's pace without the senior's reflexes, the teammate who inherits the module, next quarter's version of you. It's also why the best engineer on the team is the most sure there's no problem. He's right about his own desk. He just isn't the one holding the bag.

One developer can absorb this by hand for a while, which is the whole reason it stays invisible. An organization can't. Multiply it across everyone shipping this way and there is no trust layer left, only thousands of accepts a day, a rising balance of unverified work, and no single place that can tell you which of it was real.

Today the trust layer is you: by reflex, in the seconds between accepting one diff and starting the next. That was never going to hold. The only open question left is whether you replace it on purpose, with verification that runs as fast as the work is produced, or keep paying the debt until it picks the moment to collect.

Numbers from a local proxy run against one machine over a 48-hour window: 11,302 requests, 109 concurrent sessions and sub-agents, and a 47-trace correctness sample in which 96.9% of output looked right and 37.5% was actually right. Measured, not modeled.

If you're running an AI agent on data (pipeline diagnostics, schema exploration, ad-hoc SQL, lineage queries), you've looked at the model bill and asked the same question we did:

"The cheaper model on the leaderboard is only a few points behind the one I'm using. The endpoints are drop-in compatible. The agent code doesn't care which backbone is wired in. Why shouldn't I switch?"

That's a defensible question. It's also a question with a more interesting answer than the leaderboard suggests.

We're going to walk through what we found when we ran the swap: Claude Sonnet 4.6 to DeepSeek v4 pro inside the same data agent, run across the same benchmark, with the same workspace, prompts, and tools. The accuracy difference is small and roughly what the leaderboard predicts. The behavioral difference is not. By the end of this post we'll have a concrete framework for thinking about LLM selection in an agent context that's a lot more useful than "which model is on top of which leaderboard this week."

Throughout, the only thing we change is the backbone model.

*Note: We did not compare against Claude Opus series since Opus would intensify the cost gap (roughly 5× Sonnet's per-trial cost) without changing the model family. We plan more inter- and intra-model family comparisons soon.

Same task — Beyoncé's "Get Me Bodied" on DAB’s music_brainz dataset. The dots are tool calls. The gray bars are narration chunks. You can tell the models apart without looking at the model field.

The setup, in two paragraphs

We use altimate-code, the open-source agent runtime behind our data engineering platform. It's the harness currently sitting at #1 on the DataAgentBench (DAB) leaderboard for Pass@1 stratified accuracy, scored at 0.6320 with GPT 5.5 and Claude Sonnet 4.6. DAB is UC Berkeley's data-agent benchmark: 54 queries across 12 real-world datasets (Postgres + MongoDB + SQLite + DuckDB), 5 trials per query, 270 trial runs per submission. Real schemas, real ambiguity, validators that check the exact shape of the answer.

For this post we ran two complete Pass@5 passes through altimate-code: one with Claude Sonnet 4.6, one with DeepSeek v4 pro. 270 trials each. 540 trials total. Everything else (agent code, prompts, tools, validators, dataset hints, workspace setup) stayed identical. The result is a clean A/B with a single variable. That's what makes the contrast interesting.

The numbers that the leaderboard would show you

Here's how the two runs compare on the dimensions a leaderboard would surface:

                          Sonnet 4.6      DeepSeek v4 proᵃ
Stratified Pass@1            60.4%             56.9%
Raw Pass@1                   63.0%             60.0%
Cost per trial               $0.76             $0.29
Median duration / trial      4 min             9 minᵇ

ᵃThis run has not been sumbitted to DAB, we're working on an improved approach.
ᵇDeepSeek latency measured via OpenRouter without upstream-provider pinning; 
OpenRouter's routing across DeepSeek's underlying providers contributes to wall-clock variance.

• Accuracy gap: 3.5 stratified points (the benchmark metric).

• Cost gap: 2.7× cheaper for DeepSeek.

• Latency gap: 2.2× faster per trial for Sonnet.

If you're running an overnight batch workload where wall-clock doesn't matter, those numbers point at DeepSeek. If you're running an interactive agent where users are waiting, they point at Sonnet. That's the surface-level read, and it's not wrong.

It's also incomplete in ways that turn out to matter.

The trace tells a different story

We picked the model swap as the only variable so we could look at everything else.

Inside an agent run, the score is one bit of information: the final ANSWER, did it pass or fail. The trace (every tool call, every text emission, every step boundary, with timestamps) is thousands of bits. The agent loop produces a events.jsonl file per trial, and once you start reading them side-by-side, the score gap stops being the interesting number.

Here’s a snippet:

The trace signature is what an operator stares at when something goes wrong in production at 3 AM. It's what you log, what you alert on, what you debug from. We found that the signature changes more when you swap the model than the accuracy does. By a lot.

Here's what that looks like, concretely.

Finding 1: One model talks. The other doesn't.

Per trial:

                              Sonnet 4.6                DeepSeek v4 pro
Chat-text events (mean)             13                       10
Characters of narration (mean)      5.7 k                    1.7 k
Tools called — mean                 36                       36
Tool calls — median                 32                       36
Tool calls — min                    0                        0
Tool calls — max                    82                       75

Both models handled the same workload with the same tool volume. Sonnet writes about 3× more chat text along the way. It narrates as it works ("Let me check the schema for the orders table" / "I'll filter for active customers first" / "Trying a different join condition"). DeepSeek does its planning silently and only surfaces what it has to.

This shows up in the cost structure too. Sonnet writes 49 k tokens into Anthropic's prompt cache per trial and reads 1.22 M back from it; reasoning tokens are zero. DeepSeek writes nothing to a prompt cache, but burns 8 k reasoning tokens per trial doing chain-of-thought inline.

The downstream consequences are concrete:

• If you're tailing logs from an agent in production, Sonnet sounds like a thinking colleague. DeepSeek sounds like a quiet one.

• If you're building token-cost alerts, they'll be calibrated for one and miss the other. Sonnet's bill is input tokens + prompt cache. DeepSeek's is reasoning tokens, with no cache line at all. So a Sonnet-tuned alert ("input spend spike", "cache hit-rate drop") sits silent on DeepSeek while reasoning-token cost you're not watching adds up.

• If your platform's observability is built on chat-text logging, swapping to DeepSeek will quietly cut your visibility by two-thirds.

None of this affects the accuracy column on the leaderboard. All of it affects your team building on top of these models.

Finding 2: One model commits early. The other doesn't always commit.

altimate-code reads the agent's final answer from a file called ANSWER. Every write tool call against that file is a timestamped event. We plotted the first ANSWER write per trial as a fraction of trial elapsed:

Sonnet's median first commit lands around 70% of the way through a trial. DeepSeek's lands around 90%. And the "never" row is the operationally important one: 79 of 270 DeepSeek trials never wrote anything to ANSWER at all. For Sonnet, that number is 32.

These are two strategies, both legitimate. Sonnet commits early and refines: writes a first draft mid-trial, then rewrites it as confidence grows (mean: 1.47 ANSWER writes per trial). DeepSeek explores until forced to commit, then commits once (mean: 0.88 writes per trial).

Why DeepSeek explores-then-commits is partly architectural. Its reasoning models emit reasoning_content and content on two parallel channels (DeepSeek API docs). The agent loop consumes content; reasoning_content is inspection-only and must be stripped before resubmission (the API 400s otherwise). DeepSeek plans privately in the reasoning channel and bursts the plan into content as tool calls + ANSWER in one transition. Sonnet has no such split: its chain-of-thought shares the channel the loop consumes, so drafts and retractions stream out as visible turns.

If you're building anything where users see partial progress (a dashboard, a streaming UI, an agent that hands off intermediate results), Sonnet gives you a draft to show by minute two. DeepSeek leaves you with an empty file for eight minutes, and almost a third of the time, leaves it empty forever.

Finding 3: One model asks the database questions. The other writes scripts.

Look at the tool-call distribution across all 270 trials per model:

The agent and tool surface were identical; the tool preferences were not.

Sonnet falls back to bash 45% more often than DeepSeek. It runs Python scripts for filtering, aggregating, formatting. When in doubt, it writes code.

DeepSeek prefers the native data tools the harness exposes. It inspects schemas 2.7× more often. It runs SQL through sql_execute 64% more often. When in doubt, it asks the database.

The discovery cadence diverges before the first SQL even runs. Mean schema-related calls before the first SQL execution: 2.8 for Sonnet, 4.9 for DeepSeek. DeepSeek does a column-by-column tour before writing any SQL. Sonnet reads the prompt, takes one schema snapshot, and gets to work.

Both rhythms produce correct answers on plenty of queries. They get there with different tool budgets and different turn structures.

If your platform logs the shape of tool calls in production, you can identify which backbone is running without looking at the model field.

Finding 4: When they fail, they fail differently

This is the finding we found most operationally consequential.

Every failed trial fits one of four shapes:

                            Sonnet 4.6      DeepSeek v4 pro
Empty (nothing written)        7 (3%)         27 (10%)
Short wrong (single value)    43 (16%)        26 (10%)
Medium wrong (small table)    30 (11%)        33 (12%)
Long wrong (multi-row)        20 (7%)         22 (8%)
Total failures               100             108

Sonnet's failures cluster on short wrong: confident commits to a wrong value. DeepSeek's cluster on empty: the agent worked, explored, iterated, and never wrote anything.

The total failure count is the same; the shapes are opposite. Both shapes have architectural roots.

DeepSeek's reasoning API can return HTTP 200 with reasoning_content populated but content empty, meaning the model reasons through to a conclusion without ever generating output text. The failure mode is tracked in DeepSeek-R1 issue #314: "the status code returned is 200 (indicating a successful request in the HTTP protocol sense), the actual content of the response is empty." The issue was closed stale without a fix. Sonnet has the opposite problem for the opposite reason: with no separate reasoning channel, its planning happens in the same content stream the loop consumes. Drafts surface as soon as the model has a candidate, and once a candidate is in ANSWER there's no separate "verify-before-commit" gate to slow it down. The model satisfices: it picks the first plausible answer, polishes the prose around it, and moves on. That's how “short wrong” gets produced: a confident commit on partial evidence, with the verification step folded into the same channel that wrote the answer in the first place.

Here's a concrete example. DAB’s “GitHub Repos” query 3 asks for the count of Shell-language commits in Apache-2.0 repos under a length filter. The ground truth is 1077.

Sonnet:   "963
114"   ← committed close-but-wrong on turn 8
DeepSeek: "1077"       ← landed correctly after exhaustive schema inspection

DeepSeek did 40 tool calls, including 4 schema_inspect calls and 12 sql_execute iterations. It got there because it kept asking. Sonnet did 21 tools, jumped to bash at turn 8 with a count that was nearly right, and never noticed it was wrong. DeepSeek won this query 4 out of 5 trials. Sonnet won 0.

For an operator, the two failure modes have different downstream costs:

• Vocal-wrong (Sonnet's signature) gives you a bad answer that may pass type checks and propagate to consumers. The downstream system sees a number and assumes it's right.

• Silent-stuck (DeepSeek's signature) gives you nothing: a missing row that is easy to detect with a presence check but hard to debug without the trace.

Which is worse depends on what your downstream consumers do. If you have validators downstream that can re-check correctness, silent-stuck is more recoverable. If you don't, vocal-wrong is the silent killer.

When the signature disappears

There are times, however, when models do not fail differently.

When a question is clean (the schema is obvious, the format is pinned down, the dataset hint spells out the join recipe), Sonnet and DeepSeek behave almost identically. We watched this on a question from DAB's bookreview corpus (query 3) that asks for a multi-row table from columns that store Python-dict syntax as text. Both models passed it. Both took exactly 27 tool calls. The first eight tools in each trace were nearly identical: read question, read format_hint, list warehouses, index schema, inspect a table, write the SQL.

The same effect shows up when the agent has good orientation upstream of the main loop. We run a pre-orientation pass where a lighter model walks the schema, samples rows, identifies non-obvious column formats, and proves out join candidates with mechanical probes. The output gets dropped into the agent's workspace as a single markdown file. When that file is well-formed, you can't tell the models apart from the trace.

The trace signature shows up at the edges of the agent's capability, where it has to decide for itself when to commit, how aggressively to explore, what to do when stuck.

So before you assume model selection matters for your workload, look at your queries. If most of them are clean (well-documented schemas, unambiguous prompts, validators that accept reasonable variants), the model choice may genuinely not move your needle. Pick on cost. If your real queries push the agent into open-ended decisions (which is what most production data-engineering work actually does), the trace signature is what you should be evaluating.

How we choose now

A short field guide, based on what we measured:

Pick Sonnet 4.6 when:

• You need progress visibility mid-run. For dashboards, streaming UIs, and partial-trace debugging, you need incremental output. Sonnet writes a draft answer at the 70% mark and improves it. DeepSeek doesn't write anything for 90% of the trial, and a third of the time, never writes anything.

• Latency matters per trial. Sonnet finishes in ~4 minutes median; DeepSeek in ~9. For interactive use, the gap is felt.

• Silent failures are expensive downstream. Vocal-wrong is easier to detect than silent-stuck. If your pipelines don't double-check the agent's output, you want it to commit (so you can catch wrong answers) rather than disappear (so you don't even know it failed).

Pick DeepSeek v4 pro when:

• Cost matters more than wall-clock. DeepSeek runs at 38% of Sonnet's per-trial cost for similar accuracy on this benchmark, making the math straightforward for overnight batch, retroactive backfills, and async report generation.

• The agent needs to explore. DeepSeek's tendency to inspect the schema thoroughly before writing SQL is the right behavior when the schema is unfamiliar and the question demands precision.

• You have downstream validators that catch silent-stuck. If your platform notices missing outputs and can retry, DeepSeek's caution costs you less than Sonnet's overconfidence might.

Run both when:

• You're doing consensus or majority voting. The two backbones disagree on 19% of trials in our run. That's a real ensemble margin, not noise. A judge model can pick the right answer often enough to lift the combined score above either alone.

Case in point: In this comparison, Sonnet and DeepSeek disagreed on 51 trials. A perfect oracle judge over those would lift ensemble Pass@1 by ~9-10pp; a realistic LLM judge by ~3-5pp. Full consensus math and a real judge run is a follow-up.

We use all of the above in production. altimate-code is built to make this choice cheap: any backbone, swappable per workload, same trace format regardless. The selection logic is yours. The decision is yours.

A footnote on substrates

This post is the operational complement to an argument we made earlier in The Correctness Layer in ADE. That one argues that for the parts of data engineering that have deterministic ground truth (two queries are equivalent, a lineage edge exists, a row-level diff is correct), the LLM is the wrong substrate. The substrate should be deterministic code.

The argument here is the complement: even when you do leave a task to the LLM, which LLM you pick changes the operator experience. The substrate is still the story. The LLM is just the part of the substrate that varies the most when you swap providers.

If you want to look yourself

Everything in this post is reproducible. altimate-code is open source. DataAgentBench is open. You can obtain the per-trial events.jsonl files which preserve every tool call, every step boundary, every token count, every chat emission. The two runs we compared above are from our own benchmark_runs collection; the analysis script that produced the numbers is in the repo. If your team is making a similar swap, the same kind of diff is one fork and a few hours of compute away.

Conclusion

The leaderboard tells you what a model can do on a benchmark. The trace tells you what your agent will look like in production. We built altimate-code to win on both.

That's why we sit at #1 on DataAgentBench with Sonnet, and why the same runtime runs DeepSeek at 38% of the cost just as cleanly. The tools, prompts, and trace format are identical across both. The model becomes a variable per workload, not a quarterly commitment.

The runtime carries its share too. Tools emit a compact, structured payload to the model and a richer, human-readable one to the UI, so the agent's context window isn't burned on formatting it'll never read. Long sessions get compacted instead of truncated. Token budgets stretch further on every backbone you point at it.

Pick the model that fits the workload. Pick the runtime that doesn't lock you to one.

Your LLM provider charges by the token, but a token is not a portable unit of work, a unit of meaning, or any unit you can compare across vendors. It only has meaning inside one vendor's system and loses it the moment you step outside. The token is a currency: vendors mint it, vendors re-denominate it, and the exchange rate with English (meaning how many tokens it takes to express the same sentence) is theirs to set. So when two vendors quote the same dollars-per-token, they are still quoting in different currencies.

On April 17, 2026, the day after Anthropic shipped Claude Opus 4.7, a Hacker News user named anabranch posted a public leaderboard they had built at tokens.billchambers.me with crowdsourced before-and-after token counts on the same prompts run through Opus 4.6 and Opus 4.7. The thread title, "Opus 4.7 to 4.6 Inflation is ~45%", climbed to 117 points before lunch. anabranch's summary: "It can be more than 2x for small prompts."

The comments filled with developers reporting specific impacts on their bills. hgoel, on the Claude Max 5x plan, hit their five-hour usage limit in under two. tiffanyh exceeded the weekly limit after seven prompts on a single-file HTML/CSS/JS project under 300 lines. KellyCriterion: "killed my weekly limit with just three prompts." manmal put the number on it: "The same prompt costs roughly 30% more now, for input."

None of this was a price change. Opus 4.7 launched at the same $5 per million input tokens and $25 per million output as 4.6.

Independent measurements followed within days. Simon Willison upgraded his Claude Token Counter to compare the two models on the same input and posted: "Opus 4.7 does appear to use 1.46x times the tokens for text", with up to 3× for images, priced the same as Opus 4.6 on a per-token basis. OpenRouter ran the numbers across more than a million real customer requests and published a bucketed breakdown: 32–45% more native tokens for equivalent text (highest on prompts under 2,000 tokens), with a 12–27% increase in net cost after prompt caching absorbed some of the inflation.

What changed was not the price per token. It was the token itself. The same prompt that cost you X tokens on 4.6 now costs roughly 1.4X on 4.7: same input, same answer, same work. The vendor changed the resolution. If that sounds like a camera going from 12 to 18 megapixels, Anthropic would accept that comparison: more resolution, better model, same price per unit. But a megapixel upgrade is a one-time capability you choose. This tokenizer change bills you per pixel on every shot, cannot be switched off, and shipped with no announcement. And more megapixels never guaranteed a better photo, only a bigger file. More tokens do not guarantee a better answer. They guarantee a bigger bill.

The currency got re-denominated

Buried in Anthropic's migration guide is one easy-to-miss sentence: Opus 4.7 uses a new tokenizer that "may use roughly 1x to 1.35x as many tokens" as previous models, varying by content. The price per token did not move; the number of tokens required to express the same English sentence did.

The honest defense is that this is an upgrade: a finer tokenizer can mean better resolution and better answers, and you simply pay in more tokens for them. But "better answers" is a measurable claim. If the quality is real, it shows up as a higher success rate and the cost per finished task still holds. If it does not, you are paying more tokens for the same result. The catch is that they shipped the cost with no announcement and no opt-out, while leaving the "better outcomes" half for you to verify on your own bill.

If this happened in a fiat currency, we would call it a redenomination: the kind of operation central banks use when they need to recalibrate without admitting they are recalibrating. Your salary stays "the same" in the new currency, but the new currency buys 30% less coffee. Anthropic, of course, does not have the obligations of a central bank, so there is no announcement, no press conference, and no notice period. The tokenizer is part of the model. It changes when the model changes.

The Hacker News reactions were not just complaints about bills. They were the predictable response to opacity. Shailendra_S, a bootstrapped founder, argued that 45% inflation breaks the unit economics for most indie products and pushes builders toward open models, or back to the more token-efficient Opus 4.5. dakiol announced his team had dropped Claude entirely. Boris Cherny from the Claude Code team showed up in the thread, but to address a separate complaint about over-fixated safety warnings, not the cost question. The cost question is hard to address. The currency really did get smaller, and the vendor can keep calling it "the same price."

"Nobody raised prices. The currency got smaller."

There is no standard token

Anthropic's redenomination is not unusual. It is the rule, dressed up as an exception.

Tokens are made by tokenizers, which are deterministic functions that split text into integer IDs. Each vendor trains its own. OpenAI's tiktoken is open-source: you can download the vocabulary files and count locally before you pay. GPT-3.5 and GPT-4 use cl100k_base, with 100,256 entries. GPT-4o, GPT-5, and the o-series use o200k_base, with 200,019. The transition from one to the other in 2024 was not priced as a change. It was, of course, a change.

The pattern repeats across vendors. Llama 2 to Llama 3 jumped from a 32K to a 128K vocabulary, and Meta says that buys up to 15% fewer tokens for the same text. Mistral's Tekken tokenizer, introduced for NeMo and Pixtral, compresses code about 30% more efficiently than its SentencePiece predecessor, and Korean and Arabic by 2–3×. Gemini's tokenizer, currently a 262,144-token SentencePiece model shared with Gemma 3, has rotated through versions across Gemini 1.5, 2.x, and 3.x. DeepSeek's V3 paper documents a 100K → 128K vocabulary expansion from V2.

The vendor with the most aggressive policy of opacity is Anthropic. The legacy claude-v1-tokenization.json was published for Claude 1 and never again. For Claude 3, 3.5, 4, and 4.7, the only authoritative way to count tokens is to call Anthropic's count_tokens endpoint. The repository javirandor/anthropic-tokenizer is the community workaround, reverse-engineered by asking Claude to repeat text and observing what comes back. It is approximate, and for many teams it is also the only way to estimate Claude cost without a round-trip to Anthropic's servers.

You can quote two vendors at the same dollar-per-million-token. You cannot quote them in the same units.

The AI exchange rate is not the same for everyone

If tokens are a currency, English speakers got a significantly better deal on it.

Petrov, La Malfa, Torr, and Bibi published "Language Model Tokenizers Introduce Unfairness Between Languages" at NeurIPS 2023. They evaluated 17 tokenizers across 200+ languages on FLORES-200, the standard multilingual parallel corpus. The single most cited number from that paper: the same translated sentence can produce "differences up to 15 times" in token count across languages, and the disparity persists even on tokenizers explicitly designed for multilingual support.

Yennie Jun's parallel study across 52 languages and 2,033 messages quantified the everyday cost. English averaged 7 tokens per message. Spanish, French, and Portuguese came in at similar rates. Hindi and Bengali ran approximately 5× English. Armenian ran 9×. Burmese reached 72 tokens, roughly 10× English for the same sentence. Her summary: "some languages require up to 10 times more tokens."

Newer tokenizers narrow the gap without closing it. Tamil's நீ ("you") was four tokens in cl100k_base and is a single token in o200k_base. Gemini's larger SentencePiece vocabulary handles CJK (Chinese, Japanese, Korean) noticeably better than the GPT-4 generation did. But the underlying training distribution is still English-dominant, and the structural penalty has not been engineered away.

What this means in practice: if your customer base is Hindi-speaking and your model uses an English-trained tokenizer, you are paying roughly four to five times what an equivalent English workload costs you, for content of equivalent meaning. The vendor did not charge you more per token. The exchange rate did the charging.

The AI token fees you cannot see on the menu

Once you accept that the token is the currency and the rate is the vendor's, the rest of the LLM bill becomes simultaneously more legible and stranger.

Reasoning tokens. Newer models think before they answer, and the thinking counts. OpenAI's o-series, DeepSeek's V4 in its thinking mode, Anthropic's extended-thinking models (Opus 4.7 by default now runs its xhigh effort tier in Claude Code), and Gemini's thinking modes all generate internal chain-of-thought that gets billed at the output rate. OpenAI exposes a count of reasoning tokens in output_tokens_details.reasoning_tokens, though the actual chain-of-thought is summarized or hidden. Anthropic in Opus 4.7 omits thinking content from the response by default, but the count is billed whether or not you request a summary. A 10,000-token reasoning budget on GPT-5.5 at $30/M output costs thirty cents before the user sees a word. Most teams running translation or classification through reasoning models are paying many times what they should because nobody turned thinking off.

Caching, which sometimes is not a discount. OpenAI applies cache discounts automatically, up to 90% off on cached input, with no write premium and no code changes required. Anthropic uses explicit cache_control markers: reads are 10% of base input (the deepest discount on the market), but writes cost 1.25× base on a five-minute TTL or 2× base on a one-hour TTL. Anthropic's own pricing page says the math amortizes after one cache hit on the five-minute tier, or two on the hour-long one. But if your prefix, the stable front of the prompt you mark for caching, gets written and rarely re-read, caching costs you more than not caching. Gemini layers in a per-hour storage fee ($4.50 per million tokens cached per hour on 3.1 Pro), and the cache-read rate also doubles above 200K context, so the long-context cliff applies to caching too.

Context cliffs. Most vendors charge a flat per-token rate across their full context window. A few do not. Gemini 3.1 Pro is $2 in / $12 out per MTok up to 200K, then $4 in / $18 out above. OpenAI's GPT-5.5 jumps to $10 in / $45 out above 272K input tokens. Anthropic's Opus 4.6 and 4.7 are flat across their 1M window, with one caveat: there is a 1.1× surcharge if you request US-only inference routing. A RAG pipeline whose prompt-length distribution flirts with these thresholds can change cheapest-vendor depending on the day's typical input length.

Batch. This is the one place the price card stops misrepresenting your actual cost. Google's batch API is half the standard rate; OpenAI and Anthropic match. If your workload tolerates async latency, batch is the single most impactful switch you can make, more than swapping vendors, and usually more than switching models within a vendor. The number of production pipelines that are eligible for batch and not using it is alarming.

What the LLM price card actually tells you

Pull all of this together and the dollar-per-million-token figure on a vendor's pricing page tells you exactly one thing: the dollar cost of one million units of that vendor's specific currency at this moment for the specific model variant and inference region and effort tier and TTL choice you picked. It does not tell you the cost of doing your work. The translation from "currency" to "work" sits inside the tokenizer, the language mix of your content, your reasoning-token policy, your cache hit pattern, your batch share, and your prompt-length distribution against context cliffs.

The honest unit for cross-vendor comparison is not cost-per-token. It is cost-per-task: what you pay, end-to-end, for a unit of output your users would accept. The formula is:

cost_per_task = (price_per_token × tokens_per_request × (1 + reasoning_overhead) × (1 − cache_hit_rate × cache_discount) × (1 − batch_share × 0.5)) ÷ task_success_rate

Every term is measurable. Token count comes from each vendor's official counter. Reasoning overhead comes from the usage block on each response. Cache hit rate and batch share come from your own logs. Task success rate comes from your eval. The headline $/MTok is one term on the right side, not the answer to the equation.

Cost is not the only thing that moves. More tokens means a longer sequence, which brings more latency, more KV-cache memory, and a higher chance of crossing the 200K/272K context cliffs in ways that can change how an agent loop behaves. Tokenizer inflation lands in your latency and memory budgets, not just the bill.

The state of the major LLM currencies, May 2026

Here is what each vendor is actually doing under their published rate card, in the dimensions that matter for real bills:

The vendors split into two groups. Open-tokenizer providers (OpenAI, DeepSeek, Meta, Mistral) let you compute cost client-side before you send the request, which is useful for budgeting, useful for routing, and useful when you are trying to A/B against another model. Closed-tokenizer providers (Anthropic, Gemini in part) require a round-trip to find out what the request will cost. That asymmetry is itself a feature of the pricing, even though it does not appear on any pricing page.

Auditing your own AI currency exposure

The practical version of this post is a worksheet, not a checklist. Sit with your team and answer:

1. What does your token actually buy? Take 50–100 real prompts from production logs. Run each through every candidate vendor's official token counter. Compare the counts. A 20% gap on your real content is normal; a 50% gap on non-English content is normal. Now multiply each count by each vendor's $/MTok and look at the result. This is the real per-call cost, not the price card cost.

2. How much are you paying to think? For every reasoning-capable model you use, pull the reasoning-token count from the usage block. Multiply by output price. If a non-trivial fraction of your bill is thinking tokens on workloads that do not need them, such as translation, classification, extraction, or short Q&A, turn thinking off. This is almost always the largest single optimization available.

3. Does caching cost you or save you? Pull the actual ratio of cache writes to cache reads from your logs. On Anthropic, if your prefix gets written more often than it gets read, the write premium is making things worse, not better. On Gemini, work out whether the per-hour storage fee on rarely-hit caches is worth the discount on the hits.

4. Where in your prompt-length distribution do the cliffs sit? Histogram your input lengths against 200K (Gemini) and 272K (GPT-5.5). If the 90th percentile is anywhere near those numbers, you are paying tier-2 rates more often than you realize.

5. What fraction of your traffic could be batch? If 30% of your work is async-tolerant and you are not batching it, you are leaving ~15% of your total bill on the table.

6. What's your real success rate per dollar? A cheaper-per-token model that fails your eval 20% more often is more expensive in production. Build the cost-per-accepted-task number. Reorder vendor preferences accordingly.

Why this state of affairs persists

It is tempting to read all of the above as vendors being deliberately opaque. Mostly, they are not. The Opus 4.6 → 4.7 tokenizer change was, on the merits, the right engineering trade for downstream quality. Anthropic's cache write premium reflects real storage and warm-cache costs. Gemini's 200K cliff is honestly disclosed. OpenAI's automatic caching is genuinely user-friendly. Every individual decision makes sense on its own.

What is missing is a market norm requiring vendors to publish their tokenizers, announce changes, and standardize the comparison unit. Until that norm exists, and there is no commercial incentive for any vendor to ship it unilaterally, the dollar-per-million-token figure on a pricing page will continue to be the most quoted and least informative number in the industry.

The fix lives on the buyer side. Track cost-per-task, not cost-per-token. Tokenize your real prompts before you migrate. Route by workload, not by vendor. Audit reasoning overhead like you audit any other line item. Treat the price card as one input to a model of your true unit economics, not the model itself.

And when a thread titled "Opus 4.7 to 4.6 Inflation is ~45%" climbs the front page of Hacker News on the day after a model launch, check the tokenizer before you check the price card.

— Written from the Altimate team. We build Altimate-Code, an agentic data engineering platform for enterprises. If you're working through this kind of vendor economics for your own stack, we'd like to compare notes.

We started altimate-code with one goal: build a harness that data engineers could actually trust with their pipelines and platforms. The first thing that became clear is that LLMs are the wrong tool for half of that job. Two queries either are or are not semantically equivalent. A lineage edge either does or does not exist. A row-level diff is either correct or not. None of those are creative questions, and a probability distribution is the wrong answer to any of them.

Over the past year, we have built a three-layer architecture that puts the LLM in charge of the work it is good at (strategy, intent-parsing, code generation) and a deterministic Rust and TypeScript stack underneath for everything that has to be reproducible. The split (detailed later) is what gets us to the top of the ADE and the DAB benchmarks, and more importantly, what makes the result repeatable on someone else's hardware.

This post walks through how the split actually works: where the boundary sits, the three operations we moved out of the model entirely, and what thousands of structured benchmark traces taught us about where agents actually fail.

LLMs are probability distributions

Run the same prompt through the same model twice, and you'll get two different answers. That's not a bug. It's the substrate. Language models sample from a probability distribution every time they generate a token, and the distribution is shaped — sometimes broadly — by sampling temperature[1], prompt-cache state, and the internal nondeterminism of large batched inference on shared hardware[2].

For creative work, this is exactly what you want. Drafting a SQL query, summarizing a result, recovering from an unexpected error — these are open-ended tasks with many acceptable answers, and probabilistic output is what makes the model useful.

For correctness, it's a liability. "Are these two queries semantically equivalent?" has one right answer. "What columns does this output project from this source?" has one right answer. "Are these two tables byte-equal?" has one right answer. When a probabilistic system answers a deterministic question, you can't cache the answer (it might be different next time), you can't debug it (a single sample tells you nothing about the distribution), and you can't benchmark it (a single pass-rate is a coin flip away from a different number).

What this looks like on a real benchmark

We took one task off the ADE benchmark, a dbt refactor called asana004, and ran it through altimate-code three times with the same model, the same prompt, and the same starting state. The agent passed it once and failed it twice. Across the three runs, dbt's test counts came back as 6/6, then 4/6, then 5/6.

If you are building a data-engineering agent, that is the problem you have to design around. You cannot trust a system whose output is intrinsically random. You cannot cache it. You cannot reproduce it. You cannot debug it. And you cannot reliably benchmark it.

The comfortable go-to (but wrong) answer

The instinct is to make the model itself more deterministic. Lower the temperature. Tighten the prompts. Introduce multi-agent voting. Run test-time consensus across N samples. Vendors in the data space are responding to this demand, and their responses make sense on some level. Temperature=0 is easy (provides marginal repeatability at the cost of much of the agent's ability to explore or recover from mistakes), ensemble voting is well-understood, and prompt engineering keeps getting better.

This works, partially. Lower temperature reduces variance but does not eliminate it. Multi-agent voting trades dollars for variance reduction but never reaches zero. None of it changes the fundamental shape of the problem. The model is a probability distribution, and you are sampling from it.

So stop fixing the model

The LLM should stay out of the correctness layer.

That is the bet behind altimate-code. The LLM stays in charge of strategy, intent-parsing, and code generation, which are the genuinely creative parts of the work. Everything else runs in a deterministic Rust and TypeScript layer underneath: SQL validation, schema diffing, query equivalence, lineage extraction, cross-database join inference, data-parity diffing. The agent does not decide whether two queries are equivalent. It calls a function that proves they are.

I keep coming back to this framing because it is the only one I have seen that survives contact with a real benchmark. Once you have watched the same prompt produce three different answers, you stop wanting the model to be more reliable. You start wanting it to be less involved.

Once you have watched the same prompt produce three different answers, you stop wanting the model to be more reliable. You start wanting it to be less involved.

Three layers

altimate-code's architecture has three layers. The decision that matters most is where the boundary between them sits.

The deterministic core is a Rust library (altimate-core) that exposes 34 SQL operations as pure functions over abstract syntax trees and schemas. It parses, validates, transpiles, fingerprints, checks equivalence, diffs schemas, classifies PII, extracts column lineage, and diffs rows across warehouses. Every operation runs sub-millisecond. Our benchmark battery validates 1,000 SQL queries in 30 ms and lints 1,000 queries in 250 ms. All of it is reproducible. All of it is zero-cost at the LLM API layer because it never touches the model. The engine carries 5,700+ Rust tests, runs on a single sqlparser-based AST representation, and targets 34 SQL dialects, from Postgres and Snowflake to Trino and TSQL.

The deterministic harness is the TypeScript code that runs the agent. A dispatcher maintains a registry of native handlers. When the agent calls altimate_core.transpile, the dispatcher routes the call to compiled Rust via napi-rs bindings, not to the model. Skills (encoded playbooks) prescribe step orderings. Helpers like dialect-aware identifier quoting prevent whole classes of "the model generated almost-correct SQL" failures.

The probabilistic agent is the LLM. It reads task descriptions, plans, picks tools, writes drafts, summarizes results, and recovers from failures. It sits in the creative layer, not the correctness layer.

The agent does not always know which layer it is calling into. But it always can know. A hasNativeHandler check classifies a tool call as deterministic or not before it runs. That asymmetry is most of the value.

Consider what happens when the agent calls altimate_core.track_lineage. The tool wrapper packages the arguments. The dispatcher looks up the handler registered for that method name. The handler normalizes the agent-supplied schema, accepting either the flat {table: {col: TYPE}} shape or the nested SchemaDefinition shape, and converts both to a canonical form before passing through. The napi-rs binding calls into compiled Rust. The Rust engine builds the lineage graph and returns it. The tool formats the output for the agent. From the agent's perspective, it called a tool and got back lineage. From the architecture's perspective, every step is deterministic, including the schema normalization, which the harness does, not the model.

What this looks like in practice

I will walk through three concrete operations, because adjectives do not earn the claim.

Take two queries written differently but doing the same thing:

-- Writer A
SELECT id FROM users
WHERE status = 'active'
  AND created > NOW() - INTERVAL 30 DAY

-- Writer B
SELECT id FROM users
WHERE created > DATE_SUB(NOW(), INTERVAL 30 DAY)
  AND status = 'active'

altimate_core.checkEquivalence proves these are semantically identical without running either of them. The comparison happens on parsed ASTs against the provided schema. It handles predicate reordering, equivalent date-arithmetic functions, and identical column projection. The result is a boolean plus a confidence score, not a sample of an LLM's opinion. Downstream, that means refactor verification, migration safety checks, and regression tests that confirm a query's behavior did not change. None of it requires a live database or a model that "looks at" the queries.

Dialect translation is the same story. transpile(sql, fromDialect, toDialect) takes a Snowflake query and produces a BigQuery one (or BigQuery to Postgres, or Postgres to Databricks) via sqlparser-rs and dialect-specific AST transforms. It runs sub-millisecond. The agent never has to memorize that Snowflake's IFF(cond, a, b) becomes CASE WHEN cond THEN a ELSE b END in ANSI dialects. We have measured the LLM-direct alternative. The failure mode is not "wrong output." It is almost-right output that compiles in dev and breaks in prod when an edge-case function shows up. Determinism here is not about speed. It is about coverage. The AST transform handles every case the parser handles, not every case the model happens to have seen in training.

schemaFingerprint(schema) returns a SHA-256 hash of a serialized schema. That is the entire interface. The hash is what makes caching trustable. When the agent is validating many queries against a large schema, we compute the fingerprint once and use it as a cache key. Identical schemas produce identical hashes, and the cache hits. An LLM-derived "is this schema the same?" cannot offer that guarantee. A model might say "yes" twice and "no" the third time, and you would have to debug whether the third answer was the wrong one. The fingerprint also acts as the index for progressive context compression. The core exposes five disclosure levels: fingerprint-only, table list, column names (the default), full detail for relevant tables only, and the complete schema - and the agent picks the level it needs for the question at hand. The selection is deterministic, the cache key is deterministic, and nothing about either step is the LLM's opinion of "is this schema close enough?"

Reliability and performance turn out to be the same lever pulled twice. Determinism is what makes the cache safe. The cache is what makes the agent fast. You cannot get the second without the first.

Case study: cross-database join inference

The cleanest illustration of this principle is a tool we shipped in late April 2026. altimate-code is often connected to several warehouses at once (Snowflake, BigQuery, Postgres), and the agent needs to suggest cross-database joins. One warehouse calls an entity customer_42. Another calls it cust_42, or c-42, or 42. Naming and formatting disagree, so naive column-name matching fails.

A pure-LLM approach to this is straightforward to describe and unreliable to ship. You prompt the model with the schemas and ask "which columns join across these DBs?" It works on the cases the model has seen. It hallucinates plausible-but-wrong matches on the cases it has not. Different runs return different rankings.

The deterministic version is about 400 lines of TypeScript. The algorithm is small enough to summarize:

1. From each connection, sample ~50 string values per column.

2. Compute the longest common prefix per column, trimmed back to the last _, -, or : separator. So ["businessid_42", "businessid_43", …] produces prefix "businessid_". Sequences without a separator are rejected.

3. For each pair of columns from different databases, require distinct non-empty prefixes (so the prefix actually distinguishes the two columns), strip the prefixes, and compute the set intersection of suffixes.

4. Emit a candidate when the suffix overlap is non-empty. Rank by overlap size, then by overlap / min(left, right).

businessid_42 on one DB and businessref_42 on another both reduce to suffix 42. The algorithm pairs them with score 1.0 and reports the rule as "shared suffix after stripping distinct prefixes."

The function that computes the common prefix is fewer than 30 lines. It walks character-by-character and trims back to the last separator. Identical inputs produce identical outputs, every time, in any order, on any machine. We can ship the algorithm to a customer and tell them exactly what will happen.

Case study: column lineage with depth tiering

Lineage is the question of where each output column comes from. Given a query with joins, CTEs, and expressions like SUM(o.amount + o.tax), you need to know which upstream columns flow into each output. An LLM can answer this question in plain English. It cannot answer it precisely enough to chain operations on. Agents that act on lineage need exact edges, not summaries.

altimate_core.column_lineage does this as an AST walk. Each output column gets a list of source columns plus a transformation lens: direct projection, function call, expression, or aggregation. The walk is deterministic by construction. Subqueries and CTEs are inlined. Window functions decompose into their partition and order clauses. CASE expressions report all branches as sources.

The piece worth featuring is the depth tiering. The same engine exposes three modes:

• Basic: direct source-target edges only, with minimal context.

• Deep: adds transformation lenses (expression, function, aggregation), confidence scores, and the internals of every CTE the lineage flows through.

• Full: transitive closure across multi-query chains, plus indirect dependencies from WHERE, JOIN, and GROUP BY predicates that affect row selection.

The agent picks the depth it needs for the question at hand. The classification of what counts as "basic" versus "deep" is encoded. It is not an LLM call disguised as one. The pattern repeats: every level of detail an agent can ask for is a level the engine knows how to produce, and the choice between them is a parameter, not a guess.

The same principle controls the output format. Lineage, like every operation in altimate-core, ships with two formatters: a verbose one for humans, and a compact one optimized for an agent's context window. Compact is the default. The agent gets a headline token (LINEAGE | 5 cols, 3 tables, depth=basic) followed by structured column <- source rows it can parse without spending tokens on prose. Depth chooses how much detail; the format chooses how that detail is encoded; both compound across a trajectory of hundreds of tool calls.

Case study: data parity

A different shape of problem: prod was migrated to a new warehouse, and we need to know whether the rows match.

The naive answers are bad. You write a 200-line custom SQL diff query and hope you covered the edge cases, or you ask an LLM to "compare these two tables" and get an unactionable summary.

The engine implements two complementary algorithms. HashDiff uses bisection plus checksums for cross-database diffs. It partitions both tables, checksums each segment, recursively bisects any segment whose checksums disagree, and terminates at the row level on exactly the segments that contain mismatches. The bisection lets it pinpoint the exact rows that differ instead of scanning every row, and the module ships with 303 dedicated tests covering every dialect pair we support. JoinDiff uses a same-database FULL OUTER JOIN for when both tables live in the same warehouse. The cost characteristics differ. The correctness guarantee is the same.

The engine ships with a cooperative state-machine API. It emits the SQL it wants executed, the caller runs it, and the engine steps forward based on the result. There is no point in the loop where the engine guesses, or where its behavior depends on which LLM is in front. Given identical inputs, the diffs come out identical, every time.

"Are these two tables byte-equal?" is the antithesis of a creative task.

Smaller wins, in the harness

So far the story has been about the Rust core. The TypeScript harness gets the same treatment, and the wins are smaller per-instance but they compound.

A single function called quoteIdentForDialect holds the truth about identifier quoting across our supported warehouses. MySQL gets backticks, T-SQL gets brackets, and everything else gets ANSI double quotes. The LLM never has to remember which is which, because it never builds quoted SQL directly. It passes column names through tools that route through quoteIdentForDialect. That is not a clever optimization. It is a category of failure removed from the system. The same pattern applies to EXPLAIN-plan retrieval. Postgres has EXPLAIN (ANALYZE, BUFFERS). Snowflake has EXPLAIN USING TEXT. BigQuery has no EXPLAIN statement at all, and you read plans from job metadata. A ~50-line decision table maps each warehouse to its plan-retrieval mechanism. The agent does not ask the model "what is the BigQuery EXPLAIN syntax?" because a tool already knows.

Skills are altimate-specific procedures encoded in markdown. The dbt-develop skill prescribes a four-step workflow: plan (check the layer and naming conventions), discover (use schema-search tools, read existing YAML), write (generate SQL with validated joins), and validate (call altimate_core_validate, then column_lineage, then dbt build). The skill does not replace the LLM. It constrains where the LLM is creative and where the harness is prescriptive. Our trace data shows skill-driven sessions complete in fewer turns and with fewer tool errors than ad-hoc sessions on the same task.

None of these are individually dramatic. Cumulatively they are the difference between an agent that almost works and one you can deploy.

What thousands of session traces taught us

We ran the ADE-Bench DuckDB benchmark against altimate-code and captured structured traces for every session. Our benchmark topping score was 74.4%.[2] The traces are where the interesting story lives.

Failure has three distinct causes, and they require three different responses.

The first cause is stochastic LLM noise. Two airbnb tasks (airbnb005, airbnb006) failed the original baseline but then passed all three reruns. asana004 (the task from the opening) passed once out of three. The inputs are the same, the outputs differ, and the cause is intrinsic to the model. There is nothing to fix here except the model.

The second cause is deterministic capability gaps. asana003 failed at exactly 16 of 17 dbt tests, every single time across reruns. Same failure mode every time, with the same dbt-test count. The agent's output has a stable bug we can investigate and fix.

The third cause is bench-fixture bugs. Three airbnb tasks (airbnb001, airbnb002, airbnb008) failed reproducibly, but the root cause was a pre-existing bug in the bench's own dbt incremental models: a broken is_incremental() filter that collapses the date series to a single row, so the LAG window function in the MoM/WoW aggregation returns NULL on rerun and overwrites the correct first-run values. This is not an altimate-code problem. The maintainers shipped a test-setup workaround in dbt-labs/ade-bench#106, but the model SQL itself is still broken — fix proposed in dbt-labs/ade-bench#145.

From the outside, all three look identical. The test failed. Only the deterministic substrate (structured traces, reproducible reruns, exact dbt-test counts) lets us separate them. Without that, we would be guessing about what is actually wrong, and our improvement work would be aimed at the wrong target.

Where the market is

Most coding agents in the field are general-purpose. Cursor, Cline, Claude Code, Codex, and GitHub Copilot Workspace all sit in that category. They are built for the generic developer-with-a-text-editor case. When a data engineer uses them to write a Snowflake query or refactor a dbt model, the tools treat SQL like any other code. Text goes in, text comes out, and the model decides. There is no deterministic substrate for the data-specific operations because there does not need to be one for general coding.

The data-native AI products (dbt Cloud's AI features, in-warehouse LLMs like Snowflake Cortex) are closer to the right surface area, but they are still primarily LLM wrappers with a different UI. They generate SQL. They do not prove SQL.

Our bet is that for data engineering specifically, the LLM-as-creative-layer / code-as-correctness-layer split is the right architecture, and that the deterministic infrastructure is where the actual moat lives. Models will get better. APIs will get cheaper. Prompt techniques will improve. None of that changes the calculus on whether two queries are semantically equivalent, or whether a cross-warehouse data diff is correct. Those are facts about code and data, decidable by code that operates on code and data.

altimate-code currently sits at number one on the ADE benchmark, ahead of Claude Code, Cortex Code CLI, and dbt Labs, by a wide margin. It also recently scored #1 on the Berkeley EPIC Data Lab DAB (Data Agent Benchmark)

What is still LLM territory

It would be dishonest to claim we replaced the LLM. We did not, and we will not. The LLM still owns intent-parsing ("refactor the asana__project model into an intermediate one that…"), code generation, description-writing for dbt model YAML, and strategy under uncertainty. When a build fails for reasons that are not immediately classifiable, the model is what figures out the next move.

What's next

Comparative trace analysis across models. Sonnet, Opus, DeepSeek v4 pro, Kimi K2, and Qwen 2.5 Coder all run through the same altimate-code stack on the same benchmarks. What we don't yet have is a systematic comparison of how they differ on data-engineering tasks — which failure modes are model-specific vs architecture-specific, which models pair best with which skills, where the cost-quality frontier sits per task category. The traces are captured; we’ll share the analysis soon.

Harness resilience. The same trace data surfaces a handful of harness-side next moves. We are making the agent loop resilient to truncated model outputs from streaming providers. We are adding a per-generation timeout so a hung request does not tie up a worker for 25 minutes. We are detecting when the agent has fallen into a same-tool-same-argument loop and forcing a replan. Each one is a place where the harness can be more deterministic about how it handles a probabilistic model.

Cost / quality frontier mapping. Our DeepSeek v4 pro run on DAB scored 0.5693 stratified Pass@1 for $9 of inference, ahead of the then benchmark leader Pi's Opus 4.6 submission — is one data point in a larger frontier. We’re in touch with DAB maintainers to add this run to the leaderboard. The natural follow-up is systematic: across the model lineup we already run through altimate-code, at what cost does each model fall below a quality threshold, broken out by task category (SQL refactor, dialect translation, lineage extraction, cross-DB joins)? The architecture is model-agnostic; the right backbone for a given customer workload may be category-dependent.

Conclusion

The claim is narrower than "no LLM." It is that the LLM stays out of correctness. The probabilistic layer handles probabilistic work. The deterministic layer handles deterministic work. The boundary is where the payoff lives.

Probabilistic models are a remarkable substrate for the things they are good at: language, intent, creativity. They are a poor substrate for proof, equivalence, and reproducibility. The architectural question for any agent in this space is not "can we use the LLM for X?" It is "should we, and if not, what should we use instead?"

For the data-engineering work altimate-code does, the answer is the correctness layer underneath: a deterministic core in Rust, a deterministic harness in TypeScript, with the LLM kept exactly where it belongs.

[1]: Claude API Docs

[2]: Defeating Nondeterminism in LLM Inference

altimate-code is an open source project: github.com/AltimateAI/altimate-code.
If you are building in this space and want to compare notes, reach out by email at info-at-altimate.ai*.*

What is Blast Radius

Blast radius refers to the potential extent of damage.

For example, before you knock down a wall in your house, you want to know if there's plumbing behind it, electrical wiring within it, or if it's holding up the floor above.

Same idea with data. Before you rename a column, change a formula, or update a filter, you need to know what depends on it. That could include downstream models, dashboards, reports, or other workflows.

Need for Blast Radius Analysis

Often when you rename a column and push it to production, everything seems fine. Two days later, finance pings you on Slack: the weekly revenue dashboard has been broken since Tuesday.

You dig in. The column you renamed was referenced in three other models. One of those models feeds a Tableau workbook. That field is the denominator in the revenue metric on the CFO's dashboard.

Nobody documented any of this. The dependencies existed in the relationships between systems, invisible until something broke. This is the blast radius problem.

Altimate Code fixes this. Before any change goes through, it maps out the full impact automatically. It produces a detailed blast radius report (what will break, what's safe, what needs someone to sign off) and also performs the changes.

I ran a workflow using Altimate Code to verify this.

The workflow: renaming total_cents to gross_revenue_cents

I ran this on my open-source CartWave E-commerce dbt project, a real analytics codebase with staging models, intermediate transformations, mart tables, and a semantic layer on Snowflake.

stg_orders is a simple staging model that reads raw order data and explicitly picks the columns that are needed. The finance team wants us to rename total_cents to gross_revenue_cents to align with GAAP terminology.

The prompt

I gave Altimate Code this prompt:

"Rename the column total_cents to gross_revenue_cents in stg_orders as the finance team wants to align with GAAP terminologies. Before performing the changes, show me the blast radius of the same."

One column, one rename. It should have been straightforward, but it wasn't.

What Altimate Code did

After receiving the prompt, Altimate Code ran through a series of steps before changing a single line of code. Here's what happened, in order.

https://youtu.be/Npf7fHK43-k

Step 1: It loaded the dbt-analyze skill and scanned the project.

Altimate Code started by tracing the dependencies. It read the model SQL, searched every file in the project for references to stg_orders, and checked YAML config files for column-level references.

Within seconds it found five files that depend on stg_orders:

Five files depend on this model. But how many of them actually use the column we're renaming? That's the question that matters.

Step 2: It checked each file for column-level references to total_cents.

Altimate Code checked more than whether a file depended on stg_orders. It opened each downstream file and looked for direct references to total_cents.

Here's what it found:

int_order_enriched.sql – BREAKING

Directly selects orders.total_cents on line 31:

orders.total_cents,   -- directly uses the column we're renaming

Explanation: This file asks for a column called total_cents. After the rename, that column won't exist anymore. The query will fail.

int_customer_lifetime.sql – BREAKING

Sums total_cents across all orders on line 22:

sum(total_cents) as lifetime_revenue_cents,  -- adds up all order values

Explanation: Same problem. The column disappears, the math breaks.

mart_return_analysis.sql – SAFE

This one depends on stg_orders but only uses customer_id and order_date:

orders.customer_id,   -- doesn't touch total_cents
orders.order_date,    -- doesn't touch total_cents

Explanation: It never references the renamed column. No changes needed.

sem_orders.yml (semantic layer) – BREAKING

Two metric definitions reference total_cents by name in their formulas:

measures:
-name: total_revenue_cents
agg: sum
expr: total_cents          # references the column by name

-name: avg_order_value_cents
agg: average
expr: total_cents          # references the column by name

Explanation: This is the one that would have bitten me. The semantic layer is a YAML config file, not SQL. It doesn't show up if you're just grepping through .sql files. But if you rename the column and forget to update these metric definitions, every dashboard using "total revenue" or "average order value" breaks silently.

Step 3: It traced the impact further downstream.

Altimate Code didn't stop at the first layer. It followed the chain: if a breaking model feeds another model, that downstream model could be affected too.

Two mart tables sit downstream of int_customer_lifetime: mart_customer_360 (full customer profiles, contains PII) and mart_customer_cohorts (customer segmentation). But neither one references total_cents directly. They use a derived column called lifetime_revenue_cents, which int_customer_lifetime produces from total_cents and then passes downstream under its own name.

Think of it like fixing a pipe in the basement. The faucets upstairs keep working as long as the pipe gets repaired properly. These two marts don't need code changes, they just need int_customer_lifetime to be fixed first.

One thing worth noting: mart_customer_360 is tagged compliance: 'CCPA / GDPR' and contains_pii: true. It doesn't need code changes here, but in a governed environment, you still want to know that a compliance-tagged table sits in the blast radius.

Step 4: It presented the full blast radius report.

Altimate Code assembled everything into a single view:

The bottom line: Five files need changes. Three are safe.

Then the agent asked me:

"I've identified 6 files that need changes across 3 layers (staging, intermediate, semantic). The 2 mart models downstream are safe because they reference derived column names. Shall I proceed with the rename?"

It didn't just run ahead. It stopped, showed me everything, and waited for the go-ahead.

Step 5: I approved. It applied all changes at once.

After I said yes, Altimate Code updated all five files in a single pass. No partial updates, no window where half the models use the old name and half use the new one.

stg_orders.sql – aliased the source column:

total_cents as gross_revenue_cents,  -- raw data keeps actual source column name, downstream sees new name

The raw data still has a column called total_cents. Instead of changing the source, the staging model gives it a new name. Everything downstream sees gross_revenue_cents from here on.

int_order_enriched.sql – updated the reference:

orders.gross_revenue_cents,  -- was: orders.total_cents

int_customer_lifetime.sql – updated the aggregation:

sum(gross_revenue_cents) as lifetime_revenue_cents,  -- was: sum(total_cents)

sem_orders.yml – updated both metric definitions:

-name: total_revenue_cents
description:"Sum of gross revenue in cents (GAAP-aligned)"
expr: gross_revenue_cents   # was: total_cents

-name: avg_order_value_cents
description:"Average order value in cents (GAAP-aligned)"
expr: gross_revenue_cents   # was: total_cents

_staging.yml – updated the documentation:

-name: gross_revenue_cents
description:"Gross revenue in cents, aligned with GAAP terminology"

Step 6: It built the entire project to confirm nothing broke.

Altimate Code ran altimate-dbt build across all thirty-four models:

Thirty-six passed. Zero errors. Every staging view, intermediate table, and mart, including the PII-tagged mart_customer_360, built cleanly.

Step 7: It verified the column lineage end to end.

As a final check, Altimate Code traced the data flow from source to destination to confirm the rename propagated correctly:

The source column TOTAL_CENTS flows through the alias in staging, propagates correctly to both intermediate models, and the derived column names downstream (lifetime_revenue_cents) stay the same. Exactly what the blast radius report predicted.

Column Changes Are Never Just Column Changes

A simple rename turned out to touch five files across three layers plus the semantic layer. Without the blast radius check, I would have updated stg_orders, run the build, and moved on. The semantic layer would have broken silently. Finance would have found out before I did.

How to read a blast radius report

Every downstream asset gets one of four labels:

The report also flags compliance tags. In our case, mart_customer_360 carries compliance: 'CCPA / GDPR' and contains_pii: true. It didn't need code changes here, but those flags make sure the right people know a compliance-tagged asset is in the blast radius.

Try it out yourself:

I ran this on a real project. To reproduce it:

1. Clone the CartWave demo project: https://github.com/altimateanas/ecommerce_demos.git

2. Set up your database connection

3. Launch Altimate Code

4. Give Altimate Code the prompt "Rename the column total_cents to gross_revenue_cents..." from the article above

It'll scan the project, map every dependency at the column level and show you a structured report before changing anything.

Data engineering looks like software development from a distance, but the work has a different center of gravity. You still write code, review changes, and ship to production, but the real risk often lives outside the diff: downstream lineage, metric correctness, warehouse cost, schema drift, PII exposure, and whether the business can still trust the data after a change lands.

Most data engineering work also does not begin and end with writing SQL. A small model change can require checking upstream sources, understanding downstream impact, updating dbt docs, adding tests, comparing outputs, reviewing query performance, and making sure sensitive data is handled correctly. We created data engineering skills for Claude Code to bring those workflows into the coding agent experience, giving data teams reusable playbooks for the work they already do every day.

You can also access these skills through Altimate Code, our open-source LLM harness for data engineering, instead of installing each one individually in Claude Code. Altimate Code includes the skills out of the box and is ready to use with Claude:

npm install -g altimate-code

The skills are open source, so you can use Altimate Code or browse the full set directly on GitHub: https://github.com/AltimateAI/altimate-code/tree/main/.opencode/skills

Skills as Data Engineering Playbooks

A skill is a reusable workflow Claude Code can load when a task calls for a specific kind of expertise. For data engineering, that means more than a prompt template. Each skill gives Claude Code guidance on what context to gather, what checks to run, what tools to use, what risks to watch for, and what kind of answer or artifact to produce.

For example, a SQL review skill should not only say whether a query is syntactically valid. It should look for anti-patterns, safety issues, readability problems, performance risks, and whether the query is appropriate for the warehouse or dialect. A dbt testing skill should understand model structure, schema YAML, unit test patterns, and the kinds of edge cases that usually cause production data issues.

The goal is to make Claude Code less dependent on one giant perfect prompt from the user. Instead of asking a data engineer to spell out every step, the skill carries the workflow: build the model, document the columns, add the tests, inspect the lineage, compare the outputs, review the SQL, and summarize what changed.

The Data Engineering Workflows We Covered

We organized the skills around the workflows data teams repeat every week: building models, reviewing SQL, validating changes, debugging warehouse issues, governing sensitive data, and communicating results. Each skill is focused on a specific job, but the real value comes from how they work together across the lifecycle of a data change.

The current skill set covers six areas:

• Building and maintaining dbt projects

• Reviewing and improving SQL

• Validating changes before they ship

• Operating the warehouse

• Governing data and team knowledge

• Visualizing and explaining results

Building and Maintaining dbt Projects

The dbt skills cover the full lifecycle of a model: creating it, documenting it, testing it, understanding its downstream impact, and troubleshooting it when something breaks.

Creating and Changing Models

At it’s core, dbt is a tool that allows users to create SQL models, so we had to have a skill that creates dbt models. It is called dbt-develop.

This skill teaches Claude how to both write new dbt models and change existing ones. If you are a dbt user, you know that is not as simple as just writing SQL. You have to understand the project setup, dialect of the database, patterns across the staging, intermediate, and mart models, and the lineage of data flowing throughout the project.

Documenting the Project

If you are anything like me, you’ve spent days or even weeks creating an intricate dbt project that takes messy, raw data and forms it into beautiful marts that you deliver to your stakeholders. You neglect to create good documentation for your work because you created it so why would I need to write what it does?

A few weeks later you receive a message from a colleague asking what a specific column does, and you have no idea what the answer is. Documentation might have helped out a bit here.

When I say documentation, I mean going past the basic notes of primary key or unique ID for customer. Our skill, dbt-docs, writes documentation that will answer questions about models and columns that you might not have thought about.

The skill guides Claude to write about aggregations, primary and foreign keys, upstream tables, etc. Our hope is that this skill will allow you to stop answering questions about your models and columns and spend more time building more.

https://youtu.be/PY63_Eu3Si4

Writing Tests for dbt Models

Before working in data, I was a teacher, so I have always loved a good test. Not the kind that exists just to make someone nervous, but the kind that makes expectations clear. A good test tells you what should happen, what matters, and where the gaps are.

That is exactly why tests matter in dbt. There is a special kind of confidence that comes from seeing a model run successfully. There is also a special kind of pain that comes from realizing later that the model ran successfully and still produced the wrong answer.

The dbt-test skill helps Claude Code add schema tests, unit tests, and data quality checks to dbt models. It can guide Claude Code through common checks like not_null, unique, relationships, and accepted_values, as well as custom generic or singular tests when a model needs something more specific.

For more complex model logic, the dbt-unit-tests skill goes deeper. It helps generate dbt unit tests by analyzing the model SQL, identifying upstream refs and sources, creating mock inputs, and assembling YAML that tests things like CASE statements, joins, window functions, null handling, aggregations, and incremental behavior.

The point is not to test for the sake of testing. It is to protect the assumptions your model depends on, so the next change does not quietly break the meaning of the data.

https://youtu.be/O49qIMh2QPQ

Understanding and Fixing Impact

dbt changes have a habit of traveling. A small edit to a source model can ripple into downstream marts, dashboards, tests, and metric definitions. The dbt-analyze skill helps Claude Code reason about that blast radius before a change ships by inspecting dependency and lineage context, including column-level lineage where available.

The other side of impact is troubleshooting. When a dbt project fails to compile, a model errors at runtime, a test starts failing, or a dashboard suddenly looks wrong, the dbt-troubleshoot skill gives Claude Code a diagnostic workflow instead of leaving it to guess from the error message alone. It helps separate compilation issues, warehouse errors, test failures, incorrect data, and performance problems.

Together, these skills help Claude Code answer two questions that come up constantly in dbt work: “What could this change affect?” and “What is actually going wrong?”

Reviewing and Improving SQL

Even in a dbt-heavy world, SQL is still the language data teams use to express business logic. But good SQL is not just SQL that runs. It should be readable, safe, efficient, and appropriate for the warehouse it runs on. The SQL skills help Claude Code review, optimize, and translate queries across the systems data teams actually use.

Reviewing SQL Before It Ships

Every data team has seen a query that technically works but makes you nervous. Maybe it has a risky join, a missing filter, unclear business logic, a performance issue hiding in plain sight, or a pattern that will be painful for the next person to maintain.

The sql-review skill gives Claude Code a pre-merge review workflow for SQL. It guides Claude Code to look for syntax issues, anti-patterns, readability problems, performance risks, and safety concerns before the query lands in production.

The goal is not just to ask, “Does this query run?” It is to ask, “Is this query understandable, reliable, and safe enough for the data system it is about to become part of?”

Optimizing Slow or Expensive Queries

Slow queries are rarely just an inconvenience in data engineering. They can block development, delay dashboards, frustrate stakeholders, and quietly drive up warehouse costs. A query that was fine on a small table can become a problem as data volume grows or as more teams start depending on it.

The query-optimize skill helps Claude Code inspect SQL with performance in mind. It can look for inefficient joins, unnecessary scans, filtering issues, aggregation patterns, and opportunities to rewrite the query in a way that is easier for the warehouse to execute.

This is where Claude Code becomes useful beyond code generation. It can help reason through why a query is slow, what tradeoffs are available, and how to make the SQL more efficient without losing the business logic that made the query valuable in the first place.

Translating Across Dialects

If you’ve done much traveling, you know that speaking the same language does not always mean speaking it the same way. SQL is similar. Snowflake, BigQuery, Databricks, Postgres, Redshift, MySQL, SQL Server, and DuckDB all share the same broad language, but each warehouse has its own syntax, functions, conventions, and sharp edges.

A query written for BigQuery will not always translate cleanly into Snowflake. The same is true when moving models between warehouses, supporting multiple customer environments, or trying to standardize logic across a mixed data stack.

That is why we built the sql-translate skill. It helps Claude Code translate SQL from one dialect to another while preserving the intent of the query. Think of it as Duolingo for Claude Code, except instead of asking it to practice ordering coffee, you are asking it to keep your business logic intact across warehouses.

If your team is migrating warehouses or working across multiple SQL engines, this skill gives Claude Code a much better starting point for making those translations safely.

Validating Changes Before They Ship

In software engineering, tests often tell you whether a change broke expected behavior. In data engineering, validation can be harder to pin down. A model can run, a query can return rows, and a migration can apply successfully while the meaning of the data has still changed.

That is why this category matters so much. Data teams do not just need help making changes. They need help understanding whether those changes are safe to ship. The validation skills help Claude Code compare outputs, inspect lineage, and look for migration risks before a change reaches production.

Comparing Data Outputs

One of the most common questions in data work is also one of the most important: did this change alter the data?

Sometimes the answer should be yes. You fixed a bug, changed a business definition, or added new logic. But often, especially during a refactor or migration, the goal is to preserve behavior while changing the implementation. That is where the data-parity skill becomes useful.

The data-parity skill helps Claude Code compare two tables or query results and diagnose exactly how they differ. It can be used for migration validation, ETL regression checks, and query refactor verification. Instead of stopping at “these counts do not match,” the workflow pushes toward understanding where they differ, why they differ, and whether the difference is expected.

Think of it as asking Claude Code to check the receipt before you leave the store. The change may look fine at a glance, but you want to know whether the numbers actually add up.

Seeing How Lineage Changed

A data change is not only about the rows that come out the other side. It is also about how values move through the system. When the lineage changes, the meaning of a column can change with it.

The lineage-diff skill helps Claude Code compare column-level lineage between two versions of a SQL query or model. It can show which data flow edges were added, removed, or changed, giving the reviewer a clearer picture of how the transformation shifted.

This is especially useful when reviewing changes that look small in the SQL but may affect important downstream fields. A join changes. A source column is swapped. A derived field starts pulling from a different upstream path. The query may still run, but the story of the data has changed.

The goal is to make those invisible changes visible before they surprise someone downstream.

Catching Migration Risk

Schema changes are another place where “it ran successfully” is not enough. A migration can apply cleanly and still introduce data loss, break assumptions, or create problems for downstream consumers.

The schema-migration skill helps Claude Code analyze DDL changes before they are applied. It looks for risks like type narrowing, dropped columns, missing defaults, removed constraints, and other breaking column changes that can quietly turn into production issues.

This matters because schema migrations often feel mechanical until they are not. Renaming a column, changing a type, or tightening a constraint might look straightforward in code, but those changes can affect ingestion jobs, BI tools, reverse ETL flows, contracts, and every team that has built something on top of the data.

The skill gives Claude Code a migration review mindset: not just “Can this statement execute?” but “What could this break if it does?”

Operating the Warehouse

Once data work is running in production, a different set of questions starts to matter. What is expensive? What is slow? What changed? Which workloads are driving spend? Which queries or models need attention? The warehouse is not just where data lives. It is also where performance and cost decisions show up.

The warehouse operations skills help Claude Code reason about those questions as part of the engineering workflow. Instead of treating cost and performance as separate admin tasks, these skills bring them closer to the code, queries, and models that created them.

Understanding Cost

Warehouse spend can be difficult to reason about because the cost is usually spread across queries, users, jobs, warehouses, and workloads. A dashboard may feel slow, a bill may jump, or a team may notice that a routine transformation suddenly got more expensive, but finding the cause is not always obvious.

The cost-report skill helps Claude Code analyze Snowflake query costs and identify optimization opportunities. It can help look at expensive queries, warehouse usage, credit consumption, unused resources, and patterns that may point to waste or inefficient workloads.

This gives Claude Code a FinOps lens. Not in the “please enjoy this spreadsheet of guilt” sense, but in the useful sense: where is the money going, what changed, and what can we do about it?

Diagnosing Operational Issues

Cost is only one part of operating a warehouse. Data teams also have to deal with slow queries, failing jobs, runtime errors, and models that behave differently as volume grows.

This is where several skills work together. The query-optimize skill helps Claude Code reason about slow or inefficient SQL. The dbt-troubleshoot skill gives it a workflow for compilation failures, runtime database errors, failing tests, incorrect data, and performance issues in dbt projects.

The goal is to make Claude Code useful when something is not healthy in production. It can help move from symptoms to causes: from “this dashboard is slow,” “this model failed,” or “our warehouse spend jumped” toward a clearer explanation of what is happening and what to try next.

Governance, Privacy, and Team Knowledge

Data engineering also carries responsibilities that do not fit neatly into “write the model” or “make the query faster.” Teams need to know where sensitive data lives, whether a query exposes it, and whether new work follows the standards the team has already agreed on. The governance and training skills help Claude Code operate with more awareness of privacy, compliance, and team-specific conventions.

Auditing Sensitive Data

Sensitive data has a way of showing up where you least expect it. An email field gets added to a downstream model. A phone number moves into an analytics table. An IP address appears in a query result that was supposed to be safe to share. Unlike pie, PII is much better when it is not casually passed around.

The pii-audit skill helps Claude Code classify schema columns for personally identifiable information and sensitive data, including direct identifiers like SSNs, emails, phone numbers, names, addresses, and credit card numbers, as well as quasi-identifiers like dates of birth, zip codes, IP addresses, and device IDs.

It can also check whether a query or dbt model exposes PII, distinguish between sensitive fields used internally and sensitive fields returned in the output, and help generate a PII inventory for compliance workflows like GDPR, CCPA, and HIPAA.

The point is not to turn Claude Code into a compliance department. It is to give it enough privacy awareness to pause at the right moments, surface risk, and help data teams avoid accidentally spreading sensitive data through models, reports, or ad hoc queries.

https://youtu.be/KtWwjVIhlGI

Teaching Claude Code Your Team’s Standards

That same teaching instinct shows up in the teach, train, and training-status skills. A lot of what makes someone effective on a data team is not just knowing SQL or dbt. It is learning the patterns, preferences, definitions, and little bits of context that make work fit the team around it.

Every data team has conventions that are obvious to the people who have been there long enough and invisible to everyone else. How staging models should be named. What belongs in marts. Which patterns are encouraged. Which shortcuts should be avoided. Which business definitions have sharp edges. Most of that knowledge lives in scattered docs, review comments, Slack threads, and the brains of the people who have answered the same question ten times.

The teach skill lets you show Claude Code an example file from your codebase and extract reusable patterns from it. The train skill helps Claude Code learn team standards from a document, style guide, or review checklist. The training-status skill shows what it has learned so far.

These skills help Claude Code move closer to the way your team actually works. The goal is not just technically valid output. It is output that reflects your standards, your naming conventions, your modeling patterns, and the context your team has built over time.

Visualizing and Explaining Results

Data work does not end when the query returns rows. At some point, someone needs to understand what the data is saying. That might mean a chart for a trend, a dashboard for a team, a KPI view for leadership, or a more interactive way to explore a dataset.

The data-viz skill helps Claude Code turn data into visual interfaces: charts, dashboards, KPI cards, analytics views, and reporting experiences. It guides Claude Code toward modern component libraries like Recharts, Tremor, Nivo, D3, Victory, visx, and shadcn/ui, depending on the project.

This matters because the last mile of data work is often communication. A model can be correct, tested, documented, and optimized, but if people cannot understand the result, the work is not finished. The data-viz skill helps Claude Code move from “here is the data” to “here is what the data means.”

Claude Code is already a powerful place to work with code. These skills make it more useful for the specific work data teams do every day: building models, writing tests, reviewing SQL, validating changes, checking lineage, auditing PII, understanding cost, translating dialects, and explaining results.

The goal is not to replace data engineers. It is to give them a better teammate inside the workflows they already know. Data engineering requires code, context, caution, and communication. These skills give Claude Code more of that context, so it can help with the work around the code, not just the code itself.

If you want to use the skills without installing them one by one in Claude Code, they are included in Altimate Code, our open-source LLM harness for data engineering. Altimate Code comes with these skills installed and is ready to work with Claude:

npm install -g altimate-code

If you're trying to figure out which remaining data conferences in 2026 are worth your time and travel budget, this is the list. We've covered the biggest events on the calendar, from global platform summits drawing 20,000+ attendees to practitioner-focused gatherings where the real technical conversations happen.

Listed in chronological order.

Gartner Data & Analytics Summit

March 9-11 | Gaylord Palms, Orlando, FL | ~5,000 attendees

If your job involves data strategy, AI governance, or making the case to leadership for data investments, this one belongs near the top of your list. The summit runs on research from Gartner analysts: over 60 experts delivering 137 sessions across five main tracks and three spotlight tracks. This year's focus was squarely on moving past AI pilots and into production-grade AI that actually delivers ROI.

The crowd skews toward senior roles: CDAOs, heads of AI, data architecture leads, data management executives. The format matches that audience: keynotes and roundtables built for strategic conversation, not product demos. There are 130+ vendors in the Exhibit Showcase if that's useful, but the sessions are where the value is. Gartner also runs this summit in London and Sydney for teams outside North America.

Tableau Conference

May 5-7 | San Diego Convention Center, San Diego, CA | ~8,000 attendees

TC26 is three days of 300+ expert-led sessions, 150+ hands-on trainings, and the kind of community programming that makes this feel less like a tech conference and more like a reunion. Tableau's community calls itself the DataFam for a reason. The culture is genuinely welcoming in a way that most data conferences aren't.

Content this year centers on agentic analytics, interoperability, and new product integrations. If your team uses Tableau day-to-day, the skill-building alone makes this worth attending. Sessions run from beginner dashboard design to advanced AI integration. There's also Iron Viz (a live data viz competition), Tableau Doctor for 1:1 expert help, and Salesforce+ livestreams for anyone who can't make it in person.

Snowflake Summit

June 1-4 | Moscone Center, San Francisco, CA | ~20,000 attendees

Snowflake Summit 2026 has 500+ sessions, hands-on labs, training and certification, and an expo floor with 190+ partner booths. The Builders Hub is specifically for developers: bootcamp sessions and live demos. The Industry Zone covers vertical-specific content for finance, healthcare, retail, and more.

The theme this year is agentic intelligence: how to move from isolated data strategies to connected, AI-powered enterprise experiences. If your team runs on Snowflake, this is where you get ahead on what's coming from the platform and hear real implementation stories from companies that have already shipped it. Main keynotes are livestreamed free for non-registrants.

Catch the Altimate AI team at booth 1311

Databricks Data + AI Summit

June 15-18 | Moscone Center, San Francisco, CA | ~22,000 attendees

Data + AI Summit brings together data engineers, scientists, architects, ML engineers, and executives from over 160 countries across 800+ sessions covering data engineering, warehousing, governance, analytics, agentic AI, and AI/BI.

If you're working with Apache Spark, Delta Lake, Apache Iceberg, MLflow, or dbt alongside Databricks, this is where roadmaps get announced and where you'll find the most advanced technical content on those tools. The conference also offers new certification courses on-site. One caveat: it's a big event. Plan your session schedule before you arrive or you'll spend the week just trying to navigate the venue.

Catch the Altimate AI team at booth 577

Ai4

August 4-6 | Las Vegas, NV | ~12,000 attendees

America's largest independent AI conference, drawing 12,000+ executives and decision-makers from 90+ countries. Ai4 is enterprise-focused. The emphasis is on real-world AI implementation and measurable business outcomes, not research. Tracks are organized by industry (finance, healthcare, retail, manufacturing) and by function (AI strategy, generative AI, AI agents, automation).

The audience skews toward business and technical leadership. If you're evaluating AI strategy or want to see how peers in adjacent industries are actually deploying AI in production, this is a useful venue. For teams wondering whether their AI initiatives are in line with what the broader market is doing, the comparison is genuinely worth making.

dbt Summit (formerly Coalesce)

September 15-18 | The Cosmopolitan, Las Vegas, NV | ~3,000 attendees

Coalesce has a new name. dbt Summit 2026 is the world's largest gathering of dbt users: analytics engineers, data engineers, data architects, and the data leaders working alongside them. The rebrand reflects how the event has grown from a community conference into a full industry summit, spanning four days with 100+ sessions across keynotes, breakout tracks, and hands-on labs.

The conversations here are practitioner-driven. You won't find a lot of polished vendor pitches. You'll find engineers being honest about what's working and what isn't. Key themes include dbt Mesh, data contracts, AI-ready data architectures, incremental models, and how analytics engineering intersects with ML workflows. For data teams building on dbt, this is the one worth planning around.

Catch the Altimate AI team on the Expo floor

Big Data LDN

September 23-24 | Olympia London, London, England | Free to attend

The UK's largest data, analytics, and AI event and the most important data conference on the European calendar. Big Data LDN brings together engineers, architects, data leaders, and practitioners for two days of 400+ seminars across 15 free-to-attend conference theatres. Speakers come from Google, AWS, TikTok, Spotify, Snowflake, Databricks, and a wide range of startups.

Free registration, 130+ exhibitors ranging from major platform vendors to early-stage companies, and sessions covering AI governance, data engineering, and data products. In 2026, a co-located event called Data Driven LDN runs on September 22nd, focused specifically on AI agents, AI governance, and data products for senior practitioners who want deeper dives.

Catch the Altimate AI team on the Expo floor

How to Pick the Right Data Conference in 2026

Nobody can do all of these, so a few filters are worth applying. If your team runs on Snowflake, Snowflake Summit is the obvious call. The same logic applies for dbt and dbt Summit. Platform-specific conferences tend to have the highest signal-to-noise for teams already using those tools. Data leaders and strategists get the most out of Gartner and Ai4. Practitioners and engineers tend to get more hands-on value from dbt Summit and the Databricks summit. Tableau Conference is its own category: best for teams where Tableau is the primary visualization layer. European teams should have Big Data LDN near the top of the list. It's free, comprehensive, and the audience quality is high.

Pick the ones where your stack, your role, and your team's current problems overlap with the program. That overlap is usually obvious once you look.

Claude Code is fast, writes good software code, and can even handle dbt models and other data engineering tasks to certain extent.

At first glance Altimate Code, the data engineering harness, may seem to do similar things. So, do we need both? Is one a replacement for the other?

We ran an experiment to find out. We used the same prompt, the same model (Claude Opus 4.6), and the same codebase. One run with Claude Code alone, one run with Altimate Code. The results were not close.

What Claude Code Actually Is

Claude Code is a general-purpose agentic coding tool that reads files, runs shell commands, edits code etc. For data engineering, it draws on Claude's extensive training coverage of SQL dialects and dbt conventions.

But Claude Code is no data engineering expert. It has no deterministic SQL anti-pattern engine, no static lineage tracer, no PII classifier, no schema diff tool that programmatically flags breaking changes.

And there is no evidence the team at Anthropic is focused on making it any better at these tasks.

What Altimate Code Actually Is

Altimate Code is an open-source data engineering harness with 100+ specialized tools for building, validating, optimizing, and shipping data products. It uses LLMs (Claude, GPT, Gemini, or any of 17+ providers) as its AI backend, but routes every task through domain-specific tooling that general-purpose agents do not have:

• Live warehouse connection -- connects directly to various warehouses with auto-discovery from profiles.yml or environment variables.

• dbt-native build and test -- runs real dbt build against your warehouse, materializing tables and executing every data test.

• Column-level lineage -- traces every column from source through joins, CTEs, and subqueries to final output in real time.

• PII detection -- scans schemas across 15+ PII categories (SSN, email, phone, DOB, health data) with confidence scores.

• Impact analysis and schema diff -- calculates blast radius across your full dbt DAG and produces column-level before/after diffs with breaking change classification.

• SQL quality grading -- scores SQL on syntax, style, safety, and complexity (A-F) for objective, reproducible code review.

• Enforced agent modes -- Builder (can modify), Analyst (read-only), Plan (design only) -- enforced at the harness level, not by prompt. You cannot DROP TABLE in Analyst mode regardless of what the LLM suggests.

• Project conventions via AGENTS.md -- team-wide rules loaded into every session's system prompt for consistency across engineers and CI.

How Claude Code and Altimate Code Work Together

Claude Code and Altimate Code are not competitors or alternatives .They occupy different layers of the stack. Claude Code is a general-purpose coding agent. Altimate Code is a domain-specific data engineering harness.

When used together, Claude Code handles task orchestration and conversational context while Altimate Code’s tools handle warehouse connectivity, lineage tracing, PII scanning, build execution, and impact analysis. The LLM’s reasoning improves when it has access to specialized tools enabling access to real data, real schemas, and real test results instead of guessing from file contents alone.

The comparison in this post demonstrates why domain-specific tooling matters for data engineering work.

The Experiment: Claude Code vs. Altimate Code

As an experiment we took a realistic, broken dbt model mart_patient_360 from a demo dbt healthcare project medflow-analytics — a scenario that data engineering teams encounter regularly — and gave the exact same prompt to two setups:

1. Claude Code standalone and

2. Altimate Code + Claude Code, same codebase, and same task.

The goal: to see whether domain-specific data engineering tooling produces meaningfully different outcomes than a general-purpose coding agent when the task involves schema accuracy, HIPAA compliance, build verification, and downstream impact analysis — the things that actually matter in production data work.

Model: Claude Opus 4.6 was used in both experiments to ensure a fair comparison.

The prompt we used :

The mart_patient_360 model is incomplete. Right now it joins patients, encounters, diagnoses, medications, and lab_results but the SELECT is mostly empty — it’s missing the patient_id primary key, has no aggregated metrics, and just exposes raw PII fields like SSN and phone number. I need you to build this out into a proper patient 360 view: add the patient_id key, total encounter count, unique diagnosis count, active medication count, most recent lab result date, days since last visit, and a patient risk tier (high/medium/low based on encounter frequency and diagnosis count). The model is tagged as PII/HIPAA-restricted. Make sure the final model is safe to materialize, fix any compliance issues you see, and tell me what downstream impacts or governance concerns I should be aware of before merging.

Claude Code:

Altimate Code:

Head-to-Head Results: Data Engineering Harness Vs. General Purpose Coding Assistant

Claude Code Output:

Enhanced the model but without execution or proper validation, surfaced limited insights:

Altimate Code Output:

Unlike Claude Code, Altimate Code enhanced the dbt model, executed it, did proper validation, and surfaced detailed insights:

Detailed Findings

1. PII and HIPAA Compliance

This is where the gap was most visible and most consequential.

Why this matters: Claude Code’s SHA-256 hash of SSN is a common pattern, but it’s a weaker approach than most teams realize. SSNs are 9 digits — roughly 900 million possible values. Altimate Code’s approach of full elimination is the correct HIPAA-compliant pattern for analytical marts.

In our experiment, Altimate Code’s lineage_check tool revealed that full_name (a PII field) was flowing through a CTE even though it wasn’t in the final SELECT. Claude Code missed this entirely. Altimate Code’s lineage engine claims 100% edge match accuracy across 500 benchmark queries.

2. Schema Accuracy: Did It Actually Build?

Why this matters: Claude Code trusted the YAML documentation, which was out of sync with the actual SQL in multiple directions. Some columns were documented but missing from the SQL, while blood_type was the reverse case (in the SQL but undocumented). This is extremely common in real dbt projects. Altimate Code verified against multiple sources (SQL, YAML, seed data) and resolved the discrepancies.

The altimate_core_schema_diff tool produces a column-level before/after comparison with explicit breaking change classification (e.g. [BREAKING] Column 'ssn' removed). In our experiment, this confirmed 16 schema changes with 6 breaking giving the team an exact migration checklist.

3. Build Verification and Data Validation

Why this matters: Claude Code wrote the code and declared it done. Altimate Code wrote the code, built it on Snowflake, ran every test, queried the output data, and verified the results made clinical sense. In production data engineering, “the SQL looks right” is not the same as “it works.”

4. Downstream Impact Analysis

Why this matters: In production data environments, the most dangerous changes are the ones that look safe in isolation. Altimate Code's impact_analysis tool traverses the full DAG programmatically, and its schema_diff produces a migration checklist that a team lead can review.

5. Governance Recommendations

Why this matters: Claude Code offered textbook advice: reasonable, but generic. Altimate Code's recommendations were actionable at the PR level: a specific YAML annotation for Safe Harbor, a specific CI check for tag containment, a specific warning about non-deterministic columns that would produce different results depending on when the pipeline runs.

These are the details that prevent a compliance review from becoming a compliance finding.

The Extra Steps Altimate Code Took

These are capabilities that Claude Code simply does not have access to:

Summary: The Scorecard

In Conclusion: An AI Coding Assistant Needs a Domain Expert Harness to Master Data Engineering

The takeaway of all this is that general-purpose AI + domain-specific intelligence produces categorically better results than either alone. For data engineering work where correctness, compliance, and safety matter, the domain layer is not optional.

Altimate Code's value is not that it replaces Claude Code Its value is that it surrounds Claude Code with 100+ specialized tools that verify, build, test, scan, and validate before declaring the job done. For data engineering teams shipping to production, that difference is the entire gap between "looks right" and "is right."

Steps To Reproduce This Analysis

We’ve open-sourced the full analysis so you can reproduce it:

Repository: github.com/altimateanas/altimate_code_enterprise_demos

1. Clone the repo: git clone https://github.com/altimateanas/altimate_code_enterprise_demos

2. Navigate to medflow-analytics/directory It's a healthcare dbt project running on Snowflake with patient data, claims, encounters, diagnoses, medications, and lab results.

3. Setup your snowflake target in dbt profiles.yml.

4. Run the prompt in section "The Experiment: Claude Code vs. Altimate Code" in Claude Code (standalone) then observe the output

5. Connect Altimate Code and run the same prompt and compare

Make sure to use the same underlying LLM in both runs.

tl;dr : Today, we are launching Altimate Code, an open-source agentic data engineering harness that far exceeds generic LLMs (and other leading models) on data engineering tasks. Check out the project on GitHub, or catch the March 25th overview and AMA on YouTube.

AI agents have transformed how software gets built. But for data engineering, they have fallen short…

An AI coding agent on Replit deleted an entire production database during a code freeze, then created 4,000 fake records to fill the empty tables. An independent evaluation of Snowflake's Cortex Analyst found 38% logical accuracy. Six out of ten queries were wrong, but compiled and ran just fine. One team got a $5,000 bill from a single Cortex AI query their resource monitors never caught. 78% of AI-generated SQL errors are silent wrong joins, queries that return confidently incorrect data.

These aren't edge cases. They're what happens when AI agents operate on data infrastructure without safety layers. No schema validation, no lineage, no cost controls, no permission enforcement. The agent doesn't know your schema. It can't trace what breaks downstream. It doesn't know what a query will cost. And the system prompt telling it "don't drop tables" stops working at 100K context tokens.

The problem isn't the model. It's everything around it.

The missing layer

General-purpose coding agents treat SQL like application code. It isn't. SQL operates on schemas that change, across dialects that diverge on every function name, through lineage chains no LLM can reliably trace, with cost implications that scale to thousands of dollars per mistake.

What data engineering needs is a layer of compiled, deterministic tools that operate outside the LLM's reasoning loop. Tools that validate SQL against your actual schema in 2ms, trace column-level lineage through CTEs deterministically, and catch anti-patterns with zero false positives. Not better prompts. Better engines.

That's what we built.

Introducing Altimate Code

Today we're open-sourcing Altimate Code, a data engineering harness built on OpenCode, the open-source coding agent. Built by the team behind dbt Power User, the most widely used dbt VS Code extension.

The core idea: the LLM reasons; compiled engines validate; neither replaces the other.

We ran dbt Labs' ADE-bench, the open standard for measuring AI agents on real data engineering tasks.

A cheaper model with compiled tools outperformed a more expensive model without them. The difference isn't the model. It's the harness. Further details can be found here.

What the harness catches

Your agent writes a query. Before it touches your warehouse:

The wrong table name that would have triggered a 30-second Snowflake error? Caught in 2ms with a fix suggestion. Five agent fix cycles cost 10ms and not 2.5 minutes of warehouse round-trips.

The cartesian join that would have silently inflated your numbers by 100x? Caught before execution. 26 compiled anti-pattern rules, zero false positives across 1,077 benchmark queries.

The column your downstream dashboard depends on? Traced to its source through every JOIN, CTE, and subquery with 100% edge match on 500 benchmark queries. The agent reasons on verified lineage, not guesses.

The PII in the staging table you're about to expose? Flagged and masked. The SQL injection hiding in a Jinja template? Blocked.

All compiled. All deterministic. All in milliseconds.

Making the harness your own

Engines are the foundation. But what makes Altimate Code yours is what you build on top of them.

Persistent memory spans across sessions in two scopes: global (your preferences) and project (team knowledge versioned in git). Tell the agent "we never use FLOAT for money columns"… it remembers. Next session, next teammate, the knowledge is there. When one engineer corrects the agent, every teammate inherits the fix on git pull. No Slack message. No wiki update. The correction just propagates.

Governed agent modes enforce permissions at the engine level, not through prompt instructions that models ignore at long context lengths. The Analyst can't INSERT, UPDATE, DELETE, or DROP. Not because of a system prompt, but because the compiled engine won't execute it. The Builder gets full read/write with your SQL rules applied. The Planner maps tasks without executing.

The compactor summarizes long sessions while preserving data engineering state: warehouse connections, schema context, dbt project state, lineage findings. Multi-hour sessions maintain continuity across compaction boundaries.

The tracer captures every LLM call, tool invocation, and warehouse metric locally. No external services. No data leaving your machine. Run /trace for an interactive viewer.

Why independent?

Snowflake shipped Cortex Code. Databricks launched Genie Code. Both recognized that general-purpose agents don't work for data. But both shipped solutions that suit their ecosystem.

Cortex Code won't help you migrate off Snowflake and onto BigQuery. Genie Code is not designed to optimize a Redshift query. Your Airflow DAGs don't run inside Databricks. Your warehouses span multiple providers. Your governance crosses every platform boundary. Your AI agents should too.

Altimate Code connects to Snowflake, BigQuery, Databricks, PostgreSQL, Redshift, DuckDB, MySQL, SQL Server, Oracle, and SQLite. It runs with Anthropic, OpenAI, Google, AWS Bedrock, Azure, Ollama, and OpenRouter. With local models and the local-only tracer, it runs fully air-gapped so no data leaves your machine.

Platform agents will always have telemetry we can't access. We'll always have independence they can't offer. Neither Snowflake nor Databricks will build first-class support for the other's warehouse. Neither will tell your agent a query is unnecessary when their revenue depends on you running it.

Your harness should be yours.

Try it

npm install -g @altimateai/altimate-code
altimate /discover

Two commands. It auto-detects your warehouse, indexes your schema, and you're building.

Open source on GitHub. Docs at altimate-code.sh. Ten data stores. Any LLM. No platform tax.

What we're building next, and where you can help

The compiled engines ship today. Here's where we're headed and where we need the community:

Pipeline monitoring for Airflow and Dagster. We want this to be proactive, not just interactive.

Blast radius analysis. Before the agent acts, show what breaks downstream.

Decision memory: extract why things were built from your Git history so agents stop undoing decisions they don't know were made.

Agent Data sandboxes: changes prove themselves before touching production and many more

Some of these we'll build. Some of them you'll build first. The roadmap, the benchmarks, and every known gap are in the repo. PRs welcome. Issues welcome. Forks welcome.

We'd rather build this in the open than wait for it to be built inside a walled garden. If you're a data engineer who stitches tools together for a living, come break it.

Want to learn more or get involved?

Three options:

1. Check out the project on Github - star the repo to support the project

2. Join us on Slack - we are launching a new Slack channel for our Agentic Data Engineering efforts. Join us here.

3. Sign up for next week's live overview and AMA on YouTube.

Today, we're open-sourcing Altimate Skills — a collection of Claude Code skills specifically designed for analytics engineers. We are starting with skills for dbt and Snowflake. These encode the workflows and best practices that transform AI from a basic code generator into a capable data engineering assistant.

https://youtu.be/kvIo5PmF0Ns

Key Results:

• +25% improvement on model creation tasks (40% → 65%)

• +22% faster execution (TPC-H 1TB) with 100% logically equivalent queries generated for SQL optimization.

• 53% accuracy on ADE-bench (43 real-world dbt tasks)

• Skills that actually teach Claude how to work, not just what to write

# Get started in 30 seconds
/plugin marketplace add AltimateAI/data-engineering-skills
/plugin install dbt-skills@data-engineering-skills

GitHub: https://github.com/AltimateAI/data-engineering-skills and ⭐ the repo.

Solving the Context and Workflow Issue

If you've used Claude Code, Cursor, or any AI coding assistant for dbt development, you've experienced the frustration:

The task: "Create a staging model for the Stripe payment source."

What you expect:

-- models/staging/stripe/stg_stripe__payments.sql
{{
  config(
    materialized='view',
    schema='staging'
  )
}}

with source as (
    select * from {{ source('stripe', 'payments') }}
),

renamed as (
    select
        id as payment_id,
        amount_cents / 100.0 as amount,
        currency,
        status,
        created_at
    from source
)

select * from renamed

What you get:

SELECT * FROM {{ source('stripe', 'payments') }}

No stg_ prefix. No {{ source() }} reference. No config block. No CTEs. No understanding of your project's conventions.

Why This Happens

The core issue isn't that LLMs lack knowledge — Claude knows dbt syntax perfectly well. The problem is context and workflow:

1. No project awareness — Claude doesn't know your naming conventions, folder structure, or existing patterns

2. No verification loop — Claude declares "done" after writing code, without running dbt build

3. No convention discovery — Claude guesses at patterns instead of reading existing models first

4. Compile ≠ Success — dbt compile passes, but the model produces the wrong output

This leads to a frustrating cycle: AI writes code → You review and fix → AI loses context → Repeat.

What Are Claude Code Skills?

Anthorpic introduced Claude Skills in October 2025. Skills are markdown files that teach Claude how to approach tasks, not just what syntax to use. Think of them as encoding the workflow an experienced analytics engineer follows.

Skills matter because they make Claude more reliable and specialized: you can standardize repeatable work (like formatting outputs, following internal conventions, or running a known process) and reuse it across projects. Claude can also load Skills progressively, starting with lightweight metadata and pulling in deeper instructions only when needed, giving you targeted behavior without bloating context.

Practically, a Skill is typically packaged as a small folder of structured instructions (and optionally templates, scripts, or reference files) that define how Claude should approach a workflow. See our data-engineering-skills repo folders as an example.

A skill has two parts:

1. Trigger conditions — When should this skill activate?

---
name: creating-dbt-models
description: |
  Guide for creating dbt models. ALWAYS use this skill when:
  (1) Creating ANY new model (staging, intermediate, mart)
  (2) Task mentions "create", "build", "add" with model/table
  (3) Modifying model logic or columns
---

2. Workflow instructions — What steps should Claude follow?

# dbt Model Development

**Read before you write. Build after you write. Verify your output.**

## Critical Rules
1. ALWAYS run `dbt build` after creating models - compile is NOT enough
2. ALWAYS verify output after build using `dbt show`
3. If build fails 3+ times, stop and reassess your approach

## Workflow
### 1. Understand Requirements
- What columns are needed?
- What is the grain (one row per what)?
- What calculations are required?

### 2. Discover Project Conventions
cat dbt_project.yml
find models/ -name "*.sql" | head -20
Read 2-3 existing models to learn patterns...

When Claude encounters a task that matches the trigger conditions, it automatically applies the skill's workflow.

The Skills We Built

dbt Skills

Snowflake Skills

Deep Dive: How creating-dbt-models Changes Claude's Behavior

Let's trace through a real task from our benchmark to see the difference.

Task: "Create a model called dim_customer that renames id to customer_id and makes it a unique primary key."

Without Skills (Baseline Claude)

Claude's approach:

1. Write the SQL immediately:

SELECT
  id AS customer_id,
  *
FROM customers

1. Run dbt compile → passes

2. Declare done

Outcome: Model compiles, but:

• Wrong table reference (raw table vs staging model)

• No config block matching project patterns

• No verification that the output is correct

• No unique test added for primary key

With creating-dbt-models Skill

(See the specific SKILL.md file here)

Claude's approach:

Step 1: Understand Requirements

• Columns needed: customer_id (renamed from id), plus all other columns

• Grain: One row per customer

• Constraint: Must be unique on customer_id

Step 2: Discover Project Conventions

$ cat dbt_project.yml
# Shows model-paths, naming conventions

$ find models/ -name "*.sql" | head -10
models/staging/stg_customers.sql
models/staging/stg_products.sql
models/marts/dim_products.sql

$ cat models/marts/dim_products.sql
# Learns: CTEs named 'source' and 'final', config blocks, column naming

Step 3: Find Upstream Models

$ cat models/staging/stg_customers.sql
# Discovers the correct source to reference

Step 4: Write Model (following discovered patterns)

-- models/marts/dim_customer.sql
{{
  config(
    materialized='table'
  )
}}

with source as (
    select * from {{ ref('stg_customers') }}
),

final as (
    select
        id as customer_id,
        name,
        email,
        created_at
    from source
)

select * from final

Step 5: Update schema.yml with test

models:
  - name: dim_customer
    columns:
      - name: customer_id
        tests:
          - unique
          - not_null

Step 6: Build and Verify

$ dbt build --select dim_customer
# Actually runs the model and tests

$ dbt show --select dim_customer --limit 5
# Verifies output looks correct

Outcome: Model matches project conventions, has proper tests, output verified.

Benchmarking: ADE-bench Results

We evaluated our skills using ADE-bench, a framework for evaluating AI agents on analytics engineering tasks created by dbt Labs.

Test Setup

• 43 tasks across 5 projects (Airbnb reviews, F1 racing, Asana projects, Analytics engineering, Intercom conversations)

• Task types: Model creation, bug fixing, debugging, refactoring, data analysis

• Model: Claude Sonnet 4.5

• Database: Snowflake

• Evaluation: Automated tests comparing model output to expected results

Task Difficulty Distribution

Overall Results

Results by Task Category

What Worked

Model creation saw the biggest improvement. The creating-dbt-models skill's workflow of "discover conventions → write → build → verify" catches errors that baseline Claude misses:

1. Convention discovery prevents wrong naming/structure

2. Mandatory dbt build catches runtime errors that compile misses

3. Output verification ensures the model produces correct data

Example success — Task analytics_engineering003:

"Create a model called 'dim_customer' that renames id to customer_id, and makes that row a unique primary key."

• Baseline: Created model but wrong column reference, no test

• With skills: Discovered existing staging model, matched project patterns, and added proper unique test

What Didn't Work

Complex analysis tasks remain challenging. Tasks requiring deep reasoning about data behavior (like identifying which queries have non-deterministic results due to ties) still need human insight.

Example failure — Task f1003:

"Identify which top-N tables have inconsistent results due to ties in the data"

This task requires:

1. Understanding the semantic meaning of "ties"

2. Analyzing actual data values across 8 models

3. Reasoning about SQL ordering behavior

Skills can't encode this kind of domain reasoning — they work best for workflow guidance, not analytical judgment.

The Overhead Trade-off

Skills add overhead. The "discover conventions" step takes 15-30 seconds of additional LLM calls. Is it worth it?

Our conclusion: The 30-second overhead is worth it because:

1. Successful tasks need no human review

2. Failed tasks fail faster (3-failure rule)

3. Time saved on human review >> time spent on convention discovery

Benchmarking: SQL Query Optimization

We evaluated the optimizing-query-text skill on TPC-H SF1000 (1TB dataset).

Test Setup

• 10 queries from TPC-H benchmark

• Model: Claude Sonnet 4.5

• Evaluation: Automated comparison of query results + execution time

Overall Results

What Failed Without Skills

Baseline failed 2 queries by making "optimizations" that changed what the query returned:

• Changed deduplication behavior, returning extra rows

• Renamed columns, breaking downstream compatibility

Installation & Usage

Add the Marketplace

/plugin marketplace add AltimateAI/data-engineering-skills

Install Skills

You can browse and install via the CLI, or directly install plugins:

# Install dbt skills
/plugin install dbt-skills@data-engineering-skills

# Install Snowflake skills
/plugin install snowflake-skills@data-engineering-skills

After installing, skills activate automatically when you mention relevant tasks.

Available Skills

dbt Skills:

• creating-dbt-models — Model creation with convention discovery

• debugging-dbt-errors — Systematic error troubleshooting

• testing-dbt-models — Schema tests and data quality

• documenting-dbt-models — Generate descriptions in schema.yml

• migrating-sql-to-dbt — Convert legacy SQL to dbt models

• refactoring-dbt-models — Safe restructuring with impact analysis

Snowflake Skills:

• optimizing-query-text — Optimize SQL you provide

• optimizing-query-by-id — Optimize using query ID from history

• finding-expensive-queries — Find top cost/time queries

Usage

Skills activate automatically based on your request:

Combining Skills with Altimate MCP Tools

Skills become even more powerful when combined with Altimate's MCP server. The MCP server provides real-time access to your dbt project and data warehouse:

Example: Skills + MCP workflow

User: "The daily_revenue model is producing wrong numbers."

Claude (with skills + MCP):

1. debugging-dbt-errors skill activates

2. Uses dbt_model_details to get model SQL and dependencies

3. Uses dbt_compile to check for errors

4. Queries upstream models to verify input data

5. Identifies the issue (e.g., missing WHERE clause)

6. Fixes and rebuilds

7. Uses dbt show to verify the correct output

What We Learned Building This

1. Workflow > Knowledge

The biggest wins came from encoding workflows, not facts. Claude already knows dbt syntax — what it lacks is the discipline to:

• Check existing patterns before writing

• Run dbt build instead of dbt compile

• Verify output after build

2. The 3-Failure Rule

We added this to every skill:

"If build fails 3+ times, STOP. Step back and reassess your entire approach."

This prevents Claude from making tiny tweaks hoping they work. Instead, it forces a fundamental rethink.

3. Skills Can't Replace Domain Expertise

Skills work best for procedural tasks with clear success criteria. They struggle with:

• Tasks requiring business context

• Ambiguous requirements ("make this better")

• Deep analytical reasoning about data behavior

4. Convention Discovery Is Essential

The #1 source of Claude errors was mismatched conventions. Simply adding "read 2-3 existing models first" eliminated most of these.

What's Next

We're actively developing:

• Airflow skills — DAG development, debugging, testing

• Cross-platform migration — dbt ↔ SQL Server, Oracle

• Snowflake cost optimization — Warehouse sizing, query patterns

• Data quality workflows — Anomaly detection, freshness checks

Contributing

Altimate Skills is open source (MIT License). We welcome:

• New skills for workflows we haven't covered

• Improvements to existing skills based on your team's patterns

• Benchmark results on different datasets

GitHub: https://github.com/AltimateAI/data-engineering-skills

Try It Now

# Add the marketplace
/plugin marketplace add AltimateAI/altimate-skills

# Install the plugins you need
/plugin install dbt-skills@altimate-skills
/plugin install snowflake-skills@altimate-skills

Resources:

• GitHub Repository and ⭐ the repo.

• Altimate MCP Server Docs

• ADE-bench Framework

• dbt Slack #tools-dbt-power-user

Built by the team at Altimate AI — Making data engineering delightful.

Managing Snowflake warehouses efficiently is a constant balancing act. Set them too large, and you're burning money on idle compute. Too small, and your queries slow to a crawl, frustrating users and missing SLAs. For many organizations, this manual tuning process is time-consuming while still leaving significant cost savings on the table.

Two solutions promise to solve this challenge in very different ways: Snowflake's new Adaptive Compute eliminates warehouse management complexity entirely, while Altimate AI's Auto Tune brings intelligent automation to standard warehouses with granular cost controls. This guide breaks down how each works, their tradeoffs, and which to choose based on your specific needs.

What is Adaptive Compute?

Today, managing Snowflake warehouses requires juggling multiple decisions: choosing the right size (X-Small through 6X-Large), configuring cluster policies, setting auto-suspend timers, and more. Get these wrong, and you're likely to be overpaying for idle compute or suffering from poor query performance.

Adaptive Compute changes this equation entirely. Instead of fixed-size warehouses, Adaptive Warehouses act as pointers to a shared, elastic compute pool that Snowflake manages behind the scenes. When you submit a query to an Adaptive Warehouse, Snowflake automatically:

• Analyzes the query plan and resource requirements

• Allocates the right amount of compute from the shared pool

• Routes the query to available resources

• Scales compute up or down based on actual needs

Think of it like switching from buying dedicated servers to using serverless functions – you stop worrying about infrastructure and focus on your workloads.

Figure 1: With Adaptive Warehouses, customers still interact with named warehouses, but under the hood, each warehouse is simply a pointer to a shared compute pool within the account. Queries are automatically routed to appropriately sized compute clusters based on their resource needs and availability.

From Many Settings to Just Two

Snowflake’s standard warehouses require configuring multiple parameters:

• Warehouse size

• Minimum and maximum cluster count

• Scaling policy (Standard or Economy)

• Auto-suspend timing

• Auto-resume behavior

• Query Acceleration

Adaptive Compute replaces all of these with just two settings:

1. Warehouse Credit Limit: The maximum credits the warehouse can consume per hour. This acts as your primary control lever, determining how many concurrent queries can run based on their compute requirements. When the warehouse reaches its credit limit, additional queries are queued until resources become available.

2. Target Statement Size (temporary): An optional hint about your expected query sizes. Currently, Adaptive Compute only supports scaling down for smaller queries, and this setting defines the maximum target size. Note that Snowflake intends to deprecate this setting once Adaptive Compute’s autoscaling capabilities mature[1], so don’t build processes around it.

Figure 2: Adaptive Compute reduces configuration complexity by replacing multiple tuning settings with just two controls: a Warehouse Credit Limit and an optional Target Statement Size.

If you convert a standard warehouse to Adaptive, Snowflake sets the initial Warehouse Credit Limit based on the formula: Warehouse Credit Limit = Current Warehouse Size Credits × Max Cluster Count.

It also sets the Target Statement Size equal to the original warehouse size.

For example, if you convert a Large warehouse (8 credits/hour) with max_clusters=4 to Adaptive, you get:

• Warehouse Credit Limit: 32 credits/hour (8 credits × 4 clusters)

• Target Statement Size: Large

Your Adaptive warehouse can then concurrently run queries totaling up to 32 credits per hour; for example, four Large queries, sixteen Small queries, or any combination fitting within the credit limit.

Where Adaptive Compute Delivers

Simplified Infrastructure Management

Adaptive Compute removes much of the manual setup required to manage Snowflake warehouses, freeing up data teams to concentrate on driving business outcomes and delivering innovation. This hands-off model is especially beneficial for platform teams managing large fleets of warehouses, and less technical users who don’t want to think about warehouse tuning.

Smarter Resource Allocation

Because Snowflake's optimizer has access to detailed query plan metadata (e.g., estimated row counts, join strategies, data volumes), it can predict resource needs more accurately than external tools. Assigning just enough CPU and memory per query also reduces the likelihood of over-provisioning.

Example scenario: Two very different queries are submitted to the same Adaptive Warehouse simultaneously:

As shown in the table, Adaptive Compute predicts greater compute needs for the Query A (the heavier query) than Query B (the simpler query).

Seamless Conversion from Standard Warehouses

Converting to Adaptive Compute is simple and requires no downtime:

ALTER WAREHOUSE my_warehouse SET TYPE = 'ADAPTIVE';

Running queries complete on the old compute while new queries immediately use adaptive resources. There’s no need to update connection strings, modify scripts, or coordinate downtime windows.

Solving Warehouse Sprawl

A common issue is warehouse sprawl, where companies create separate warehouses for different teams, workload types, or usage patterns. For example, a marketing team might operate five different warehouses:

• MARKETING_REPORTS_L (idle 22 hours/day)

• MARKETING_ETL_XL (runs 2 hours nightly)

• MARKETING_DASHBOARD_L (idle 20 hours/day)

• MARKETING_ADHOC_M (sporadic usage)

• MARKETING_TEST_XS (barely used)

The Problem:

Managing many separate warehouses leads to unnecessary costs due to idle compute time and minimum billing increments. Even quick 5-second queries incur charges for a full minute of compute. Across hundreds of warehouses, this adds up to significant wasted spend each month.

The Adaptive Solution:

With Adaptive Compute, the marketing team can convert each of their five warehouses to Adaptive, and still maintain the same warehouse names for accounting purposes. The shared compute pool ensures resources are available when needed without incurring idle time charges across five warehouses.

Critical Unknowns: What's Still Unclear

While Adaptive Compute offers clear operational benefits, it’s still in private preview and several important details remain unclear. And these details matter, especially for organizations under tight cost or performance SLAs.

Limited Cost Control Mechanisms

Adaptive Compute optimizes for performance and simplicity, not cost reduction. If Target Statement Size is deprecated as planned[1], you'll have only one lever: the Warehouse Credit Limit. Without any settings to guide compute size per query, cost-saving strategies like pushing low-priority workloads to smaller compute are no longer possible.

Consider this scenario: Your team runs a daily report that takes 2 minutes on Small compute (2 credits/hour = 0.067 credits) or 1.5 minutes on Medium compute (4 credits/hour = 0.10 credits).

With standard warehouses, your team can choose Small to save 33% on credits, but Adaptive Compute might decide on Medium due to the speed improvement, increasing your spend regardless of your priorities.

What you lose:

• Ability to force non-critical workloads onto smaller compute to save money (e.g., running non-critical reports on X-Small warehouses)

• Option to set aggressive auto-suspend for development warehouses (e.g., setting 1-minute auto-suspend for ad-hoc warehouses)

• Control over running specific workloads on larger warehouses when deadlines require it

Cache Behavior Uncertainty

Snowflake’s local disk cache can significantly boost query performance, with warm-cache queries often running 5–10x faster than cold ones. Standard warehouses with longer auto-suspend settings are more likely to retain this cache between queries. But with Adaptive's shared compute model, cache behavior becomes unpredictable, potentially impacting both performance and costs.

Why this matters:

• Dashboard queries that typically benefit from warm cache might see inconsistent performance

• ETL pipelines that read the same base tables repeatedly could lose optimization opportunities

• Cost implications if queries that previously hit cache now require full table scans

Monitoring and Debugging Challenges

With standard warehouses, you can directly observe and control which warehouse a query runs on. With Adaptive Compute, the routing decisions happen behind the scenes, and it’s unclear what additional metadata Snowflake will surface around how those routing decisions are made, such as:

• What size compute ran each query and how routing logic behaved

• Whether the router over‑ or under‑sized compute for specific queries

• Whether queries are being queued due to credit limits or resource availability

Impact on Spend

Snowflake's launch materials for Adaptive Compute emphasize ease of use and performance, with no mention of cost savings[2, 3, 4]. This positioning suggests Adaptive Compute isn't designed to reduce spend, which is a critical consideration for organizations with FinOps accountability or tight budget constraints.

How Altimate AI's Auto Tune Delivers Control and Automation

For organizations that want automation without sacrificing cost control, Altimate AI's Warehouse Auto Tune offers a powerful alternative. It brings intelligent optimization to your existing standard warehouses to optimize size, idle time, and cluster configuration while providing even finer-grained control over speed versus savings.

Figure 3: Daily cost breakdown showing Auto Tune savings over time. Each bar displays actual spend (yellow) and realized savings (green) from warehouse suspension, scaling, and sizing optimizations. On July 23, 2025, Auto Tune saved between $34.85 and $45.38, with most savings from Warehouse Suspension and Scaling. Total realized savings during the selected period exceeded $1.1K.

Proactive Suspension and Cluster Scaling

Auto Tune minimizes idle time by scaling back clusters and suspending warehouses before Snowflake's native auto-suspend triggers. It continuously monitors activity and proactively reduces compute allocation during quiet periods.

Predictive Workload-Based Resizing

Auto Tune doesn't just react to current load – it predicts upcoming demand based on your historical workload patterns to determine when warehouses can be safely downsized without impacting performance.

How it works in practice:

Let's say you have a Large warehouse that processes heavy ETL jobs from 2-6 AM but runs mostly small queries the rest of the day. Auto Tune identifies this workload pattern and automatically:

• Maintains Large size during the 2-6 AM ETL window

• Downsizes to Medium during low-demand periods

• Returns to Large before the next ETL window

Granular Performance Safeguards

Auto Tune provides multiple mechanisms to protect critical workloads while optimizing costs:

1. Custom Schedule Blocks Custom schedule blocks define specific time windows where warehouses must maintain a certain size. Examples:

• Weekdays 9-10 AM: Keep EXECUTIVE_DASHBOARD_WH_XL at default size X-Large for C-suite daily reviews to ensure performance

• Sunday 2-5 AM: Keep ARCHIVE_ETL_WH downsized to Small while reprocessing past months of data to reduce costs

2. Real-Time Backoff Configuration

Set performance thresholds that automatically reverse downsizing decisions if query latency or queue times exceed acceptable levels. When triggered, the system reverts to default size and pauses auto-resizing for one hour.

Example Backoff Configurations for Different Workload Types:

Transparency and Control Over Decisions

Auto Tune provides full visibility into every optimization decision:

• Daily/weekly/monthly spend trends with savings clearly separated by agent

• Detailed logs explaining why each resizing decision was made

• Calendar view of all scheduled and completed optimization windows

Figure 4: Auto Tune history log showing recent warehouse actions taken. The agents resized warehouses dynamically based on query patterns, including downsizing from Large to Medium size to reduce cost without impacting performance.

Warehouse Eligibility and Smart Targeting

Auto Tune automatically evaluates each warehouse to determine if it's a good candidate for resizing optimization. Eligible warehouses typically have:

• Periods of low utilization with mainly small queries

• Predictable daily or weekly patterns

• Meaningful compute spend to optimize

This ensures resizing agents only focus on warehouses where they can deliver real value and help cut costs.

Making the Right Choice: Adaptive vs. Auto Tune

Here are a few pointers for choosing the right solution.

Choose Adaptive Compute When:

• Simplicity is paramount: You want to eliminate warehouse management entirely

• Workloads are unpredictable: Highly variable query patterns make sizing difficult

• Performance matters most: You're willing to pay more for consistent speed

• You're consolidating many underutilized warehouses: The reduction in idle time offsets other costs

Choose Auto Tune When:

• Cost control is critical: You have specific spending targets to hit

• Workloads follow patterns: Your queries have predictable daily/weekly cycles

• You need granular control: Different workloads require different cost/performance tradeoffs

• You want transparency: Detailed visibility into optimization decisions and savings

Hybrid Approach

Many organizations will benefit from using both solutions strategically. For example:

• Convert your DATA_SCIENCE_SANDBOX warehouse to Adaptive Compute since data scientists run unpredictable queries at random times

• Keep your PRODUCTION_ETL warehouse on standard with Auto Tune, reducing spend through predictable downsizing during known quiet periods

• Use Adaptive for all development/test warehouses to eliminate idle charges

• Apply Auto Tune to customer-facing warehouses where you need cost control with performance guarantees

Final Thoughts

Snowflake's Adaptive Compute and Altimate AI's Auto Tune represent two philosophies for warehouse optimization: total simplicity versus intelligent control. Adaptive Compute eliminates management overhead but sacrifices cost optimization levers. Auto Tune maintains the flexibility of standard warehouses while adding automation that typically reduces costs by 10-20%.

Start by analyzing your warehouse utilization patterns and cost targets. If predictable savings matter more than operational simplicity, Auto Tune is likely your best choice. If you're drowning in warehouse management complexity and can accept higher costs for simplicity, Adaptive Compute offers compelling benefits. For most organizations, a hybrid approach will deliver the best of both worlds.

If your team already has tools in place for analyzing warehouse usage patterns, monitoring cost targets, and automating warehouse optimization, you're all set. If not, we can help you with the analysis, optimization, and choosing between Adaptive and standard warehouses based on your needs. Reach out to us for a no-cost POC.

References

1. What's New: Faster Insights with Snowflake Standard Warehouse - Gen2 and Adaptive Compute. Snowflake Summit 2025 Session Catalog. June 3, 2025.

2. Snowflake Unveils Next Wave of Compute Innovations For Faster, More Efficient Warehouses and AI-Driven Data Governance. Snowflake press release, June 3, 2025.

3. Introducing Even Easier-to-Use Snowflake Adaptive Compute with Better Price/Performance. Snowflake Blog, June 3, 2025.

4. How Snowflake’s Adaptive Warehouse Is Revolutionizing Data Operations at Pfizer. Snowflake Builders Blog on Medium; Matt Massey, June 3, 2025.

Deliver more performance and cost savings with Altimate AI's cutting-edge AI teammates. These intelligent teammates come pre-loaded with the insights discussed in this article, enabling you to implement our recommendations across warehouses, tables, and millions of queries effortlessly. Ready to see it in action? Request a recorded demo by sending us a chat message) and discover how AI teammates can transform your data teams.

Introduction

We’re excited to announce a major new capability in the dbt Power User VSCode extension: an embedded Model Context Protocol (MCP) server. This MCP server is now built directly into the extension, acting as a bridge between AI-powered development tools and your dbt project. The motivation behind implementing an embedded MCP server is to enable seamless communication between AI assistants (like those in Cursor IDE) and dbt, eliminating the friction of context switching. In practical terms, this means your AI coding assistant can query your dbt project’s schema, compile models, run tests, and more – all through a standardized protocol – leading to faster development workflows and greater efficiency for dbt developers.

By leveraging the MCP server, dbt Power User can expose rich project context and operations to AI agents in real-time. This empowers data engineers to automate repetitive tasks (like looking up column definitions or running model builds) and accelerates development. The embedded MCP server unlocks new AI-driven workflows for dbt, from intelligent autocompletion to on-demand documentation generation, all while keeping the interaction localized and secure within your development environment.

Technical Architecture and Cursor IDE Integration

At a high level, the MCP server runs as a lightweight web service within the dbt Power User extension environment. The server spins up automatically when you open a dbt project (if you have enabled the MCP feature) and listens on a local port. It selects an available port dynamically at launch to avoid conflicts, and registers itself with Cursor by updating Cursor’s MCP configuration. Once running, the Cursor IDE detects the MCP server and establishes a connection so its AI assistant can invoke dbt-related “tools” exposed by the server.

Server Components and Design: The MCP server architecture is composed of a few key components working in tandem:

• Server Core: the core HTTP server (built with the MCP SDK) that handles incoming requests and responses. It defines the MCP endpoints and protocol handling (e.g. registering available tools, managing sessions, etc.). The server core orchestrates requests from Cursor’s AI and ensures each is routed to the appropriate handler in the extension.

• Tool Registry: a registry of tools – individual operations or actions related to dbt. Each tool has a name and a function implementing its behavior. When the server starts, it registers a suite of tools (detailed in the next section) that allow the AI to perform various dbt tasks. The Tool Registry essentially maps MCP requests to the extension’s internal methods or dbt CLI calls. It also defines metadata for each tool (like input parameters and descriptions) that Cursor’s AI can utilize to formulate requests.

• SSE Transport: the server uses Server-Sent Events (SSE) for real-time communication back to the client. After Cursor connects to the MCP server, it maintains an open SSE channel. When a tool is executed, results (and any interim output or logs) are sent as a stream of events over this channel to Cursor. This means the AI can receive data or status updates from long-running tasks in real time, without polling. For example, if you execute a model run via the AI, the MCP server can stream back logs or a success/failure message as soon as it’s available. Using SSE for output ensures a responsive, event-driven interaction between the extension and the AI assistant.

Architecture of the MCP Server integration with Cursor.

The diagram shows the Cursor IDE’s AI assistant interacting with the MCP server via tool requests. The MCP server (within the dbt Power User extension) processes these requests using its registered tools. It interacts with in-memory dbt project artifacts for quick context and, when needed, invokes the dbt Integration module (which in turn may query the data warehouse) to fulfill the request. Results and logs are returned back to the AI client in real-time via SSE.

Communication Flow: The end-to-end flow of a typical interaction involves several steps that connect the AI to your dbt project:

1. AI Initiates a Tool Request: Within Cursor, the AI assistant decides to call a tool (for example, “get the SQL definition of model X” or “run tests for model Y”) and sends a request through the MCP protocol.

2. MCP Server Receives the Request: The server core matches the incoming request to one of the registered tools in its registry and hands off execution to that tool’s handler.

3. Tool Execution: The tool’s handler runs inside the extension process. It may fetch information from the in-memory dbt artifacts (e.g. to get a model’s definition or lineage) or execute a dbt operation. The handler leverages the extension’s existing capabilities – for example, calling a function in the extension that wraps dbt operations to perform the task.

4. In-Memory Artifact Access: If the tool needs to read project structure (models, sources, tests, etc.), it queries the up-to-date in-memory representation of the dbt project (more on this in the next section). This avoids expensive disk or CLI operations for retrieving metadata. If the tool needs to run SQL or dbt operations, it might invoke the dbt integration module and use the database connections configured in the dbt profile.

In essence, the MCP server turns the dbt extension into a real-time API for your dbt project. Cursor’s AI no longer operates in a vacuum – it can actively query your project’s state and perform operations, treating the dbt project as an interactive context it can reason about. All of this happens locally within your development environment, leveraging the work the extension is already doing to maintain dbt context.

In-Memory dbt Artifacts Lifecycle and Performance Optimization

One of the standout benefits of the MCP server is that it can serve data to the AI assistant directly from memory, without constantly hitting your filesystem or database. The dbt Power User extension maintains an in-memory representation of your project’s artifacts as you work. This includes the parsed project manifest (models, sources, tests, exposures, etc. along with their relationships), and metadata like the active target and project name. By tapping into these in-memory artifacts, the MCP server can answer many questions instantly and run operations more efficiently.

Creation and Initialization: When you open a dbt project in VS Code, the extension immediately initializes and parses the project. It loads manifest information (by invoking dbt’s parse operation under the hood and translating it to its own in-memory representation through its own parser) to build a graph of models and references in memory. This means the extension knows about all your model definitions, sources, tests, and their lineage as soon as the project is ready. At the moment the MCP server starts (which typically coincides with project load or when you enable the feature), it attaches to this ready-to-use project context. The server core may wait until the dbt project is fully initialized and then mark its tools as available for the AI. Essentially, the server boots up after the dbt context is prepared, ensuring the AI always sees a consistent project state.

Live Updates: As you modify files or the project state changes, the in-memory artifacts are updated incrementally. The extension listens to file system events and dbt execution outcomes – for example, if you edit a model file, it can re-parse that model and update the manifest graph in memory. Likewise, running dbt deps will refresh the list of installed packages in memory. The MCP server subscribes to these internal events. If the manifest is updated or a new run is completed, the server can invalidate or refresh relevant cached data. This way, tools like “get children models of X” will always use the latest information. The lifecycle is managed such that any time the dbt project changes, the in-memory state and the MCP server’s view of it remain in sync. This is much faster than re-running dbt commands for each query and ensures the AI doesn’t get stale data.

Efficient Querying: Because the data is in memory, tools that retrieve info (e.g. listing models, showing a model’s SQL, computing lineage) execute extremely quickly. There’s no need to spawn a subprocess or read from disk for these – the extension can simply look up the Python/TypeScript object representing that model. Even complex lineage traversals or dependency graphs can be resolved by the extension’s code using pre-computed relationships. This optimization is critical for performance: it allows the AI to ask a series of detailed questions about your project without incurring heavy overhead. For example, an AI reasoning about why a model’s test failed can rapidly pull the model’s SQL, its parent models, and the test definition via multiple MCP tool calls in seconds, whereas doing this via CLI repeatedly would be far slower.

Cleanup and Lifecycle End: The MCP server’s lifetime is tied to your VS Code session and project. If you close the project or disable the extension, the server is stopped and the in-memory artifacts are freed. The implementation ensures that sockets are closed and event listeners are disposed to avoid resource leaks. Notably, the server core exposes a clean dispose() method that the extension calls on shutdown, which shuts down the SSE stream and prevents further tool calls.

Overall, the approach of leveraging in-memory artifacts means that the MCP server can provide a fast, rich context to the AI. It capitalizes on the work already being done by the dbt Power User extension’s internal DBT parser and project tracker. This not only speeds up AI interactions but also avoids unnecessary load on your data warehouse – the AI can get a lot of information without executing a single query, unless absolutely necessary for the task.

Capabilities Unlocked: Tools Provided by the MCP Server

The MCP server exposes a robust set of tools that the AI assistant can use. These tools cover most day-to-day dbt tasks and are grouped into a few categories for clarity. Below is a breakdown of major capabilities and examples of tools in each category:

Project & Environment Management Tools

• Project Information: Usually the AI will fetch all the dbt projects and its roots first, to know on which project it should work. The AI will use the root of the project as a parameter to the other tools. For instance, a tool GET_PROJECT_NAME returns the dbt project name for the given root, and GET_SELECTED_TARGET returns which target/profile is active. This is useful for the AI to contextually understand which environment it’s working with (production or development).

Model and Source Exploration Tools

• Model SQL and Schema: Tools in this category allow the AI to fetch the content or properties of dbt models and sources. For example, a COMPILE_QUERY tool can return the raw SQL of a model file, and a GET_COLUMNS_OF_SOURCE could return details about a source (like database/schema and columns as defined in YAML). With these, the AI can, say, read the logic of a model to help debug or suggest changes.

• Lineage and Dependency Queries: We provide tools like GET_CHILDREN_MODELS and GET_PARENT_MODELS, which return the immediate downstream and upstream models of a given model. The AI can use these to understand the dependency graph – for instance, finding what will be impacted if a model is changed or which upstream data sources feed into it. Internally, these tools leverage the manifest graph in memory to quickly compute the relationships. Additionally, there are count-based queries (like a tool to count the number of models depending on a source) to aid impact analysis.

• Column Lineage and Documentation: (Upcoming) The extension already has features for column-level lineage and documentation. Through MCP, we intend to expose tools so the AI can answer questions like “which models use column X from source Y” or even retrieve documentation strings for a model or column. This will bridge the gap between documentation and code by letting the AI pull in documented context as needed.

SQL Compilation & Execution Tools

• Compile SQL: The AI might want to see how dbt renders a model’s SQL with Jinja templating. The COMPILE_QUERY tool takes a query (and optionally the model it is derived from) and returns the compiled SQL, exactly as dbt would generate it. This is extremely useful for checking logic or debugging macros – the AI can get the SQL that would run on the warehouse, without actually running it. It uses dbt’s compilation engine (via the extension’s interface to dbt Core) under the hood.

• Execute Arbitrary SQL: Sometimes the AI will need to run a query to answer a question (for example, “preview the first 10 rows of model X”). The EXECUTE_SQL tool (with a configurable row limit) allows running a SQL query against the project’s data warehouse connection and returns results, ensuring we don’t pull massive datasets by accident. The AI could use this to validate assumptions or provide sample data to the user. Users must explicitly permit this feature, as executing SQL will share data with the AI.

• Run dbt Models: For full pipeline execution, we offer tools to run or build models. The RUN_MODEL tool triggers dbt run for a specific model (and optionally its dependencies), while BUILD_MODEL can run the model and its tests (equivalent to dbt build for that model). There’s also a BUILD_PROJECT for running the entire project (dbt build). These commands execute in the background through the extension’s job runner. The AI can thus initiate a model run or a full project build from within the conversation – for example, “Re-run the revenue model to see if the issue is fixed.”

Testing and Validation Tools

• Execute Tests: The server provides a RUN_TEST tool to execute a specific dbt test by name, as well as RUN_MODEL_TEST to run all tests associated with a particular model. This allows the AI to verify data quality or detect if recent changes broke any tests. For instance, after modifying a model, the AI might call RUN_MODEL_TEST to ensure all tests still pass and then report the results. Test results or failures will be returned via the event stream.

• SQL Validation: (Planned) Even without running on the warehouse, the extension can do a lightweight SQL parse/validation. A tool is available to validate a SQL query (checking for syntax errors, missing refs, etc.) without executing it. This is similar to the extension’s existing SQL Lint feature, now accessible to the AI. It can prevent the AI from running a bad query by first validating it and catching errors.

• Performance Analysis: (Planned) We aim to add tools that use dbt’s compilation stats or query planner insights to help the AI assist in performance tuning. For example, a tool to show the compiled query explain plan (if the warehouse supports EXPLAIN) or to estimate the cost of a query. Such tools would let the AI caution the user if a query might be expensive or if a model lacks proper pruning filters, etc.

Package Management and Repository Tools

• Install Packages: Managing dbt packages is made easier with tools like INSTALL_DEPS which runs dbt deps to install the packages listed in packages.yml. There’s also an ADD_DBT_PACKAGES tool that can add new packages on the fly. For example, if the AI suggests using a package (like dbt-utils), it could call ADD_DBT_PACKAGES with the package name and version – the extension will add it to packages.yml and run dbt deps internally to install it, returning the result. This showcases agentic behavior: the AI can not only suggest but also take action to modify your project structure (with your permission).

• Upgrade dbt Version: Although not explicitly a tool in this release, the infrastructure could allow the AI to assist in upgrading the dbt version or dependencies. For example, by reading requirements.txt or checking the installed dbt version, and then running pip installs. This would likely be added as a safe tool in future once we ensure such actions are gated behind user approval.

• Git Operations: While primarily focused on dbt tasks, the larger vision includes enabling some project-level Git interactions via MCP (for example, checking out a new branch for a fix, or opening a pull request). The current server does not directly include Git tools, but it lays the groundwork for integrating such capabilities alongside dbt tasks, so the AI could orchestrate end-to-end workflows (code change -> run model -> test -> commit).

All these tools are defined in the extension and registered with the MCP server at startup. When Cursor’s AI is formulating a plan to help you, it queries the MCP server for an index of available tools (the MCP protocol allows the client to discover tools and their input schema). The AI sees something like “I have a tool called get_children_models that needs a table name” and it can then decide to use it if the user asks a question about model dependencies.

Under the Hood: Port Management, Error Handling, and Security

Building an embedded server inside an IDE extension comes with its own engineering challenges. We put significant effort into making sure the MCP server runs smoothly without disrupting your workflow.

Port Management: Since the MCP server is local, it needs to choose a port to listen on. We default to letting the OS assign an open port (by binding to port 0, which yields an ephemeral port). Once the server is listening, the extension logs the chosen port. If for some reason the port is not reachable or Cursor cannot connect, the extension will surface an error message. We also handle the case where the server might need to restart (for example, if you re-initialize the project) by cleaning up the old server and starting a new one potentially on a new port. The extension ensures there is only one MCP server instance per workspace to avoid port conflicts. In the future, we may allow a fixed port configuration if users want to expose the server to other tools, but in this beta it’s fully managed and isolated.

Error Handling and Resilience: The MCP server includes robust error handling so that failures in tools do not crash the server or hang the AI. Each tool execution is wrapped in try/catch logic. If a tool throws an exception (say, dbt fails to compile a query due to a syntax error), the error is caught and sent back as an error event to the AI, and the server remains up for subsequent requests. We’ve instrumented comprehensive logging via the extension’s output channel (the “dbt Power User” output panel in VS Code) to record any exceptions or problematic requests. This makes it easier to debug if something goes wrong – both for us as developers and for users who can check the logs. Overall, the server is designed to be long-running and robust, recovering gracefully from errors. In our tests, issues like missing profiles or syntax errors are caught and reported back to the AI without terminating the server.

Security Measures: Since the MCP server allows automated actions on your project, we have restricted its accessibility to ensure safety. The server binds only to localhost and is not accessible from external machines. This means only processes on your computer (like the Cursor IDE) can communicate with it, mitigating risk of any outside interference. Additionally, the server enforces that requests conform to the expected MCP schema – it will reject any malformed requests or unknown tool calls rather than executing them arbitrarily. We also incorporated a simple authentication check in this beta: the MCP server feature will only start if certain preconditions are met (for instance, verifying that you have a valid Altimate API key configured, which is used to enable AI features in the extension). This is mainly to gate the feature during beta and ensure usage can be monitored and supported. In the future, we plan to add more sophisticated security, such as an authentication token for the MCP connection or user consent prompts in VS Code before executing potentially destructive actions (like modifying files or running DDL statements). It’s important to note that the AI agent in Cursor will only use tools in ways you allow – you’re always in control and can disable or limit certain tools if desired. We recommend keeping the extension and Cursor up-to-date to get the latest security improvements. As the community and usage of MCP servers grow, we’ll continue to harden the server (for example, adding rate limiting or sandboxing if the scenario requires it).

Resource Usage: Running an MCP server is lightweight. It piggybacks on the extension’s process and uses minimal additional memory (mostly just for the HTTP server overhead). The in-memory artifact store is already in place for the extension’s normal operation. We’ve observed negligible impact on VS Code’s performance when the server is idle. During heavy use (like a lot of model runs triggered by AI), there is some CPU and memory use, but it’s equivalent to you running those dbt commands manually. When idle, the server simply holds open a socket for Cursor. If needed, you can always stop the server by disabling the feature in settings, which immediately frees the port and associated resources.

Future Enhancements: In upcoming releases, we plan to enhance both the capabilities and the internals of the MCP server. On the capability side, we’re looking at adding more tools (as mentioned, possibly Git operations, documentation queries, etc.) and integrating deeper with other systems (imagine the AI triggering an Airflow DAG after a successful dbt run, via a future Airflow MCP server). We’re also considering a plugin mechanism where users could write custom MCP tools for their own macros or operations and have the server load them. Finally, as the MCP protocol evolves (it’s an open standard in active development), we will adapt our server to remain compliant and take advantage of new features. For example, Anthropic’s Claude models recently embraced MCP natively, and we expect more LLMs to follow – our goal is to have the dbt MCP server work with any AI agent that speaks MCP, not just Cursor’s.

Conclusion: Our Vision Toward an AI-Integrated Data Platform

The introduction of the MCP server in the dbt Power User extension is a significant step toward an AI-integrated data development platform. We’ve turned the VS Code + dbt environment into an AI-accessible service, enabling smarter assistance and automation. This capability is not an isolated feature – it’s part of our broader vision to integrate AI with all facets of the modern data stack. In the near future, you can expect similar integrations with Snowflake and Postgres (for example, an AI agent that can profile or manipulate your warehouse data), with Airflow (an AI that can inspect DAGs or trigger workflows), and more. Our DataPilot platform is evolving into an ecosystem of AI “teammates” that collaborate across tools. The dbt MCP server is one of these teammates, focused on analytics engineering, and it will collaborate with others (imagine an AI assistant that understands your data warehouse, your pipelines, and your dbt models collectively).

We invite you to try out the MCP server. Since it’s a beta release, we greatly value your feedback. Join our community Slack channel (#tools-dbt-power-user on dbt Slack) to share your experiences or ask any questions. . If you need help setting it up or want to explore how this could fit into your team’s workflow, please reach out to us . We’re also eager to collaborate with teams interested in pushing the boundaries of AI in data engineering. Whether you have ideas for new MCP tools or want to integrate our platform with your own, let’s start a conversation. Together, let’s redefine what a “power user” can do by combining the strengths of dbt with the intelligence of AI. This launch is just the beginning, and we’re excited to build a more connected, intelligent data platform with your input.

In the past five years, Snowflake has disrupted the database, data storage, and analytics industry by massively simplifying the entire data and analytics landscape. However, a new wave of innovation is horizon - Artificial Intelligence and AI Teammates.

This article won't discuss Generative AI, which is (shockingly) already old news, but how AI Teammates working with AI Agents can collaborate alongside your (human) Snowflake System Administrator to help maximize performance and optimize spending on your Snowflake system.

After reading this article, you’ll realize just how powerful this technology is and what a potentially massive game-changer it presents.

What is an Agentic AI?

Unless you've been living under a rock for the past two years, you'll be well acquainted with Chat-GPT (other AI Chatbots are available), but these are simply passive assistants that quietly wait for your questions and generate text.

That's not denigrate the impact of Large Language Models (LLMs), as a recent McKinsey Report indicates that generative AI will add up to $4 trillion to the global economy with up to 70% less employee work required. Indeed, Artificial Intelligence is well on the way to being integrated into almost every aspect of Data Warehousing.

However, unlike Chat GPT that passively waits for user requests, an AI Teammate works alongside existing staff to monitor query patterns, identify potential issues and offer actionable tuning suggestions. They act as an artificially intelligent advisor, perhaps using machine learning to refine recommendations based on administrator feedback.

AI teammates work autonomously to recommend real-time system configuration and processing adjustments, reacting to events and recommending remedial actions to improve Snowflake customer experience or optimize cost.

Whereas an AI Teammate provides a consulting role, communicating with System Admins in natural language, AI Agents work autonomously to handle the low-level monitoring, analyzing query patterns and resource usage. In many ways, AI Agents are the back-room workers, providing data to be analyzed and interpreted by the AI Teammates which in turn support the Snowflake Admins.

Taken together, these are often referred to as Agentic AI.

Key Features of “AI Teammates” and “Agentic AI”

It's all well and good using jargon like “AI Agents” - but what makes a solution “Agentic” as opposed to, (for example), a Python script that uses machine learning to identify and alert users of potential problems.

Here's a list of the key features an “Agentic AI” system.

• Dynamic Problem Solving: Instead of relying on fixed scripts, in exactly the same way Chat-GPT can currently generate and execute python code, AI agents will generate source code and scripts to analyze data and solve problems . AI Teammates work just like a real person — assessing the situation, deciding what needs to be done, and then executing the plan.

• Reduced Technical Skills: IT automation typically needs technical experts (DBAs and System Admins) to configure and maintain it through scripting. In contrast, Agentic AI allows anyone, including non-coders, to instruct the system in natural language. For example, a user could give the AI an objective like “reduce warehouse costs by reducing idle time” and the AI would figure out the best way to do that, without needing predefined scripting.

• Autonomous Operation: Agentic AI doesn’t need constant supervision. Once given a goal, it will continuously monitor for opportunities to act and take the next steps without waiting for further instructions.

• Real-World Interaction: These AI systems will connect with external tools and databases, pulling in real-time information, and even executing actions like sending emails or making configuration changes. For example, an AI Teammate tasked with reducing Snowflake Virtual Warehouse costs could automatically resize warehouses in real-time to maintain a balance between cost and performance.

• Efficiency & Adaptability: These systems can handle complex workflows by breaking down tasks into smaller steps, calling different tools, and even calling upon AI Agents to handle specific tasks. As AI Teammates are largely autonomous, they are highly efficient needing little human interaction and constantly adapt to changing circumstances.

How can AI Teammates work on Snowflake?

AI Teammates can play a vital role in optimizing a complex Snowflake Data Warehouse deployment by automating performance tuning tasks. Here are some practical examples of how Agentic AI could be deployed:

Tuning Query Performance

• Identifying Slow Queries: An AI Teammate could continuously monitor query elapsed times against an objective, prioritizing performance over cost. It would then suggest optimizations, such as improving partition pruning or moving a given workload to a different warehouse size.

• Query Rewriting: If a query can be rewritten for better performance (e.g., using specific functions like LIMIT to reduce data scanning), the AI Teammate could automatically suggest or implement these changes after analyzing the query execution plan. Of course, this may need a “human in the loop” to verify, test, and deploy the changed code.

Virtual Warehouse Management

The screenshot below illustrates how AI Agents deployed by Altimate.AI automatically monitor and tune system configuration to identify potential cost savings and implement configuration changes to eliminate waste.

• Warehouse Configuration: AI Teammates can continuously monitor virtual warehouse usage patterns and (for example), automatically adjust the AUTO_SUSPEND, WAREHOUSE SIZE or CLUSTER COUNT settings to optimize efficiency. For instance, if a warehouse remains idle longer than expected, the AI Teammate could reduce the auto-suspend timeout or automatically resize warehouses during off-peak hours to reduce costs.

• New Feature deployment: Agentic AI could automatically detect and recommend potential improvements using new Snowflake features. For example, an AI Teammate tasked with maintaining strict performance targets could recommend deploying Snowflake Query Acceleration or Search Optimization Service to improve query performance. This reduces the need for System Administrators to constantly research, understand, and test new Snowflake features, reducing the dependency upon highly skilled (and expensive) resources.

Managing Snowflake Cluster Keys

• Identifying Clustering Opportunities: Snowflake Cluster Keys can significantly impact Snowflake query performance, but deployment is more an art than a science. An AI Teammate could automatically analyze query patterns and data distribution to recommend adding, modifying, or removing cluster keys. For instance, if certain columns are frequently used in filter conditions but are not well clustered, the system could recommend creating cluster keys on those columns to speed up query performance.

• Monitoring Clustering Efficiency: As query workloads change over time, once appropriate cluster keys may become less efficient over time. For example, if a data pipeline changes, adds frequent update operations, or changes the data cardinality, this could make clustering less cost-effective. An AI Teammate could monitor the situation and recommend or even automatically deploy changes to improve query performance and optimize cost.

Dynamic Workload Allocation

The diagram above illustrates the overall challenge faced by System Administrators. Despite constantly increasing system workloads, we need to balance the conflicting needs to maximize system throughput (for example loading and transforming data), with maximizing end-user query performance (for example, delivering query results to analytics), and optimizing spend. Agentic AI can help with this challenge by automating system monitoring and adjusting workloads as needed.

The diagram below illustrates a typical mixed workload frequently found on Snowflake MEDIUM size warehouses.

While it's not obvious, research into Snowflake Workload Segmentation demonstrates that the above workload would be more efficiently deployed over two or four warehouses, separating workloads by size and frequency. This has the potential to improve system performance and cost efficiency significantly.

Most customers deploy workload separation by team, which leads to poor utilization and high spending. AI Teammates constantly monitor jobs, recommending or even moving them to an appropriate warehouse size to achieve the best balance of throughput and cost efficiency.

For example, high-priority analytical workloads could be processed on larger warehouses while smaller, less important workloads could be shifted to more cost-effective, smaller warehouses configured for batch processing and prioritizing throughput and spend over individual query performance.

Proactive Snowflake Monitoring and Alerts

The screenshot below illustrates automated Snowflake cost monitoring across all sectors, including Warehouse computing, Storage, and Snowflake Serverless processing. AI agents can build upon basic monitoring to alert administrators proactively or take action.

Anomaly Detection: AI teammates can track the system’s performance and detect anomalies such as sudden spikes in query times, sudden increases in storage usage, or unexpected credit consumption.

Once detected, the system could either notify System Admins or attempt to automatically resolve the issue (e.g., by suggesting or implementing query optimizations).

In one case, a major Snowflake customer in the UK accidentally spent over $120,000 due to a simple manual configuration change that took a week to identify and resolve, which could have been automatically identified by an AI Teammate within seconds, potentially avoiding a significant bill.

End-to-End ELT Optimization

According to Snowflake, automated data transformation tasks account for around 80% of customer spending. However, tuning increasingly complex pipelines can be remarkably resource-intensive, often requiring an army of highly skilled data engineers.

The diagram below illustrates the challenge with a simple transformation pipeline with potentially conflicting delivery timelines. The challenge comes from optimizing data delivery within the required SLA, but avoiding unnecessary processing.

In one example, a Snowflake customer in Munich, Germany was able to reduce overall system cost by 50% simply by suspending real-time analytics during out of hours. In short, if nobody is consuming the data, there's no need to produce it.

AI Teammates can help in the following ways:

• Transformations and Load Efficiency: AI Teammates can analyze transformation pipelines, looking for inefficiencies or bottlenecks. They can recommend breaking down complex SQL statements into smaller tasks or parallelizing operations to improve load and transformation speeds.

• Dynamic Table Optimization: Snowflake's Dynamic Tables massively simplify and streamline transformation pipelines with opportunities to dynamically change processing frequency and warehouse deployment. AI Teammates can automatically monitor both workload size and data consumption and adjust the pipeline configuration to balance and optimize performance and cost.

• The diagram below illustrates one potential tuning step in which the ETL pipeline can be automatically adjusted to match changing the frequency at which data must be refreshed. In this case, a critical data feed is refreshed every five minutes. Once deployed, an AI Teammate can automatically adjust the refresh frequency for example reducing the frequency during out-of-hours to optimize cost.

Summary & Conclusion:

AI Teammates in a Snowflake environment can act as intelligent assistants that continuously monitor and optimize various aspects of a Snowflake deployment.

They can provide actionable recommendations and potentially even implement changes themselves—whether it’s tuning queries, managing warehouse resources, or optimizing storage usage. This can lead to improved performance, reduced costs, and more efficient use of Snowflake’s powerful features, all with minimal manual intervention from human operators.

In conclusion, having spent nearly 10 years as a Senior Solution Architect much of that time working for Snowflake UK, I’ve a fair understanding of the huge challenges that Snowflake System Administrators face. I firmly believe that AI Teammates are indeed the secret to efficient Snowflake cost and performance management.

However, unlike the pessimistic view of Artificial Intelligence which warns about the machines taking over, the fact that AI Teammates work collaboratively with DBAs and Admins mean they’ll simply make an almost impossible task way more effective.

Deliver more performance and cost savings with Altimate AI's cutting-edge AI teammates. These intelligent teammates come pre-loaded with the insights discussed in this article, enabling you to implement our recommendations across millions of queries and tables effortlessly. Ready to see it in action? Request a recorded demo by just sending a chat message (here) and discover how AI teammates can transform your data teams.

Once upon a time, an analyst in the trouble

Imagine Leila, an analyst, asked to build a monitoring dashboard for finance - given a tight deadline. While searching for a useful dataset, she comes across a table called “sh_monthly_revenue” that seems perfect based on its name and the columns it contains. However, there’s little documentation available, and the person who created it has already left the company. With time ticking, she makes a few assumptions based on the column names and dives into the analysis.

Weeks later, after presenting the dashboard findings, one of the business stakeholders points out that the data does not match the numbers in the financial reports at all. It turned out that the dataset she used wasn’t appropriate since it only covered revenue from a subset of customers, leading to incorrect assumptions and, ultimately, flawed conclusions. This scenario didn’t just waste time and resources - it also damaged Leila’s credibility and may have led to misguided business decisions.

If this sounds familiar, you’re not alone. Many data professionals - whether analysts, data scientists, or engineers - have found themselves in a similar situation. Poor documentation doesn’t just slow down work; it can lead to significant errors with far-reaching consequences. On the other hand, writing and maintaining comprehensive documentation can be a tedious and time-consuming task.

In this post, we’ll explore best practices for writing better documentation for your data models and how GenAI can help make the process less tedious and time-consuming.

Good documentation matters for many reasons

It’s easy to overlook documentation when you're in the final stages of building a data model, eager to push your work into production. We've all been there: you’ve just completed a model you’re proud of, and you’re ready to ship it out the door. But then comes the task of writing documentation (and possibly adding a few data tests). This critical step often gets rushed or skipped altogether, yet there are compelling reasons to take the time to do it right:

• Reduces time spent on data discovery

• Reduces dependency on tribal knowledge

• Reduces Ad-hoc questions you get from data consumers

• Your org makes you follow governance and best practices guidelines ^_^

Understanding why documentation matters is just the first step. The real challenge lies in consistently applying best practices to ensure your documentation is both comprehensive and usable. Let’s explore how you can achieve this with a few best practice tips.

Best practices for writing excellent data documentation

Creating effective data documentation isn’t just about listing columns and definitions - it’s about ensuring that anyone who interacts with your data can understand and use it correctly, even if they have no prior context. Here are some best practices to help you design documentation that’s both comprehensive and user-friendly:

“Always Consider the Reader’s Perspective”

When documenting a data model, it’s easy to fall into the trap of writing from your perspective, assuming that others will inherently understand the nuances you do. Instead, try to put yourself in the shoes of someone without prior knowledge of the dataset. When it comes to writing the docs for your model, you can follow these guidelines:

• Clearly outline the Purpose and Utility: Define the key questions this dataset is designed to answer and the specific use cases it supports. This context helps users understand the data’s intent and how best to leverage it.

• Identify Caveats: Highlight any limitations or potential pitfalls associated with the data, such as missing information, known inaccuracies, or situations where the data might be misleading. These warnings are crucial for preventing misuse.

• Specify Granularity: Detail the level of detail provided by the data, whether it’s aggregated daily, weekly, or monthly, or at the transaction level or summarized. Understanding the granularity helps users determine the dataset’s appropriateness for their needs. In the case of columns, include information about primary keys, etc.

• Explain Contextual Background: Provide the reasoning behind the table’s creation, including any business processes or decisions that shaped its structure. This background helps users appreciate the dataset’s origins and relevance within the broader business context.

• Highlight Key Dimensions and Metrics: Point out the most important dimensions (such as date, location, and product) and metrics (like sales, revenue, and click-through rate) within the dataset. Explain how these elements interact and any specific considerations for their use. In the case of categorical columns, you can possibly include details about accepted values.

• Optimize for AI and Semantic Search: As AI tools become more integrated into data management, ensure your documentation includes relevant business terms, acronyms, and synonyms associated with the data. This optimization enhances discovery and usability, particularly for AI-driven tools like semantic search engines or analytics agents.

Here is an example of how this could look like for our model from above:

Make documentation less painful by using AI

Despite your best efforts, documentation can still be time-consuming and tedious, especially when balancing tight deadlines and complex data models. This is where generative AI (GenAI) comes into play, turning what once felt like a chore into a largely automated and streamlined process.

Tools like the Power User for dbt (DataPilot) are designed to handle the heavy lifting of documentation, automatically generating detailed table and column descriptions from your existing data models. These tools use the metadata context available in your environment automatically like schema information, column lineage etc. and write so good documentation that has surprised me many times. Some of the key features include:

• Effortlessly Document Hundreds of Columns:
  Have 900 columns to document? No problem. With bulk generation, you can quickly create reliable descriptions for every column, allowing you to focus your efforts on refining and curating the details that matter most.

• Tailored Documentation for Different Audiences:
  The extension allows you to generate documentation for different user personas. Use technical language for your intermediate layer tables and a more business-friendly tone for end-user-facing tables. This ensures that each audience gets the information they need in the most effective format.

• Seamless Integration with Your Workflow:
  The tool integrates directly with your dbt YAML schemas, injecting newly generated documentation into existing schemas or updating current documentation with just a few clicks. All this happens directly within the model SQL file, so you can avoid those days when your IDE tabs blow up like confetti due to the number of open file tabs.

More details can be found here

Overall, documentation is an important aspect of analytics engineering and it’s essential that we follow best practices, otherwise, we run the risk of writing documentation for namesake or not writing it at all! GenAI based automation is going to be the major boon in this area, it has its kinks but early signs are very promising making me quite excited for the future!

In my previous article on Dynamic Tables, I explained their purpose, to help simplify building transformation pipelines for either batch or real time data feeds in Snowflake. In this article, I'll explain what they cost to run and how Artificial Intelligent Agents can be used to automatically monitor, alert and even deploy remedial action to control cost.

Skeptical? Read on to see how to reduce the annual cost of running three Dynamic Tables from $6.3m to just $26,280.

Dynamic Tables Cost Components

It may not be obvious but Dynamic Tables include multiple cost elements including:

• Cloud Compute: Dynamic Tables are automatically refreshed on a repeating sequence depending upon the TARGET_LAG. For example, once per minute or once per hour.

• Data Storage: Like any table, Dynamic Tables store data, and this will be billed at £23 per terabyte per month.

• Warehouse Cost: Every time the Dynamic Table is refreshed it (currently) needs a Virtual Warehouse to execute the SQL, and this is billed on a per-second basis with a minimum of 60 seconds.

Let's consider each of these elements and it's potential impact upon the overall cost of ownership.

Cloud Compute Cost

This is by far the least expensive of the three cost elements and results from the automatic scheduling executed in the Cloud Services layer. Determined by the TARGET_LAG, the DT "wakes up" and checks if there any changes in the upstream table(s). This is likely to be thousands of a dollar each time, and for 99% of customers is free, unless it becomes more than 10% of your daily warehouse cost.

In short, don't worry about this.

Dynamic Table Storage Cost

While you need to store data in Snowflake (and therefore you'll be billed a passthru cost of £23 per terabyte), it's not clear that Dynamic Tables add to the storage cost. Consider the diagram below.

In the above diagram tables T1 and T2 are default tables, wheres T3, T4 and T5 are dynamic tables. If we assume T1 and T2 hold 1TB of data storage, and the transformation to T3 and T4 is simply to perform a basic data quality correction (for example, setting in incorrect SALES_TYPE column to NULL when the values are incorrect), then tables T3 and T4 will equally hold around 1TB of storage.

If the results of T3 and T4 are then combined using a JOIN operation into a single table T5 - ready for querying, this adds another 2TB to storage.

In total we store 6TBs of storage.

However, if we were not using Dynamic Tables, then tables T1 and T2 could be deleted down each time the new entries where processed. Likewise, once the process to join the tables and produce T5 was complete, tables T3 and T4 could be deleted.

This means, a solution without Dynamic Tables could deploy the same functionality in 2TBs compared to 6 TBs.

Data storage is a relatively minor cost on most Snowflake systems, but it's a cost worth understanding.

Dynamic Tables Warehouse Cost

This is almost certainly the greatest cost associated with Dynamic Tables, although unlike the data storage example above, there's no additional cost when compared to a transformation pipeline without DTs.

Effectively, each time a Dynamic Table triggers a refresh (provided there's changed data in the upstream tables), Snowflake executes the associated SQL on a Virtual Warehouse. However, even this has a potential gotcha. Consider again the diagram below.

The diagram above illustrates the same transformation pipeline, with two source tables and three dynamic tables. Notice that two of the tables are marked as Downstream? This means their refresh rate is determined by the downstream table - in this case a table refreshed every minute.

Let's assume new data is automatically inserted into the Sales table using Snowpipe which runs on a 24x7 basis. The Stores data is however, refreshed once per day.

Can you see what might happen to the cost profile of this simple transformation pipeline?

As new sales are automatically loaded into the Sales table (T1), Dynamic Tables T3 and T5 are automatically refreshed based upon the one minute TARGET_LAG.

Let's assume all Dynamic Tables were initially deployed on an XSMALL warehouse, this incredibly simple transformation pipeline would cost up to $1,576,800 per year. Worse still, if the warehouse was subsequently increased in size to a MEDIUM, this would increase to a shocking $6.3m a year for a single warehouse.

Around four years ago, I did a consultancy visit to a customer in Munich Germany who had deployed a similar pipeline, also running 24x7 and were shocked by the high cost. The customer justified the 24x7 operation because the system tracked global internet based sales, and could therefore receive sales 24x7.

After some time on the whiteboard to map out the equally simple pipeline, I understood the problem and asked the customer - "So, where are your Data Analysts based".

I'm sure they thought it was a stupid question, as their reply was like talking to a five year-old. "Here in Munich, John".

In response, I asked whether the Data Analysts were looking at the data at 4am, and if not, why were we processing data at 4am and paying for a virtual warehouse to wake up every minute leading to a huge bill?

Finally, I also questioned, why they were continuously processing data every minute. What action would the analysts take that needed the data to be as little as one minute stale?

In conclusion, the same transformation pipeline cost, even on a MEDIUM size warehouse could be cut by 50%, simply by suspending the Dynamic Table refresh from 8pm to 8am using a Snowflake Task.

We could further reduce the cost by a factor of four by executing the following SQL statements.

alter dynamic table T3 
   set warehouse = DT_XSMALL_WAREHOUSE;
alter dynamic table T4
   set warehouse = DT_XSMALL_WAREHOUSE;

The two simple SQL statements move the refresh operation to an XSMALL warehouse costing 25% of the cost. Of course, we'd need to check the refresh performance was acceptable, but assuming an INCREMENTAL refresh method, this should be pretty fast to complete.

Finally, we should question whether the Data Analytics really need the data refreshed every minute. Executing the following SQL would reduce costs by a factor of 60.

alter dynamic table T5
   set target_lag = '1 Hour';

"Never give a user what they ask for, give them what they need" - Dr. David Pearson.

Can Artificial Intelligence Help?

As I've described before, a new breed of Artificial Intelligent Agents can be used to tirelessly monitor, alert or even adjust system configuration. It's like having an Artificially Intelligent Database Administrator proactively tuning your system.

However, the challenge of tuning a multi-petabyte analytic platform which charges using a pay-for-use basis is more challenging that you'd expect.

The diagram above illustrates the primary challenge, balancing the need to maximize throughput of large (mainly batch) transformation jobs, and maximizing performance of end-user queries while also optimizing spend.

There are also some critical facts about Snowflake which need to be understood:

1. Around 80% of the cost of running a Data Warehouse comes from repeating automated jobs. The majority of these tend to be batch oriented and many prioritize throughput over performance - the need to regularly deliver data, perhaps once per hour or once per day.

2. Years of research with over 50 Snowflake customers and thousands of queries against real customer data have demonstrated that most customers have a poor virtual warehouse deployment strategy which potentially leads to significant cost overspend.

3. Snowflake Dynamic Tables have the fastest growth in uptake of any Snowflake feature and are likely to become the gold standard for data transformation on Snowflake. However, unlike hand-coded ELT jobs, the warehouse size and execution frequency can be easily adjusted as these are entirely configurable in the Dynamic Table definition.

Taking both the problem (balancing throughput, performance and cost), with the configurable nature of Dynamic Tables means it's possible to automatically monitor and even change system configuration in real-time using Agenic AI.

The screenshot above shows the frequency of AI Agent decisions used to monitor and adjust Snowflake to maximize performance and optimize costs.

For example, if an ELT pipeline normally needs a MEDIUM sized warehouse but regular end-of-month processing would justify an X4LARGE, an AI Agent could automatically identify the opportunity and even change the configuration subject to System Administrator approval. This would not only deliver results over 100 times faster, but it would avoid a common mistake made by almost every Snowflake customer, using an inappropriate virtual warehouse size, and effectively the largest possible machine for the highest expected workload.

Conclusion

While the cost of Cloud Services compute and even storage for Dynamic Tables is likely to negligible, we need to consider the cost of running a virtual warehouse.

With just three SQL statements, we reduced the maximum cost of this simple transformation pipeline from $6.3m per year down to $26,280.

To summarize what we learned:

• Cloud Compute and Storage costs can (in most cases) be ignored. They shouldn't add significant cost to a pipeline deployed using a Dynamic Table.

• Warehouse Costs however, can be remarkably high and lead to an eye-watering annual bill if deployed without thought.

• Snowflake is not expensive! Like any tool, if used incorrectly it can lead to a poor outcome. Applying some common sense goes a long way.

• TARGET_LAG: Is critical in determining the frequency with which a Dynamic Table query is executed. If the data changes frequently, it can lead to significant cost. Carefully choose a TARGET_LAG based on how fresh you need the data. Don't just set it to once per minute because you can.

• Dynamically Change the lag: You can set up a simple schedule to suspend Dynamic Table refreshes or adjust the TARGET_LAG during off-peak or holiday periods. Suspending the refresh overnight will immediately reduce the costs by up to 50%.

Finally, in addition to manually adjusting Dynamic Table configuration, we could deploy AI Agents to autonomously monitor, alert or even tune Dynamic Tables to automatically achieve the almost impossibly complex goal of balancing the requirements of throughput, performance and cost.

Deliver more performance and cost savings with Altimate AI's cutting-edge AI teammates. These intelligent teammates come pre-loaded with the insights discussed in this article, enabling you to implement our recommendations across millions of queries and tables effortlessly. Ready to see it in action? Request a recorded demo by just sending a chat message (here) and discover how AI teammates can transform your data teams.

Working with data in any capacity—be it as a data/analytics engineer, data scientist, or data analyst—inevitably dealing with the need for documentation. Whether it's clarifying unclear fields, understanding dataset structures, or seeking explanations for data discrepancies, clear and comprehensive documentation is important. In the constantly evolving data world, where accuracy and clarity are non-negotiable, tools like dbt (data build tool) emerge as lifesavers.

Recognizing the importance of clear documentation in the data world, dbt™ makes documentation significantly easier through its dbt™ docs feature.

dbt provides a way to generate documentation for your dbt™ project. The documentation for your project includes:

• Project Information: including model code, a DAG of your project, any tests you've added to a column, and more.

• Data warehouse Information: including column data types, and table sizes. This information is generated by running queries against the information schema.

A standout feature of dbt™ lies in its capability to enhance documentation comprehensively. Users can add detailed descriptions to models, columns, sources, and other project elements, enriching the documentation with context and clarity. This not only aids in understanding the data structures but also streamlines collaboration and knowledge sharing within teams.

Here's an example of how dbt™ models can be documented:

version: 2

models:
  - name: events
    description: This table contains clickstream events from the marketing website

    columns:
      - name: event_id
        description: This is a unique identifier for the event
        tests:
          - unique
          - not_null

      - name: user-id
        quote: true
        description: The user who performed the event
        tests:
          - not_null

Overcoming Documentation Challenges

Despite the robust dbt™ docs feature, documentation often takes a backseat due to the inherent gap between project work and the documentation process. After completing a model, users frequently find themselves navigating to various directories to locate and manually edit schema YAML files for documentation purposes. This manual approach is time-consuming and prone to errors, relying heavily on individual skills and consistency.

Introducing Power User for dbt Extension: A Game-Changer in Documentation

To address these challenges and streamline the documentation workflow, Altimate AI introduces the Power User for dbt vscode extension. This extension leverages AI capabilities to revolutionize documentation generation, making it a seamless and efficient process. Specifically, Altimate AI is designed to emulate the behavior of human engineers by leveraging contextual insights drawn from schema, sources, column-level lineage, metadata, and existing documents to create tailored documentation, rather than just generating generic documentation.

Within the documentation editor, users can access a range of functionalities:

1. Model Documentation: Users can edit individual model documentation directly within the editor, eliminating the need for manual YAML editing. Additionally, AI-driven generation allows for instant and accurate descriptions, tailored to specific use cases such as professional or more casual documentation styles

   

2. Column Documentation: The extension enables users to generate detailed documentation for individual columns, automatically saving them in the 'schema.yml' file with the appropriate syntax. This automation not only accelerates development but also ensures consistency and accuracy across documentation.

   

3. Multiple Languages Documentation: Users have the flexibility to create documentation in several languages, including French, English, Dutch, and German. This feature enables the generation of content that caters to diverse linguistic needs.

4. Persona-Specific Documentation: The extension offers a variety of presets designed for different audiences. Users can select the persona that best matches their needs, whether it be a technical user, business user, or general user. For general users, the documentation is crafted without a specific persona focus, ensuring it is accessible and understandable to a broad audience.

Impact of Effective Documentation in dbt™

Quality documentation significantly improves how models are used and reduces unnecessary communications between teams, boosting productivity. Using the dbt Power User extension, you can save at least two minutes per column in documentation tasks. Depending on the size and complexity of the models, this efficiency can lead to a reduction of 10%-20% in development expenses. This not only translates to substantial cost savings but also frees up engineers to focus on more critical tasks.

Harnessing the Power of AI for Documentation Excellence

Maintaining documentation in a dynamic data environment presents a substantial challenge. As business requirements expand and projects evolve, the task of updating documentation to reflect changes in models, schemas, or sources becomes increasingly complex and resource-intensive. This constant need for updates can significantly drain time and productivity. Hence, the integration of AI-driven documentation generation within dbt Power User extension represents a significant advancement in data documentation practices. By automating tedious tasks and offering customizable options, dbt™ enables engineers to concentrate on development work while ensuring that the documentation stays comprehensive and up-to-date.

Final Thoughts

In the rapidly changing world of data management, understanding the essence of quality documentation has never been more critical, dbt Power User extension documentation capabilities shine as a beacon of efficiency and excellence. By leveraging advanced technologies like AI, dbt Power User extension not only simplifies the documentation process but also promotes data transparency, collaboration, and innovation. Embracing these powerful tools paves the way for enhanced productivity, informed decision-making, and data-driven success.

For data teams, the clock is always ticking. In a business landscape defined by rapidly expanding datasets, slow queries can grind insights to a halt. That's why performance tuning is especially critical when working with a platform like Snowflake. This platform, renowned for its lack of traditional indexes, presents unique challenges and opportunities for optimizing query performance. To navigate these uncharted waters, data engineers and analysts need a mastery of specialized optimization tactics. Join me as we dive into the art of writing fast code that unlocks the full capabilities Snowflake offers. We’ll review examples ranging from basic query tuning to more advanced optimizations that can mean the difference between swift analytics and stalled projects.

Snowflake Performance Tuning Strategy

Before we start, let’s put the overall performance-tuning strategy in context. The diagram below illustrates the typical data processing pipeline you’ll see on any Snowflake platform, which consists of the following:

• Data Loading: This involves copying data from Cloud Storage into a Table Landing zone. This normally uses the Snowflake COPY or SNOWPIPE commands.

• Data Transformation: Data from multiple sources and loading into Data Integration and Data Delivery zones. This involves the execution of Insert, Update, or Merge statements to clean, manipulate, and transform the data ready for consumption.

• Data Querying & Consumption: The data by end user dashboards, reports, or analysis. This typically involves SELECT statements against the Data Delivery zone.

It’s worth considering the above because the approach and solutions tend to differ. Let’s consider each in turn.

Data Delivery

As indicated above, data delivery is all about quickly using SELECT statements to deliver results to end-users. This means it's vital SQL is tuned for maximum performance. When tuning query performance, the first place to look is the query profile.

The screenshot above illustrates the first place to check for performance issues. Many assume this indicates the time spent, but it's the time waiting for resources.

The above case shows the query spent just 10% of its time waiting on CPU resources, almost zero (0.2%) of the time waiting for Local Disk I/O to complete, but nearly 90% of the time waiting on Remote Disk I/O.

Reviewing the overall Snowflake hardware architecture illustrated in the diagram below is helpful to understand what this means.

The above diagram shows how queries submitted to Snowflake are handled by the Cloud Services Layer, determining the best way to execute the query. The SQL is then passed to a Virtual Warehouse, a physical computer with CPU cores, memory, and SSD (Local) storage.

When queries are executed, the data is read from Remote Storage, cached in SSD, and results computed and returned to the user.

The key insight here is that (SSD) Local Storage is hundreds of times faster than Remote (disk) storage. One of the most effective ways to maximize query performance is limiting Remote Disk accesses.

The most effective way to reduce remote disk I/O is using the query WHERE clause to filter out rows. Complex SQL queries often include a series of Common Table Expressions (CTEs or inline views), and the best practice is to filter out rows as early as possible. The fewer rows fetched from the Remote Disk, the faster the query executes.

The next step in tuning Snowflake query performance is to review the query statistics illustrated in the screenshot below.

Snowflake stores data in 16MB blocks called micro partitions and automatically prunes these to maximize query performance, but it's entirely dependent upon the WHERE clause. If (for example), a query includes no WHERE clause, Snowflake will need to fetch the entire table from Local or Remote storage with a significant impact upon performance.

Tuning the query WHERE clause

You can make simple changes to the WHERE clause to maximize query performance. Take, for example, the following SQL statement:

select *
from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.ORDERS
where to_char(o_orderdate,'DD-MON-YYYY') = '01-APR-1992';

The above query returns ORDERS placed on 1st April 1992. However, we limited Snowflake's ability to prune query performance automatically because we formatted the O_ORDERDATE as a character in the WHERE clause.

select *
from SNOWFLAKE_SAMPLE_DATA.TPCH_SF1000.ORDERS
where o_orderdate = to_date('01-APR-1992','DD-MON-YYYY');

The above query produces the same result ten times faster, completing in under four seconds instead of nearly 40 seconds. The reason is we left the O_ORDERDATE column unmodified.

Surprisingly, despite converting a DATE to a CHARACTER field, Snowflake can still perform some partition pruning. However, modifying WHERE clause columns using a user-defined function will almost certainly lead to a full table scan.

select *
from orders
where format_date_udf(o_orderdate,'DD-MON-YYYY') = '01-APR-1992';

The above query will perform even worse as Snowflake cannot evaluate the output of a User-Defined Function and will always lead to a full table scan.

Tuning Sort Operations

Designing an analytics solution without including an ORDER BY or GROUP BY clause in the query is almost impossible. However, sorts are the most computationally expensive operations and are often the most significant cause of query performance issues.

Often, data analysts need to determine whether a given column is unique and will execute the following SQL:

select count(distinct l_orderkey)
from snowflake_sample_data.tpch_sf1000.lineitem;

However, the following query will execute around 600% faster with an accuracy of around 99%:

select count_approx_distinct(l_orderkey)
from snowflake_sample_data.tpch_sf1000.lineitem;

The COUNT_APPROX_DISTINCT is one of a series of approximation functions that accurately estimate large data volumes orders of magnitude faster.

Dealing with high-cardinality data

While dealing with large data volumes is often necessary, you should be especially careful when sorting high cardinality data, for example, by a unique key. Take, for example, the following SQL statements, which appear similar but have wildly different query performance.

Query 1 - Unique Key

select o_orderkey,
       sum(o_totalprice)
from orders
group by 1;

Query 2 - Low Cardinality

select o_orderpriority,
       sum(o_totalprice)
from orders
group by 1;

You may be surprised to find that query 2 (Low Cardinality) runs around 22 times faster than query 1 (Unique Key), as illustrated in the screenshot below:

The reason for the huge query performance difference is that Query 1 - Unique Key (which takes over six minutes to complete), includes a GROUP BY on a column with 1.5 billion unique values whereas Query 2 - Low Cardinality (executing in 11 seconds) has just five distinct values.

In this case, the best practice is to be aware of the high cost of sort operations and avoid GROUP BY or ORDER BY operations on highly unique keys.

Using the Snowflake LIMIT or TOP option

There are sometimes use cases where we need to ORDER very large data volumes by unique keys, but it’s still possible to improve query performance using the simple LIMIT or TOP options on SELECT statements.

Consider the following SQL statements:

Query 1 - Order by Every Row

select *
from lineitem
order by l_quantity;

Query 2 - Order by With Limit

select *
from lineitem
order by l_quantity
limit 10;

Using the LIMIT 10 option on Query 2 (only first ten rows), leads to an 11 times improvement in query performance from over 17 minutes to just 1m 35s.

The reason for the huge difference in query performance is revealed in the Query Plan. Query 1 (Order by every row) on the left includes a Sort operation that returns 600 million rows, whereas Query 2 (order by with limit) on the right uses a SortWithLimit operation to reduce the entries from 600m to just 10.

Another significant contributor to the query performance is that the Order by with limit scans half the number of micro-partitions and eliminates spilling to storage, significantly reducing query execution time.

Spilling to Storage

The diagram below illustrates the internal structure of an XSMALL virtual warehouse. It (currently) consists of eight CPU cores, gigabytes of memory, and hundreds of gigabytes of fast SSD storage attached to slower Remote Storage.

hen Snowflake executes a query with an ORDER BY or GROUP BY clause, it must sort the results it attempts to complete in super-fast main memory. However, sorting large data volumes in a smaller warehouse runs out of memory and needs to spill working data to SSD, which is thousands of times slower.

When sorting even bigger data volumes, it may need to spill to Remote Storage, which is many hundreds of times slower again.

It’s good practice to avoid spilling to storage where possible, but how to do this?

There are several Snowflake best practice tuning strategies to avoid spilling to storage, including:

1. Reduce sort volume: Similar to the best practices above, this involves using the WHERE clause to filter out results as early as possible to reduce the number of sorted rows.

2. Use a LIMIT: As we can see above, a LIMIT clause is pushed down into the query, eliminating spilling to storage with a huge improvement in Snowflake query performance.

3. Scale Up: The diagram below illustrates the internal structure of a MEDIUM-size warehouse, which includes four servers**.** By scaling up to a larger warehouse, we can spread the workload across multiple servers, which gives (in this case) four times the memory and SSD, which is the simplest way to resolve the issue.

4. Cluster the Data: A final (although sometimes challenging) approach involves clustering the data by the sort key. This often eliminates the SORT operation and can have a massive impact on query performance.

Clustering the Data

The quickest and most simple way of clustering data is to sort it by one or more columns. The SQL snippet below will produce a sorted version of a table:

insert overwrite into orders
select *
from orders
order by o_orderdate;

The above query replaces the contents of the ORDERS table with data sorted by ORDER DATE, which means any GROUP BY or ORDER BY won’t need to sort the data. This approach is, however, not without its drawbacks, and it’s worth reading the article Snowflake Clustering Keys Best Practice for further guidance.

Maximizing Snowflake Data Transformation Performance

Data Transformation involves fetching, cleaning, merging, and transforming data into a form suitable for end-user consumption, and this often accounts for 60-80% of the overall system cost. It, therefore, must be the primary focus for Snowflake query tuning. In addition to producing results faster, tuning transformation jobs can significantly impact Snowflake's cost.

Avoid row-by-row-processing

Data Engineers and Data Scientists have the option to execute procedural code against Snowflake using Python, Java, Javascript and Snowflake Scripting in addition to Scala using Snowpark. There's a risk however, that coding in a language which supports LOOPS, leads to repeated single-row query execution - also know as "Row-by-Row" processing.

Row by Row Processing - is "Slow-by-Slow" - Tom Kyte

One of the most important query-tuning best practices on Snowflake is to avoid row-by-row processing. Unlike on-premises databases like Oracle, Snowflake is designed primarily for set-at-a-time processing of large data volumes rather than single-row inserts, updates, or deletes.

Take the following row-by-row solution, which takes around ten seconds to copy just ten rows into the CUSTOMER table:

declare
v_counter number;
begin
  for v_counter in 1 to 10, do
     insert into customer
     select *
     from snowflake_sample_data.tpcds_sf100tcl.customer
     limit 1;
  end for;
end;

Then, consider the following set-at-a-time query that copies data from the same table in 9.4 seconds on an XSMALL warehouse.

insert into customer
select *
from snowflake_sample_data.tpcds_sf100tcl.customer
limit 15000000;

In conclusion, using set-at-a-time queries, Snowflake can process fifteen million rows simultaneously as row-by-row processing copies ten rows. That’s a performance improvement of 1.5 million times.

The only exception to the above scenario is when dealing with Snowflake Hybrid Tables, designed for OLTP-type applications processing single-row DML operations in milliseconds.

Scale up for larger Data Volume.

This best practice is true for almost any end-user or transformation queries and involves executing long-running queries on a larger virtual warehouse.

In principle, provided the query workload can be distributed across all the available servers, any query will run twice as fast on the next warehouse size. Consider the following table showing the benchmark results of copying a 1.3 TB table on different warehouse sizes.

The results above demonstrate very large data volumes can be scaled from an XSMALL warehouse running for four hours to an X6LARGE in just 60 seconds. Notice the improvement is around 76% to 100% each time. This is the most important key performance indicator (KPI) when warehouse sizing. Provided the workload completes around twice as fast each time, it’s worth trying the workload on the next size up.

The challenge of identifying the optimum warehouse size is illustrated in the diagram below.

The above diagram shows the performance of a benchmark query executed on an XSMALL to X2LARGE warehouse. In this case, the query processed a relatively small data volume (several gigabytes), and although each time we scaled up to a larger warehouse, (for example from an XSMALL to a SMALL), the query was executed twice as fast.

Since the charge rate per second doubles with warehouse size, we can conclude that each step where the performance is at least double, that the cost will remain the same.

Snowflake has an amazing super-power. You can often get the same work done twice as fast for the same cost. - Kent Graziano (The Data Warrior)

However, each time the performance gain is less than 100%, it costs slightly more, although of course you get the extra performance improvement.

For example, running the query on an XLARGE warehouse would cost 12% more than a LARGE warehouse for an 88% performance improvement.

The real insight here, however, is that the query time is reduced from 80 minutes on an XSMALL warehouse to just 8 minutes on a LARGE for the same cost.

Be aware though, there's almost no performance benefit in scaling up relatively simple, short running queries. The best approach is to execute your workload (all the queries for a given job or set of dashboard users), on a warehouse size, and then scale up until the performance improvement is less than 100%.

Maximizing Snowflake Data Loading Performance

One of the single biggest mistakes made by data engineers is assuming a COPY operation will execute faster on a bigger virtual warehouse. The correct answer is, yes, it will, but it depends.

If you execute the following SQL to load data files from the @SALES stage, Snowflake will use the currently assigned virtual warehouse to load data into the SALES table. Let’s assume you use a MEDIUM-size warehouse to maximize load performance.

copy into sales
from @sales_stage;

If you’ve only a single file to load, the diagram below illustrates how Snowflake executes the COPY operation using a single CPU.

In summary, this won’t improve COPY performance and will take the same time to load on a MEDIUM-size warehouse as an XSMALL.

If you deliver hundreds of files, each file will be loaded in parallel on a separate CPU core with massive load performance. As a MEDIUM size warehouse has four servers, each with eight CPU cores, you can load up to 32 files in parallel, effectively improving load performance by 3,200%.

Be aware, however, it’s best practice to ensure data files are sized between 100-250MB, which means you’ll need a huge volume of data to justify splitting files. It’s, therefore, good practice to use just two warehouses to load all data into Snowflake. One XSMALL warehouse for single files (data loads of up to 250MB compressed), and another MEDIUM or larger warehouse for huge data loads split into file chunks of up to 250MB.

Remember, an XSMALL warehouse can load just eight files in parallel; during very busy times, you may be loading more than eight files in parallel. It’s, therefore, also a best practice to configure data-loading warehouses to use the Snowflake scale-out feature. The SQL statement below illustrates how:

create warehouse load_vwh with
   Warehouse_size = XSMALL
   min_cluster_count = 1
   max_cluster_count = 8
   auto_suspend = 30;

The diagram below illustrates how the solution works. It shows how multiple parallel COPY operations are executed on a single XSMALL warehouse. As the number of operations reaches eight parallel loads, Snowflake automatically adds additional servers to cope with the additional workload.

Conclusion

In conclusion, Snowflake deploys a revolutionary hardware and software architecture to maximize query performance, supporting scaling up (to a larger warehouse) and out (for more concurrent queries). Using the simple query performance strategies outlined in this article can greatly impact overall query performance, often for very little effort.

Deliver more performance and cost savings with Altimate AI's cutting-edge AI teammates. These intelligent teammates come pre-loaded with the insights discussed in this article, enabling you to implement our recommendations across millions of queries and tables effortlessly. Ready to see it in action? Request a recorded demo by just sending a chat message (here) and discover how AI teammates can transform your data teams.