altimate-code + DeepSeek V4 Pro achieves 78.0% pass rate (32/41 tasks on DuckDB) — matching Sonnet 4.6 at a fraction of the cost.
About ADE-Bench
ADE-Bench is a benchmark created by Benn Stancil (founder of Mode) in collaboration with dbt Labs. It evaluates AI agents on real-world analytics and data engineering tasks using actual dbt projects and databases. Each task runs in a Docker container sandbox; the agent attempts to resolve the task, and success is measured by whether all dbt tests pass afterward. Tasks include realistic data problems: vague requests like "it’s broken," debugging, schema issues, and complex analytics queries.
Test Configuration
Benchmark Comparison
Agents evaluated on ADE-Bench with DuckDB.
Per-Task Results — DuckDB
Best Run — 32 passed, 9 failed out of 41 tasks
| # | Task | Result | Score | Pass Rate |
|---|---|---|---|---|
| 1 | airbnb001 | ✓ | 10/10 | 100% |
| 2 | airbnb002 | ✓ | 11/11 | 100% |
| 3 | airbnb003 | ✓ | 7/7 | 100% |
| 4 | airbnb004 | ✓ | 2/2 | 100% |
| 5 | airbnb005 | ✓ | 4/4 | 100% |
| 6 | airbnb006 | ✓ | 7/7 | 100% |
| 7 | airbnb007 | ✓ | 11/11 | 100% |
| 8 | airbnb008 | ✓ | 4/4 | 100% |
| 9 | airbnb009 | ✗ | 0/1 | 0% |
| 10 | analytics_engineering001 | ✓ | 1/1 | 100% |
| 11 | analytics_engineering002 | ✓ | 2/2 | 100% |
| 12 | analytics_engineering003 | ✓ | 2/2 | 100% |
| 13 | analytics_engineering004 | ✗ | 1/2 | 50% |
| 14 | analytics_engineering005 | ✓ | 3/3 | 100% |
| 15 | analytics_engineering006 | ✗ | 4/7 | 57% |
| 16 | analytics_engineering007 | ✓ | 10/10 | 100% |
| 17 | analytics_engineering008 | ✓ | 1/1 | 100% |
| 18 | asana001 | ✓ | 2/2 | 100% |
| 19 | asana002 | ✓ | 3/3 | 100% |
| 20 | asana003 | ✗ | 16/17 | 94% |
| 21 | asana004 | ✗ | 5/6 | 83% |
| 22 | asana005 | ✗ | 7/8 | 87% |
| 23 | f1001 | ✓ | 6/6 | 100% |
| 24 | f1002 | ✗ | 9/10 | 90% |
| 25 | f1003 | ✓ | 4/4 | 100% |
| 26 | f1004 | ✓ | 2/2 | 100% |
| 27 | f1005 | ✓ | 4/4 | 100% |
| 28 | f1006 | ✓ | 4/4 | 100% |
| 29 | f1007 | ✓ | 6/6 | 100% |
| 30 | f1009 | ✓ | 1/1 | 100% |
| 31 | f1010 | ✓ | 2/2 | 100% |
| 32 | f1011 | ✗ | 5/6 | 83% |
| 33 | intercom001 | ✓ | 2/2 | 100% |
| 34 | intercom002 | ✓ | 4/4 | 100% |
| 35 | intercom003 | ✓ | 2/2 | 100% |
| 36 | quickbooks001 | ✓ | 12/12 | 100% |
| 37 | quickbooks002 | ✓ | 8/8 | 100% |
| 38 | quickbooks003 | ✗ | 5/14 | 35% |
| 39 | quickbooks004 | ✓ | 48/48 | 100% |
| 40 | simple001 | ✓ | 1/1 | 100% |
| 41 | simple002 | ✓ | 1/1 | 100% |
Sources
- ADE-Bench on GitHub (dbt-labs) — Benchmark repository and methodology
- Snowflake Blog: Cortex Code CLI Expands Support — Cortex Code benchmark results (65%, Opus 4.6)
- dbt Labs Blog: Introducing ADE-Bench — dbt Labs benchmark results (59%, Sonnet 4.5)
ADE-Bench FAQs
ADE-Bench is a benchmark created by Benn Stancil (founder of Mode) in collaboration with dbt Labs. It evaluates AI agents on real-world analytics engineering tasks using actual dbt projects running in Docker containers. Success is measured by whether all dbt tests pass after the agent attempts the task. It covers 43 tasks across Snowflake and DuckDB.
Altimate Code achieved 74.4% (32/43 tasks) on ADE-Bench using Claude Sonnet 4.6 on Snowflake. On DuckDB with DeepSeek V4 Pro, it scored 78.0% (32/41 tasks). Cortex Code CLI scored 65%, dbt Labs scored 59%, and the Claude Code baseline scored 40%.
Altimate Code on Sonnet 4.6 beats competitors running Opus 4.6 — a more capable, more expensive model. The reason is the harness: purpose-built, deterministic tools for data engineering outperform raw model capability. SQL analysis, column-level lineage, and dbt integration are handled by specialized tools, not general-purpose code generation.
Each ADE-Bench task runs in an isolated Docker container with a real dbt project and database. The agent receives a task description (sometimes intentionally vague, like "it's broken") and must resolve it. Success is binary — all dbt tests must pass. Tasks include debugging, schema issues, complex analytics queries, and data engineering work across 43 scenarios.
Yes — ADE-Bench is open source on GitHub at github.com/dbt-labs/ade-bench. All benchmark tasks, scoring methodology, and submission instructions are public. Altimate Code's submission is PR #44 on the DataAgentBench repository.
ADE-Bench currently covers Snowflake and DuckDB. Altimate Code achieved 74.4% on Snowflake (Sonnet 4.6) and 78.0% on DuckDB (DeepSeek V4 Pro). Altimate Code also supports BigQuery, Databricks, Redshift, Postgres, MySQL, and SQLite in production.
Reproduce This on Your Warehouse. Same Agent, Your Data.
The Altimate Code build behind these ADE-Bench results runs on your own dbt project. Install it, or book a demo and we run it with you.
