A lakehouse typically starts with files: usually Parquet files written to object storage. Delta Lake adds a transaction log, ACID transactions, schema enforcement, and versioned table state on top of those files. But using Delta Lake does not make the physical table maintenance-free. As data is inserted, streamed, merged, updated, and deleted, three familiar problems appear:
| Problem | Why it matters | Maintenance operation |
|---|---|---|
| Many small or poorly arranged files | Queries open and scan more files than necessary | OPTIMIZE rewrites files to improve size and layout |
| Old files are no longer referenced by the current table state | They continue consuming storage and preserve data beyond the intended retention window | VACUUM removes unreferenced files after the retention period |
| Table and column statistics become stale | The optimizer has less reliable information when choosing a query plan | ANALYZE refreshes statistics |
These issues need managing. Optimized writes and auto compaction can reduce small-file pressure, but they do not replace every file-layout, retention, and statistics task. So many platform teams built Databricks OPTIMIZE, VACUUM, and ANALYZE commands into notebooks or SQL tasks, like these:
OPTIMIZE main.sales.orders;
VACUUM main.sales.orders;
ANALYZE TABLE main.sales.orders
COMPUTE STATISTICS;The platform team chose (or guessed at) the tables, the commands, the schedule, and the compute. A nightly or weekly schedule helped avoid a harder question: does this table actually need maintenance now? A fixed schedule works, but it is very inefficient.
What is Databricks Predictive Optimization?
Databricks Predictive Optimization is a managed service that automatically runs maintenance operations like OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, based on table needs and workload telemetry, not a fixed schedule.
Predictive Optimization moves that maintenance decision off a schedule maintained by your team and onto Databricks. For eligible Unity Catalog managed Delta Lake and Apache Iceberg tables, Databricks identifies tables that would benefit from maintenance and queues OPTIMIZE, VACUUM, or ANALYZE. The operations run on serverless compute for jobs rather than on a cluster you configure for a maintenance workflow.
| Without Predictive Optimization | With Predictive Optimization |
|---|---|
| Your team selects tables | Databricks evaluates eligible tables |
| Commands run at a fixed time | Operations run when Databricks decides they are useful |
| Your team provisions and monitors compute | Databricks uses serverless jobs compute |
| Cost is attached to a known job | Cost appears as platform-managed usage |
| A failed schedule is visible in Jobs | You inspect table history and system tables |
Predictive Optimization has three practical benefits:
Maintenance is based on observed need rather than only a calendar.
Teams do not provision separate job compute for eligible tables.
Statistics and file layout can be maintained more consistently.
But the same automation changes visibility. Predictive Optimization can be inherited from the account through the catalog and schema to the table, so behavior can change without anyone adding a new job. Databricks enables it by default for accounts created on or after November 11, 2024, while lower-level settings can override the inherited value.
That creates two problems. First, the work has no owner: it runs on serverless compute for jobs, billed under a SKU that several other Databricks features share, so the line on your bill doesn't say it came from table maintenance. Only billing_origin_product = 'PREDICTIVE_OPTIMIZATION' identifies it. Second, if a team already schedules its own OPTIMIZE and VACUUM job on the same table, two systems now maintain that table, and each has its own billing line. Databricks doesn't document what Predictive Optimization does when the table is already compacted, so the only way to find out is to look at system.storage.predictive_optimization_operations_history for your own tables.
Databricks Predictive Optimization covers only Unity Catalog managed tables
In real Databricks workspaces, table format, catalog, and storage ownership are separate concerns. Converting Parquet data to Delta adds the Delta transaction log. It does not necessarily turn the result into a Unity Catalog managed table. For example, CONVERT TO DELTA can convert Parquet data in an external location into an external Delta table. That table gains Delta semantics, but it remains external. Predictive Optimization supports only Unity Catalog managed tables.
| Table state | Predictive Optimization coverage |
|---|---|
| Parquet files or a Parquet table | ✗ |
| Delta table outside Unity Catalog | ✗ |
| Unity Catalog external Delta table | ✗ |
| Unity Catalog managed Delta or Iceberg table | ✓ Eligible, subject to enablement and prerequisites |
| Managed table that still has a legacy maintenance job | ✓ Eligible, but audit both maintenance paths |
Predictive Optimization also excludes tables received through OpenSharing. It requires a Premium plan or above, a supported region, and SQL warehouses or Databricks Runtime 12.2 LTS or later. So don’t assume it is available or active.
Predictive Optimization does not run ZORDER, and it ignores Z-ordered files. Its VACUUM retention behavior follows the table's delta.deletedFileRetentionDuration property, with a seven-day minimum when Predictive Optimization runs VACUUM FULL. A manual job that applies a deliberate Z-ordering strategy or encodes special retention behavior might therefore remain necessary.
Enablement does not prove Predictive Optimization ran
Enablement and execution are different things. Check a table directly:
DESCRIBE TABLE EXTENDED main.sales.orders;The Predictive Optimization field shows whether the setting is enabled, disabled, or inherited. It does not prove that Databricks has executed maintenance.
On Databricks Runtime 18 LTS and later, you can inspect the latest evaluation and selected skip reasons with:
DESCRIBE TABLE EXTENDED main.sales.orders AS JSON;The JSON includes predictive_optimization_evaluations for supported operation types. Only the latest evaluation is shown, results can take up to 24 hours to appear, and skip reasons are not provided for ANALYZE.
For historical activity across tables, query the Predictive Optimization system table:
SELECT
metastore_name,
catalog_name,
schema_name,
table_name,
table_id,
operation_type,
COUNT(DISTINCT operation_id) AS successful_operations,
SUM(usage_quantity) AS estimated_dbus,
MAX(end_time) AS last_run
FROM system.storage.predictive_optimization_operations_history
WHERE start_time >= CURRENT_TIMESTAMP() - INTERVAL 30 DAYS
AND operation_status = 'SUCCESSFUL'
AND usage_unit = 'ESTIMATED_DBU'
GROUP BY
metastore_name,
catalog_name,
schema_name,
table_name,
table_id,
operation_type
ORDER BY estimated_dbus DESC;The result tells you which tables Databricks maintained, which operations ran, and the estimated DBUs attributed to them. The estimate is approximate when several operations share a cluster. The table can also contain operation types beyond the three headline commands, including compaction, clustering, column selection, deletion, purge, and compatibility work.
How much does Databricks Predictive Optimization cost?
Predictive Optimization uses billable serverless jobs compute. In system.billing.usage, identify it by this filter rather than by the SKU name alone:
billing_origin_product = 'PREDICTIVE_OPTIMIZATION'The following query returns recorded DBUs and an estimated list-price cost:
SELECT
DATE_TRUNC('day', u.usage_start_time) AS usage_day,
u.workspace_id,
u.sku_name,
p.currency_code,
SUM(u.usage_quantity) AS dbus,
SUM(u.usage_quantity * p.pricing.effective_list.default) AS estimated_list_cost
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
ON u.account_id = p.account_id
AND u.cloud = p.cloud
AND u.sku_name = p.sku_name
AND u.usage_unit = p.usage_unit
AND u.usage_end_time >= p.price_start_time
AND (p.price_end_time IS NULL OR u.usage_end_time < p.price_end_time)
WHERE u.billing_origin_product = 'PREDICTIVE_OPTIMIZATION'
AND u.usage_unit = 'DBU'
AND u.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
DATE_TRUNC('day', u.usage_start_time),
u.workspace_id,
u.sku_name,
p.currency_code
ORDER BY usage_day;These four values are distinct:
Estimated per-operation DBUs
≠ recorded billing DBUs
≠ estimated list-price cost
≠ final invoice amountThe operation-history table normally updates within two hours. Billing data can take up to 24 hours. List-price cost can differ from the invoice because of negotiated rates, commitments, credits, taxes, and billing corrections such as retractions and restatements.
Should you keep your existing OPTIMIZE and VACUUM jobs?
Some of your existing Databricks VACUUM and OPTIMIZE jobs may no longer be needed, and some still are. Review them one at a time, in three steps.
Run the operation-history query above to list the tables Predictive Optimization maintained.
Find the jobs that run
OPTIMIZE,VACUUM, orANALYZEon those same tables.Decide what to do with each job, using the table below.
A table that appears in both lists has two maintenance paths. That shows possible overlap, but it doesn't prove you paid twice, because the two paths may run different operations or touch different files.
| Existing job | What to do |
|---|---|
Plain scheduled OPTIMIZE on an eligible managed table | Compare recent automatic and manual runs, then consider pausing it as a test |
OPTIMIZE ... ZORDER BY | Keep it. Predictive Optimization doesn't run ZORDER |
VACUUM with a deliberate retention requirement | Check the table's retention property and your recovery needs first |
| Job on an external or shared table | Keep it. Predictive Optimization doesn't cover these tables |
| Notebook that also runs checks, alerts, or downstream tasks | Move the maintenance commands out and keep the rest of the notebook |
Don't delete a job because a table shows ENABLED or because one automatic operation succeeded. Test on one table first. Pause one low-risk schedule, keep a way to turn it back on, and watch the table through several normal write and query cycles. Then compare:
Table history and operation metrics
Query performance
Storage behavior
Predictive Optimization cost
The cost of the job you removed
The saving is the cost of the removed job, minus anything that went up afterwards:
Net saving = removed job cost
- increase in Predictive Optimization cost
- increase in query and storage costIf Predictive Optimization cost rises by about as much as the job used to cost, the net saving is close to zero.
What to check before you pause a maintenance job
Databricks Predictive Optimization moves maintenance from schedules your team owns to decisions Databricks makes, and it moves part of the cost from named jobs into serverless usage. This change reduces unnecessary maintenance toil, but in a workspace that mixes Parquet data, external Delta tables, managed Unity Catalog tables, and older jobs, the change is easy to overlook. Before you change any schedule, check four things:
Eligibility. Confirm the table is a Unity Catalog managed table and see whether the setting is enabled, disabled, or inherited.
What ran. Use the operation-history table to see what Databricks actually evaluated and ran on it.
What it cost. Filter
system.billing.usageonbilling_origin_product = 'PREDICTIVE_OPTIMIZATION'to see the DBUs.What overlaps. Compare that activity with the jobs your team still owns.
With those four answers, you can decide which schedules are still useful and what Predictive Optimization is costing you.
Frequently asked questions
What is Databricks Predictive Optimization?
Databricks Predictive Optimization is a managed service that automatically runs maintenance operations like OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, based on table needs and workload telemetry, not a fixed schedule. The operations run on serverless compute for jobs rather than on a cluster you configure.
Is Databricks Predictive Optimization enabled by default?
Databricks enables it by default for accounts created on or after November 11, 2024. The setting is inherited from the account through the catalog and schema to the table, and lower-level settings can override the inherited value. DESCRIBE TABLE EXTENDED shows whether it is enabled, disabled, or inherited, but not whether Databricks has executed maintenance.
How is Databricks Predictive Optimization billed?
Predictive Optimization uses billable serverless jobs compute, under a SKU that several other Databricks features share. Filter system.billing.usage on billing_origin_product = 'PREDICTIVE_OPTIMIZATION' to see its DBUs, and join system.billing.list_prices for an estimated list-price cost.
Which tables does Predictive Optimization not cover?
Predictive Optimization supports only Unity Catalog managed tables. Parquet tables, Delta tables outside Unity Catalog, Unity Catalog external Delta tables, and tables received through OpenSharing are not covered.
Does Predictive Optimization replace a Databricks OPTIMIZE job with ZORDER?
No. Predictive Optimization does not run ZORDER, and it ignores Z-ordered files, so keep a Databricks OPTIMIZE job that applies a deliberate Z-ordering strategy. A Databricks VACUUM job with a deliberate retention requirement also needs a check first, because Predictive Optimization follows the table's delta.deletedFileRetentionDuration property.
