Jobs Classic vs Jobs Serverless vs DBSQL who wins on TPC-DS?

TL;DR We ran the full TPC-DS benchmark suite across three compute planes: Databricks serverless SQL warehouses, Databricks serverless Jobs Compute and Databricks classic Jobs Compute, using Apache JMeter for standardized load generation. The goal: quantify latency, throughput, scalability and cost-per-query under controlled realistic workloads. After running more than 4,750 TPC-DS queries the data shows that Jobs Classic beats both Jobs Serverless variants on cost, performance and consistency. The real TPC-DS winner? Serverless SQL warehouses. Read on for all the details.

Databricks Serverless is the next evolution in data processing. It is a managed compute service that automatically provisions clusters and comes with built-in autoscaling and zero cluster management. It offers a compelling solution for many, especially those new to Databricks: why deal with cluster configurations when serverless can handle everything for you? 

We ran thousands of analytical queries daily to simulate real life production workloads and assess where Jobs Serverless stands in regards to costs and performance compared to Jobs Classic. This article presents our findings from a comprehensive TPC-DS benchmark across two scale factors (SF1 and SF1000), which simulated standard analytics queries at a production scale.

Here’s a rundown of what we tested and the metrics we benchmarked against:

Dataset: TPC-DS Scale Factor 1000 (1TB of data) and 1 (1GB of data)

Queries: 95 TPC-DS queries representing realistic analytical workloads complete with:

• Complex multi-table joins 

• Window functions and aggregations 

• Varying selectivity and complexity 

Total executions: More than 4,750 queries across multiple runs for statistical validity.

• Serverless SQL warehouses: Small, medium, large, X-large 

• Jobs Classic: Standard compute cluster (rd-fleet.xlarge) with Photon both enabled and disabled, 8 worker nodes, autoscaling disabled, on-demand and Spot (fallback to on-demand), release 16.4.12. This classic cluster with both Photon and Spot enabled is used as the baseline for comparisons throughout this report. We also tested variations with 2, 4 or 6 worker nodes, but defaulted to 8 worker nodes in the benchmark below.

• Jobs Serverless: Jobs Serverless - Standard (cost optimized) and Jobs Serverless - Performance (performance optimized) were both tested.

• Actual DBU consumption from system.billing.usage 

• Real costs based on list pricing 

• AWS cluster costs for Spot and on-demand instances

• Query latencies (P50, P90, P95, P99): P50 represents the median, meaning that 50% of queries complete in this time. P95 is approaching worst case scenarios where 1 in 20 queries exceed this value and P99 is the worst case where clusters still have to run regularly even if it’s just 1 in 100 queries which exceed this value. P99 is often the metric that determines the SLA guarantee.

• Performance consistency (coefficient of variation) 

All of this to say that our analysis and findings are based on real data from production-scale testing with actual billing information. 

You can download the notebooks we ran and try it for yourself below.

We ran the entire benchmark analysis across the spectrum of Databricks’ Jobs Compute. We analyzed both the Jobs Serverless offering as well as Jobs Classic, including variations with Photon and Spot instances disabled.

For this experiment we used a r6idn.xlarge instance with Spot pricing at the time of this article of $0.19 per hour and on demand pricing of $0.391 per hour. The job takes just under an hour to run. The table below contains a summary of the costs including the AWS cluster costs. The data indicates a consistent pattern:

At first glance, it seems like the serverless variants can match Jobs Classic’s P50 performance (5.41s) while using fewer DBUs (37 vs 60). But this is where Databricks’ pricing model changes the story.

The 133% DBU rate premium for Serverless ($0.35/DBU vs $0.15/DBU) negates the efficiency gains. Let’s break it down:

• Jobs Classic: (60 DBUs x $0.15) + $1.29 AWS charge = $10.22

• Jobs Serverless - Standard: (37 DBUs x $0.35) = $12.95

• Jobs Serverless - Performance: (52 DBUs x $0.35) = $18.2

As you can see from the calculation above, you end up paying 27% more for Jobs Serverless - Standard and 78% more for Jobs Serverless - Performance.

But that’s just one part of it. We haven’t even touched on the latency degradation. More about that next. 

If you care about SLAs, P90, P95 and P99 metrics are the ones keeping you up at night. Half your queries might run just fine, but 10% of them are running 30% slower? That’s a production incident. While median performance of Jobs Classic and Jobs Serverless appears to be somewhat competitive, the tail latencies expose some instability with serverless offerings: 

• Jobs Classic P99: 82.22 seconds, which means 99% of queries complete in under 82 seconds 

• Jobs Serverless - Standard P99: 231.45 seconds, which means 99% of queries complete in under 231 seconds (2.8x slower than Classic) 

• Jobs Serverless - Performance P99: 159.08 seconds, which means 99% of queries complete in less than 159 seconds (1.8x slower than Classic) 

For time sensitive pipelines like revenue reports, fraud detection and real-time analytics that kind of variability can lead to consistency issues.

Our analysis indicates that AWS cluster costs are not enough to offset the higher DBU rate of serverless jobs for the scale of the TPC-DS benchmark dataset. At the same time, the data shows that Jobs Classic configuration outperformed both Jobs Serverless offerings in P99.

So we decided to test the impact that different worker counts have on the cost and performance of Jobs Classic vs Jobs Serverless. Here are our findings:

The data shows that using 6 workers vs 8 actually reduced the number of DBUs consumed therefore reducing total costs, without significantly affecting the P99 time. However, reducing the number of workers to 4 or 2 degrades the workload as it causes the DBU consumption to spike to 94 or 96 DBUs, respectively. At these worker counts we also see an even greater hit to the performance of Jobs Classic, all at a cost which is 45% higher than the baseline. 

What does this mean for you? Experimentation with cluster configuration is still very much a requirement in order to optimize Classic Jobs. Sometimes scaling up actually lowers costs and scaling down increases costs, as seen in the results above. Thankfully, there are tools that can help you rightsize your clusters, from Databricks’ own autoscaler to external solutions like Capital One Slingshot.

Serverless is supposed to eliminate waste through better resource management. While that might be true, the data show that the efficiency gains do not outweigh the additional costs of serverless.

When we take into consideration the different DBU rates for Jobs Classic vs Jobs Serverless ($0.15/DBU vs $0.35/DBU), apparent savings transform into actual losses even when we add in the cloud cluster costs for the classic clusters.

As you can see above, lower DBU consumption does not necessarily mean lower costs.

Beyond just pure costs, the data also suggests that Serverless Jobs have higher performance variance:

A P50-to-P99 ratio calculates how far away your extreme values are from the median. The higher this multiplier the more variable the workload. The massive multipliers for the Jobs Serverless variants may indicate performance instability, that might make it difficult to guarantee SLAs.

When we calculate dollars per query weighted by performance, we can clearly see that Jobs Compute on classic clusters is the most cost efficient compute:

As you can see in the table above, the data indicates that Jobs Serverless comes out much lower on cost-efficiency compared to Jobs Classic. 

The data also indicates that P99s skyrocket for Classic Jobs when both Spot and Photon are disabled.

Following the data, we decided to take a closer look at the different configurations tested. Throughout this article we use Jobs Classic cluster with 8 workers, Photon enabled and Spot with fallback to on-demand as the baseline configuration. We systematically disabled these features to reveal the true cost structure for Jobs Compute Classic.

See the impact of Photon and Spot instances have on the cost and performance of Classic Jobs in the table below. 

As a reminder, Photon is Databricks’ native C++ vectorized query acceleration engine and Spot instances are highly discounted unused cloud compute.

Key takeaways: 

1. Enabling Photon increases the DBU/hr consumption rate. As a result, we found that Photon adds a 43% DBU premium (60 vs 42 DBUs) for 2x performance.

2. Disabling Photon lowers total cost by 23% but doubles the P99 latency.

3. Spot instances are critical for performance, without them P99 balloons to 285 seconds.

Interestingly, while we also found that the use of Spot impacts performance, it does not significantly impact unit economics. DBU consumption, cost per query and total costs for Jobs Classic with and without Spot are comparable.

Across nearly 5,000 TPC-DS queries at both scale factors, the pattern favors Jobs Classic vs Jobs Serverless.  

To recap our key findings:

1. Jobs Classic costs 27-78% less than the Serverless variants 

2. Jobs Classic delivers 1.8-2.8x better P99 performance 

3. Jobs Classic is far more predictable in terms of performance 

4. Disabling Photon will save 23% if you can tolerate the double P99 latency

The data shows that Jobs Classic comes out on top compared to the Serverless offering of the same compute type, but what happens when we pit it against serverless SQL warehouses? 

Now that we have established that Jobs Classic is more cost-effective than its Serverless counterparts, we need to address the elephant in the room. The TPC-DS SF1000 workload is all SQL, so how does it perform on Databricks compute specifically optimized to run SQL?

As you can see in the table above, every single serverless DBSQL warehouse configuration came out costing less than Jobs Classic, despite costing 4.67x more per DBU ($0.7/DBU vs 0.15/DBU): 

• Small SQL warehouse: Cost 82% less than Jobs Classic

• Medium SQL warehouse: Cost 74% less than Jobs Classic

• Large SQL warehouse: Cost 69% less than Jobs Classic

• XL SQL warehouse: Cost 38% less than Jobs Classic 

On top of costing less for the same workloads, DBSQL serverless warehouses are also dramatically faster. When we compare a medium sized SQL warehouse, for example, to Jobs Classic we see the following performance differential: 

• P50: 0.91s vs 5.41s (5.9x faster) 

• P90: 1.20s vs 26.86s (22.4x faster)

• P99: 2.05s vs 82.22s (40.1x faster)

With a 2-second P99 latency, serverless SQL warehouses are very capable of real-time analytics that a Jobs Classic cluster could not support. For instance, you could run up to 40 sequential queries on a serverless SQL warehouse in the time Jobs Classic compute takes to complete just one at P99.

While the cost per DBU for serverless SQL warehouses is higher than that for Jobs Classic, the data shows that serverless DBSQL is more cost-effective than Jobs Classic. The key to this seemingly counterintuitive pricing structure is all in the computation efficiency. SQL warehouses consumed just 4 DBUs on a medium warehouse compared to the 60 consumed by the Jobs Classic: a 15x improvement which more than offsets the cost. In fact, if we look at the cost per query percentile second which is an efficiency metric we can see: 

• Classic Jobs: $10.22/82.22 seconds = $0.124 per P99 second 

• Medium serverless DBSQL warehouse: $2.70/2.05 seconds = $1.37 per P99 second 

On the surface it looks like you’re paying $1.37/second vs around $0.12/second but when you take into consideration the P99 of 2 seconds vs 82 seconds the savings disappear as the workload takes 40x less time to run. As a result a medium serverless warehouse costs $2.7 vs $10.22 to run the same queries using Jobs Classic.  

The data shows that the same workload under the medium sized serverless SQL warehouse will be 40x faster and cost 74% less than on classic jobs clusters.

DBSQL warehouses achieve this efficiency through a myriad of optimization, but some features in particular make them particularly competent: 

• Native Photon acceleration included in the base price (not togglable like with Jobs)

• Persistent compilation caches between queries 

• Columnar caching optimized for analytical patterns

On top of the efficiency, the Coefficient of Variation (CV), which measures the performance predictability, is significantly more stable for serverless DBSQL warehouses. The CV is calculated as standard deviation divided by the mean, which makes it better than standard deviation alone when comparing performance consistency across compute types.

For example, a medium sized serverless SQL warehouse has a mean latency of 1.15 seconds and standard deviation of 0.29 seconds, so the standard deviation is only 25.3% of the mean which indicates very tight consistent performance. In contrast, Jobs Serverless - Standard achieved a mean latency of 42.5 seconds with a standard deviation of 86 seconds. The standard deviation here is more than 2x the mean indicating wild swings, where slow queries can take 3-4x longer than average.

This Databricks benchmark report clearly show that the serverless SQL warehouses aren’t just for analytics, they can be the optimal choice for batch processing too. But it's not a hard/fast rule. Here’s a breakdown of the workloads that are better suited for DBSQL: 

1. Pure SQL transformations

2. dbt models and SQL-based ETL

3. Data quality checks and testing 

4. Reporting and BI workloads 

5. COPY INTO and data ingestion 

6. Large-scale aggregations 

Jobs Compute, on the other hand, is still very much necessary for patterns like these: 

1. R/Python/Scala UDFs and Complex Logic

2. Machine learning windows

3. Streaming applications

4. Custom libraries and dependencies 

5. File format conversions

After running close to 5,000 of TPC-DS queries and conducting rigorous statistical analysis, this Databricks benchmark report reveals somewhat surprising patterns that might challenge the conventional narrative. 

First, we found that Jobs Serverless is less efficient than Jobs Classic despite using fewer resources. In fact, we found that the serverless variants consume 13-38% fewer DBUs than classic Jobs, yet deliver slower performance. This suggests there might be an architectural inefficiency that prevents serverless from matching the performance of classic job clusters even when the comparison should favor it. Combined with the 2.33x higher DBU rate ($0.35 vs $0.15), you’re paying 27-78% more for slower runtimes.

Secondly, we found that serverless SQL warehouses are the bargain we need. Despite having a higher price point, $0.70 per DBU versus Jobs’ $0.15 - a 4.67x premium, they end up costing less due to consuming fewer DBUs. 

Overall, our TPC-DS benchmark report reveals an uncomfortable truth: serverless compute isn't always an upgrade. For the vast majority of production data workloads, such as predictable ETL pipelines, scheduled analytics and SLA-driven processing, Classic Jobs deliver better performance, lower costs and more reliable execution than the serverless options.

The real winner? Serverless SQL warehouses. They're not just for BI anymore. For analytical SQL workloads, they're 7x faster than any Jobs compute type at equal or lower cost.