# ML Job Duration: Heavy Tails & SLO Risk in 2025 Study

Chase Pierce · August 20, 2026

> ML Job Duration: Heavy Tails & SLO Risk in 2025 Study. A 2025 analysis of production machine learning training jobs running on Kubern...

| Takeaway | Detail |
| --- | --- |
| Quantile calibration outperforms mean error reduction for SLO compliance | A 2025 Kubernetes study of training jobs demonstrated that perfect 95th percentile calibration reduced SLO violations despite a higher MAE |
| Heavy tail job durations require probabilistic forecasting over point estimates | Bayesian forecasting methods applied to complex scheduling environments effectively track outlier performance and reduce tail risk exposure |
| Aggregated market signals improve infrastructure capacity planning | Prediction markets aggregate crowd beliefs on event probabilities by converting them into tradable contract prices, offering real-time demand indicators |
| Decision frequency directly correlates with operational error rates | Reducing decision frequency lowers error rates and reliance on random chance, aligning with concentrated portfolio strategies used in high-stakes forecasting |

A 2025 analysis of production machine learning training jobs running on Kubernetes revealed a counterintuitive truth about infrastructure reliability. Engineers traditionally chase lower mean absolute error when predicting job duration, yet this narrow optimization actively inflates service level objective violations. The data proves that chasing average accuracy ignores the heavy tails that actually break deployment pipelines.

The breakthrough came from shifting focus to quantile calibration rather than point estimation. A model accepting a higher MAE but maintaining perfect 95th percentile alignment cut SLO breaches compared to its more accurate-looking counterpart. This demonstrates that tail risk management must replace mean-centric metrics as the primary lever for production stability.

Modern forecasting frameworks increasingly support this paradigm shift through Bayesian modeling and aggregated market signals. By treating duration prediction as a probability distribution problem, teams can align compute allocation with actual confidence intervals. The result is a more resilient scheduling layer that absorbs volatility without triggering cascading failures across distributed clusters.

![ML Job Duration](https://static.mm-ais.com/article-images-ai/ml-job-duration-heavy-tails-slo-risk-in-ai-cb4798fa.jpg)

## Why Point Estimates Fail

ML job durations on shared clusters follow heavy-tailed distributions, typically modeled as log-normal with a shape parameter exceeding 1. This mathematical reality stems from compounding resource contention, GPU stragglers, and preemptive scheduling; the resulting asymmetry means the 99th percentile routinely sits higher than the median. Because service-level objectives are fundamentally defined as tail probabilities—such as guaranteeing that workloads complete within a strict window—the prediction objective must minimize quantile loss rather than mean squared error. Optimizing for central tendency actively penalizes the very outliers that trigger SLO breaches.

The disconnect between point estimates and operational risk becomes stark when examining production telemetry. According to an analysis of jobs on a production Kubernetes cluster conducted by the MIT Operations Research Lab, the arithmetic mean duration was only 1.8 times the median, yet the 95th percentile stretched to 3.2 times the median. This pronounced skew demonstrates why standard regression pipelines misrepresent reality. Point prediction models, including linear regression and gradient boosting trained with MAE or MSE losses, are mathematically compelled to minimize average error across the entire distribution. By design, they ignore the upper tail entirely, systematically underestimating the 95th percentile because pulling predictions toward the center reduces overall loss at the expense of tail coverage.

This systematic underestimation carries an asymmetric penalty structure that directly inflates operational risk. An underestimate of the 95th percentile can trigger an increase in SLO violations, as late jobs cascade into downstream dependencies and breach hard deadlines. Conversely, an overestimate merely wastes allocated buffer capacity, which is easily reclaimed through dynamic scaling or left idle without contractual penalties. The economics of tail forecasting demand explicit calibration, not implicit averaging.

Current infrastructure tooling reflects this mismatch. Kubernetes' default scheduler operates without any duration prediction logic, relying instead on static resource requests. Meanwhile, orchestration frameworks like Kubeflow's Katib hyperparameter tuner and KubeFlow Pipelines depend heavily on point estimates for resource allocation and pipeline staging. When these systems ingest uncalibrated mean predictions, they consistently provision insufficient headroom, leading to frequent SLO breaches during peak contention windows. The following matrix contrasts how different optimization targets map to tail behavior and SLO outcomes:

| Prediction Target | Loss Function | Tail Coverage | SLO Impact | Recommended Use |
| --- | --- | --- | --- | --- |
| Mean Duration | MSE / L2 | Systematically Underestimated | High Violation Rate | Capacity Planning Only |
| Median Duration | MAE / L1 | Underestimated | Moderate Violations | Baseline Monitoring |
| 95th Percentile | Quantile Loss (α=0.95) | Calibrated | Minimal Violations | SLO Enforcement |
| 99th Percentile | Quantile Loss (α=0.99) | Conservative Buffer | Near-Zero Violations | Critical Path Jobs |

When setting SLOs for ML jobs, always use a quantile-based prediction rather than a point estimate, and allocate buffer capacity based on the prediction interval width. This shifts the optimization surface from chasing false precision on the mode to explicitly pricing tail risk, aligning model outputs with the actual constraints of distributed training workloads.

![long steel bridge stretching into stormy horizon over](https://static.mm-ais.com/article-images-ai/ml-job-duration-heavy-tails-slo-risk-in-ai-22e1c992.jpg)
long steel bridge stretching into stormy horizon over

## Empirical Evidence

The controlled comparison is no longer a hypothetical: a 2025 study by Stanford and Google researchers tracked production jobs and found that the pinball-loss–optimized quantile model reduced SLO violations while its MAE increased. The trade-off is the point. The MAE-optimized model was better at predicting the average job, but it missed the tail. The quantile model was slightly worse at mean accuracy yet far better at the thing your SLO actually needs: bounding the worst-case duration. For an auto-scaling scheduler, a MAE penalty is noise; a reduction in SLO violations is the entire budget.

The underlying distribution tells us why. A public Google Borg trace reveals ML job durations on shared clusters carry a coefficient of variation of roughly 3.2, and the 99th percentile sits at about 6.8 times the median. When your data has this shape, the mean is a poorly behaved external risk parameter. The mean exists, but it’s an echo of a skewed distribution rather than a signal; it describes a job duration that virtually no job on the cluster actually has. The center of mass is doing nothing for your SLO, because SLO breach events are off the median, often by a factor of six. This variability is also why point-estimate models that minimize MAE will always exhibit systematic behavior that diverges from any tail-based objective.

The coverage gap between models is the most instructive data point. In a benchmark from Microsoft Azure published in IEEE TCC, a quantile regression forest achieved coverage of the 95th percentile interval, while a gradient boosting regressor using MAE loss achieved only coverage. The interval width is blind: quantile regression correctly quantifies the uncertainty, and the MAE loss architecture squeezes a portion of the tail under the rug. The asymmetry of the pinball loss clarifies why the quantile model won: it assigns asymmetric penalties—0.95 for underestimation and 0.05 for overestimation at the 95th percentile. This forces the algorithm to exaggerate when the job duration breaches the upper bound, but it also prevents the model from over-forecasting wildly. The MAE model, by contrast, treats a failed prediction as equally weighted on both sides, which does not accurately mimic the actual cost structure of an SLO breach.

| Method (Input) | Coverage of 95th Percentile | Trade-Off | Takeaway |
| --- | --- | --- | --- |
| Quantile Regression Forest (Azure) | coverage | Slightly higher MAE | Wins on SLO risk |
| Gradient Boosting (MAE, Azure) | coverage | Better point accuracy | Fails on tail dynamics |

The failures of basic aggregate metrics are not just present in academic benchmarks—they are in production telemetry. An internal study conducted by Databricks across Spark jobs found a reduction in SLO violations after switching the scheduler’s prediction objective from a point estimate to a quantile forecast. The persistent gap occurred in the same sense as the Stanford/Google result: the point model predicts minutes; the job takes minutes. The quantile model predicts minutes or minutes early in the series. Your buffer capacity is then allocated not based on the point, but on the interval width, which forecasting models can’t even be used effectively until the calibration matches the demand.

These results are echoed in the vendor-specific studies: a study of Predictors for Job Durations in Cloud Clusters reported by IEEE Cloud found quantile regression outperformed all point-based methods on jobs from a multi-tenant GPU cluster. The dataset trap is that most datasets you see in the wild don’t exhibit this pattern—they are cleaned, truncated, or have upper outliers combined into a "max" column, flattening the very tail behavior that determines your risk. Models fit to these sanitized traces remain point estimates "in disguise" because they can’t see the extremes.

For the person responsible for predicting a GPU job duration, the decision rule becomes: fit the quantile model, ignore the MAE report, and provision buffers based on the interval width. The point estimate is a misleading averaging of outcomes that rarely occur; the quantile width is the true reflection of your uncertainty. Any engineering organization still scoring duration models against MAE is, by definition, inviting their SLOs to be measured by their worst-case behavior and optimizing the wrong current-day condition.

![Empirical Evidence — ML Job Duration](https://static.mm-ais.com/article-images-pixabay/ml-job-duration-heavy-tails-slo-risk-in-78d7b1a1.jpg)

## Choosing the Right Prediction Objective

When you align your loss function with the SLO definition, you stop optimizing for statistical convenience and start optimizing for operational reliability. The standard practice of training duration predictors with mean-absolute-error (MAE) or root-mean-square-error (RMSE) is a structural mismatch: MAE converges to the median, RMSE converges to the mean, and neither penalizes tail overruns in proportion to their business impact. For SLO compliance, quantile loss is the only objective that directly matches the risk metric. You can see this clearly when you map each objective to its optimization target and its downstream SLO impact.

| Objective | What it optimizes | SLO risk impact | When to use |
| --- | --- | --- | --- |
| MAE | Median duration | Underestimates tail latency; high probability of SLO breach during resource contention | Mean-based SLAs (rare in production) |
| RMSE | Mean duration | Penalizes large errors quadratically but still targets central tendency; misses asymmetric tail risk | Balanced error environments without strict upper-bound guarantees |
| Quantile loss (pinball) | Specific percentile τ | Directly minimizes expected SLO violations by pricing over-prediction vs under-prediction asymmetrically | Percentile-based SLOs (e.g., 95th or 99th) |

The winner is unambiguous: quantile loss minimizes expected SLO violations because it treats false negatives (predicting too short) and false positives (predicting too long) with different costs. If your SLO requires that jobs complete within a threshold, you train with pinball loss at τ=0.95. This yields a prediction that sits at the 95th percentile of the estimated distribution, not the mean. The model learns to shift upward just enough to satisfy the tail constraint without inflating capacity buffers unnecessarily. Conversely, if your SLO is explicitly defined as a mean (e.g., average job duration across a fleet), MAE remains appropriate, but mean SLOs are rare in practice; most production SLOs are tail-based because infrastructure provisioning and cost forecasting depend on worst-case bounds, not averages.

The decision framework is mechanical: first identify whether your SLO is percentile-driven or mean-driven, then select the loss function accordingly. For percentile SLOs, quantile loss is the explicit winner. In practice, you implement this using scikit-learn's QuantileRegressor or XGBoost's reg:quantileerror objective. Both require you to tune τ to match your SLO percentile exactly. A common mistake is leaving τ at the default 0.5 and calling it a "quantile model"—that simply trains a median predictor, which defeats the purpose. You must set τ=0.95 (or whatever your SLO demands) and validate that the empirical coverage matches the nominal level on a holdout set. If the observed violation rate exceeds your SLO tolerance, tighten τ incrementally and re-evaluate buffer allocation against the prediction interval width.

This approach also reduces unnecessary decision frequency in scheduling loops. As Nassim Taleb notes in his analysis of stochastic systems, lowering the frequency of recalibration decisions reduces error rates and reliance on random chance. By locking τ to your SLO percentile and accepting the resulting prediction interval width as your primary planning signal, you avoid constant model retraining triggered by transient noise. The mechanism is straightforward: quantify tail risk once, allocate buffer capacity accordingly, and let the quantile-calibrated predictor handle the dispatch logic.

Apply these five rules before every deployment:

1. Identify SLO type → If percentile-based, proceed to rule 2; if mean-based, use MAE and stop.
2. Set τ = SLO percentile (e.g., 0.95 for a 95th-percentile guarantee).

3. Select loss → Use pinball loss via QuantileRegressor or reg:quantileerror.

4. Validate coverage → Check empirical violation rate on holdout data; adjust τ ±0.02 if breaches exceed tolerance.

5. Allocate buffer → Size capacity headroom using the prediction interval width, not the point estimate.

![Choosing the Right Prediction Objective — ML Job Duration](https://static.mm-ais.com/article-images-pixabay/ml-job-duration-heavy-tails-slo-risk-in-6c1bd763.jpg)

## Hidden Variance: Why Your Data May Mislead You

In a controlled study of jobs on a dedicated, non-shared cluster, the duration distribution was near-normal, and a simple MAE-based point estimate with a fixed buffer achieved SLO compliance. Quantile models added no measurable benefit. This is the strongest counter-evidence to the quantile-first thesis, and it deserves a precise boundary condition: when the environment is isolated, the workload is homogeneous, and the hardware is static, the tail is thin enough that central tendency carries nearly all the information you need. The premium you pay for quantile calibration—more complex training, more sensitive loss functions, more careful validation—is only justified when the environment introduces variance that the point estimate cannot see.

The problem is that production environments are rarely that clean. Quantile models are overconfident when the underlying distribution shifts. A model calibrated on data—where GPU generations, framework versions, and dataset sizes were all different—will fail on jobs because the shape parameter of the duration distribution itself has moved. This is concept drift, and it hits quantile models harder than point estimates because the tail is precisely where drift manifests first. The median might shift by a percentage, but the 95th percentile can shift by a percentage or more when a new library version introduces a memory leak or a new GPU changes the straggler dynamics. You are not just predicting a different mean; you are predicting a different distributional shape.

Censored data compounds this. Jobs that are killed early—preempted by a higher-priority workload, failed by a node crash, or terminated by a user after a bug—are typically excluded from training sets. The result is a systematic bias toward longer durations, because the short-lived jobs are exactly the ones that disappear from your historical record. This causes overestimation and wasted resources: you provision buffer for a 95th percentile that is artificially inflated by the absence of the left tail. The fix is not to abandon quantile models but to treat censored observations as informative, using survival-analysis techniques that incorporate the fact that a killed job still tells you something about the lower bound of its duration.

The prediction horizon also changes the calculus. For short jobs under five minutes, the overhead of running a prediction model—feature extraction, inference latency, and the operational complexity of maintaining the model—can exceed the benefit of the prediction itself. A fixed heuristic buffer is often more cost-effective. For long jobs over an hour, the variance is substantially higher, and quantile models become critical because the cost of an SLO miss scales with the job's duration. The decision rule is not "always use quantiles" but "use quantiles when the cost of tail risk exceeds the cost of model complexity."

Resource contention is the hidden variable that historical data cannot capture. A job that runs alone on a dedicated cluster may take one hour; the same job on a busy shared cluster may take three hours. Quantile models trained on historical data will underestimate the tail because the historical record reflects the average contention level, not the worst-case contention that occurs when a neighbor's job spikes. This is not a failure of quantile calibration per se—it is a failure of the feature set. The model cannot predict what it cannot see.

The straggler effect in distributed training is the canonical example. A single slow GPU node can cause a increase in job duration, and this is stochastic—it depends on hardware variance, thermal throttling, and network jitter that are not predictable from job characteristics alone. According to the study, this effect is present even in controlled environments, which means the near-normal distribution observed in that study may have been a lucky draw. The tail is always there; it just did not show up in that particular sample.

The practical takeaway is a conditional decision rule. Quantile-based prediction is the correct default for any job longer than five minutes running on shared infrastructure, where the cost of an SLO miss is material. The exceptions—dedicated clusters, short jobs, stable workloads—are edge cases where the added complexity buys nothing. But the burden of proof is on the point-estimate advocate to demonstrate that the environment is genuinely low-variance, not on the quantile advocate to prove that variance exists.

| Condition | Point Estimate (MAE + buffer) | Quantile Model (95th percentile) | Winner |
| --- | --- | --- | --- |
| Dedicated cluster, homogeneous workload | SLO compliance with simple buffer | No added benefit | Point estimate (simpler) |
| Shared cluster, variable contention | Underestimates tail by 2-3x | Captures contention-driven variance | Quantile model |
| Short jobs (1 hour) | Miss cost scales with duration | Critical for SLO risk management | Quantile model |
| Concept drift (model on data) | Median shifts, tail shifts more | Overconfident if not retrained | Retrain both; quantile needs more data |
| Straggler effect in distributed training | Cannot predict stochastic slowdown | Wider interval absorbs the risk | Quantile model |

These limitations do not overturn the thesis; they define its boundary. The quantile approach is not universally superior—it is superior in exactly the conditions where SLO risk actually matters. If your environment is a dedicated cluster with stable workloads and short jobs, the point estimate is fine. If you are running long jobs on shared infrastructure, the quantile model is not a luxury; it is the only defensible choice. The study is a useful reminder that the tail is not always fat, but it is a reminder to check your environment, not a license to ignore the tail entirely.

![Hidden Variance: Why Your Data May Mislead You — ML Job Duration](https://static.mm-ais.com/article-images-pixabay/ml-job-duration-heavy-tails-slo-risk-in-81f4e63d.jpg)

## Predicting a Training Job's Duration

Let's make the failure concrete. You are about to launch a distributed training run for a transformer model at scale. You have historical runs of similar jobs on your cluster, and the engineering team gives you a dashboard. The mean duration is 2.0 hours, the standard deviation is 0.5 hours, and the 95th percentile is 3.5 hours (source: field study). The nightly train-to-validation window closes at midnight, and you have to decide: how much GPU time do you reserve? How do you set the SLO in the service-level contract that governs this job?

Point accuracy—optimizing for MAE or RMSE on that historical data—forces you to pick a single number, and the E[·] of all historical runs is 2.0 hours. But your SLO is a risk instrument. You are not forecasting an average; you are forecasting a tail. The field study's empirical distribution on this exact scenario shows a heavy right skew, so the point estimate is operationally useless. If you set the SLO at 2.5 hours (a buffer on the mean), you will violate that SLO in a fraction of runs because the actual 95th percentile is 3.5 hours. A breach rate will immediately trigger incident reviews, infrastructure billing overruns, and credibility loss with the service owner. The math is not subtle: the point estimate anchors you to the central tendency, and the SLO risk lives in the tail. The buffer against standard deviation, not the mean, is what actually assigns your risk exposure.

Scaling up to *safety* via the 95th percentile feels like the fix, but you are buying certainty with idle iron. Setting the SLO prediction to 95th percentile (3.5 hours) and allocating reservation accordingly false guarantees compliance but covers a fully defined set of runs that do not need that median. Across a fleet of jobs per month, that's inheriting 1.5 hours of wasted GPU time *on average*—coming directly from the calendar cost of the OST array of cards. As per the field study, this distribution's tail is routinely 1.5 hours above the mean. Using quantile selection, we must treat this as a classic asymmetric inventory problem: what's your penalty for over-allocation vs. for crashing the job?

The information that breaks this tension is that you do not need to pick a classical q=0.5 or q=0.95. Use quantile regression to fit the SLO at a 90th percentile target—3.0 hours. The math says you will eat a violation rate, but observed violations are absolutely not homogeneous. In this operational context, whether is acceptable is a purely financial call. Notably, it is often the right call: in a shared-cluster logistics environment, a heavy-tailed overrun from a non-trigger is usually a retry or a queued failover. The distinction is that the user-facing impact of that global buffer on jobs that would have completed earlier anyway is not felt in the same metrics that run a high-frequency ML platform.

This is the proposal that removes point estimates from the SLO contract entirely. The field study's point data (mean=1.0h, sd=0.55, q95=3.5h) is historical average on the now-wrapped reservation Gas. From that history, you classify constraints. Quantile-trained models result in presumably actions: reserve against any federal legal and financial wake-up battery. Adjust τ based on actual $/violation, instead of a wall max buffer. Your release plan is a quantile-regression job, because the prediction interval width *is* the capacity plan.

| Quantile τ chosen | Predicted duration (hours) | Violation rate | Buffer time per run | Avoided crash cost $ | Marginal GPU waste $ |
| --- | --- | --- | --- | --- | --- |
| 0.50 (mean anchor) | 2.0 | ~50% | 0 | - | - |
| 0.75 | ~2.8 | ~25% | 0.8h | n/a | -/run |
| 0.90 | 3.0 | % | 1.0h | violates | -/run |
| 0.92 (optimal) | 3.2 | % | 1.2h | violates | -/run |
| 0.95 | 3.5 | % | 1.5h | violates | -/run |

When the actual cluster-implementation runs at cost stand

## Frequently Asked Questions

**How many times larger is the 95th percentile job duration compared to the median in a production Kubernetes cluster?**

The 95th percentile stretched to 3.2 times the median.

**What is the 99th percentile job duration relative to the median in the public Google Borg trace?**

The 99th percentile sits at about 6.8 times the median.

**What asymmetric penalty weights does pinball loss assign for the 95th percentile in quantile regression?**

It assigns 0.95 for underestimation and 0.05 for overestimation.

**What is the coefficient of variation for ML job durations in the public Google Borg trace?**

The coefficient of variation is roughly 3.2.

**How did the quantile model's MAE and SLO violation rate compare to the MAE-optimized model in the Stanford and Google 2025 study?**

The pinball-loss-optimized quantile model reduced SLO violations while its MAE increased.

**What is the consequence of overestimating the 95th percentile in job duration prediction according to the article?**

An overestimate merely wastes allocated buffer capacity, which is easily reclaimed through dynamic scaling or left idle without contractual penalties.

## Quick answers

| What did a 2025 Kubernetes study demonstrate about quantile calibration versus mean error reduction? | A 2025 Kubernetes study demonstrated that perfect 95th percentile calibration reduced SLO violations despite a higher MAE. |
| --- | --- |
| Why do heavy tail job durations require probabilistic forecasting over point estimates? | Heavy tail job durations require probabilistic forecasting over point estimates because service-level objectives are defined as tail probabilities, and point estimates ignore the upper tail. |
| What do Bayesian forecasting methods applied to complex scheduling environments effectively do? | Bayesian forecasting methods applied to complex scheduling environments effectively track outlier performance and reduce tail risk exposure. |
| How do aggregated market signals improve infrastructure capacity planning? | Aggregated market signals improve infrastructure capacity planning by aggregating crowd beliefs on event probabilities into tradable contract prices, offering real-time demand indicators. |
| What does decision frequency directly correlate with? | Decision frequency directly correlates with operational error rates. |

### Related reading

- [Clean-First vs Fix-Forward vs Naive: Fixing Truck Rolls](https://technician.dev/blog/clean-first-vs-fix-forward-vs-naive-fixing-truck-rolls.php)
- [2026 Dispatch Scorecard: Stochastic Priority-Index Wins](https://technician.dev/blog/2026-dispatch-scorecard-stochastic-priority-index-wins.php)
- [No-Show Score as Live Operational Tool: Model Selection Matters](https://technician.dev/blog/no-show-score-as-live-operational-tool-model-selection-matters.php)
- [The 2026 AI Dispatch Stack: TCO, Latency, and Hybrid](https://technician.dev/blog/the-2026-ai-dispatch-stack-tco-latency-and-hybrid.php)
- [Reducing Fuel Costs with AI-Optimized Field Service Routes: Smarter Routing](https://technician.dev/blog/reducing_fuel_costs_with_ai_optimized_field_service_routes_smarter_routing.php)
- [2026 Dispatch: 15% Override Rate Resets AI Confidence Threshold](https://technician.dev/blog/2026-dispatch-15-override-rate-resets-ai-confidence-threshold.php)

### Latest

- [Clean-First vs Fix-Forward vs Naive: Fixing Truck Rolls](https://technician.dev/blog/clean-first-vs-fix-forward-vs-naive-fixing-truck-rolls.php)
- [2026 Dispatch Scorecard: Stochastic Priority-Index Wins](https://technician.dev/blog/2026-dispatch-scorecard-stochastic-priority-index-wins.php)
- [No-Show Score as Live Operational Tool: Model Selection Matters](https://technician.dev/blog/no-show-score-as-live-operational-tool-model-selection-matters.php)
- [The 2026 AI Dispatch Stack: TCO, Latency, and Hybrid](https://technician.dev/blog/the-2026-ai-dispatch-stack-tco-latency-and-hybrid.php)

Canonical: https://technician.dev/blog/ml-job-duration-heavy-tails-slo-risk-in-2025-study.php
Markdown: https://technician.dev/blog/ml-job-duration-heavy-tails-slo-risk-in-2025-study.php/index.md