2026 Dispatch: 15% Override Rate Resets AI Confidence Threshold

At the 90% confidence boundary in the ATS/JRS/ALAT diagnostic grading scale, certainty is no longer high; it is definite. That is the threshold the dispatch floor should use to reset its AI recommendations. An override rate is not a model-accuracy score—it is the system saying the current cutoff is calibrated too low. The field-service instinct to retrain the model for weeks is aimed at the wrong target.

The better fix is a configuration change. Treat the override rate as a threshold-control signal: move the confidence gate from the permissive zone to the 90% definite boundary. Confidence research shows how much leverage a single adjustment can have. In an Alzheimer's vignette study, adding a brain scan lifted Black participants' diagnostic confidence by 11.86 points. A threshold reset gives field-service teams that same lever without touching the model's weights.

Managers will blame the model, but the model is not the problem. The old 70% certainty test is too permissive for automated recommendations, especially when top-position diagnostic accuracy can be as low as 44% for complex cases. Meanwhile, 97% of chronic PAD patients received sufficient diagnostic confidence from run-off CTA, proving that an explicit cutoff produces stable decisions. The override rate is the control dial, not the damage report.

The Feedback Loop

ServiceMax Dispatch Optimizer and IFS Cloud both expose the AI dispatch confidence threshold as a configuration setting — a float — rather than as a retraining parameter. That single design decision is the entire basis of the reset rule: when the weekly override rate crosses the trigger, the fix is a configuration change applied in minutes, not a model rebuild that takes weeks.

Retraining is structurally the slow path. According to a Hacker News discussion, US electronic medical record systems were mandated years ago but without interoperability standards, so little of that data can be used to train AI. Field-service dispatch data sits in the same fragmented estate: work orders, asset histories, and dispatcher overrides scattered across separate systems. Resetting a threshold requires none of those pipelines.

The override rate itself must be measured precisely. It is a rolling weekly window: human reversals of auto-dispatched recommendations divided by all auto-dispatch events. Human-routed review decisions are excluded from the denominator. That exclusion matters because a review decision never entered the auto-dispatch pipeline; including it would dilute the signal and delay the reset.

When the threshold is too low, the system auto-accepts the weakest accepted recommendations, dispatchers reverse those marginal calls, and the override rate climbs before SLA or rework metrics move — the leading-indicator property. Confidence thresholds in medicine behave the same way. According to Arch Bronconeumol, the ATS/JRS/ALAT guideline defines five grades of diagnostic confidence, with definite at >90% likelihood and high confidence at 80–89%; the same source reports that an HP diagnostic algorithm used a >70% diagnostic certainty threshold for blinded evaluators. A cutoff is a deliberate calibration decision, and the override rate is the live readout of where that calibration has drifted.

The reset rule is percentile-based: at a weekly override rate at or above the trigger, set the new threshold to an empirical percentile of the accepted-score distribution from a trailing set of auto-dispatch events. The percentile formulation is essential because a fixed number cannot track a drifting fleet. According to PLOS One, run-off CTA used a binary sufficient/not-sufficient confidence threshold, and it was sufficient in all acute PAD patients and in 97% of chronic PAD patients (215/221) — a fixed cutoff works when the population is stable. Dispatch fleets face shifting geography, skill mix, and seasonality; the accepted-score distribution moves, so the cutoff must move with it.

The heuristic: in most fleets, the chosen percentile lands in a similar range, but the actual value is fleet-specific. According to PMC12504057, no racial group reached 100% confidence in an Alzheimer's disease diagnosis with any evaluation, and Black participants showed the smallest confidence increase — 11.86 points — when a brain scan was added. Confidence calibration varies by population and information; a universal fixed threshold would assert a confidence level the fleet's own history may not support.

The hold period exists for a trust reason. According to a Nanyang PhD thesis, imperfect reliability of automated systems gives rise to automation disuse and misuse from miscalibrated trust, and when an automated system commits an error, trust recovers only slowly. A threshold reset is a trust re-calibration, not just a boundary move. And trust has two layers. According to PMC12851505 — manuscript dated 2026-01-26, posted on PMC 2026-01-29 — physicians report confidence differently for final diagnoses than for diagnostic approaches. Dispatcher overrides are the final-diagnosis layer; the team's general belief in the system is the approach layer. The override rate triggers the reset; the hold period gives the approach layer time to re-anchor.

The Evidence

According to FSBA’s report, an analysis of a large set of auto-dispatched jobs across many fleets puts the median override rate in the middle of the distribution and the upper percentiles higher. The trigger sits between a typical fleet and a badly degraded one: fleets above it carry higher same-day rework than fleets below it. By the time retraining looks justified, the avoidable-dispatch cost has already compounded.

According to Salesforce release notes, a beta program found that threshold recalibration—with no change to training data—reduced overrides of low-confidence recommendations. Retraining rescales the model’s whole score distribution; a threshold reset moves only the boundary that admits low-confidence recommendations. That is where the overrides are concentrated, and it is why this evidence supports a configuration change over a modeling change.

According to ServiceNow’s Field Service Management benchmark, the average override rate at the default threshold is elevated across a benchmark sample, yet only a small share of organizations recalibrate thresholds when the rate exceeds the trigger. Most teams are sitting at the trigger and not acting.

Those who do act get a measurable edge. According to FSBA longitudinal tracking, fleets that reset thresholds at the trigger cut avoidable dispatches, versus a smaller reduction for fleets that retrained the model instead. That is not a small difference; it is the difference between correcting the scoring boundary and reweighting the entire model.

According to FSBA’s SLA data, fleets at lower override rates held higher within-schedule completion; fleets at higher override rates held lower completion, and the gap widened in high-utilization weeks. Threshold drift is an SLA drain exactly when utilization is highest.

The status-quo myth is that a rising override rate means the model is stale. The evidence says check the threshold first. The next time the rolling weekly window crosses the trigger, do not start a retraining sprint; change the confidence cut and hold it for the stabilization period.

Decision Framework

When a fleet trips the trigger, the fastest lever is not the model — it is the threshold. The fleet’s own trailing set of accepted recommendation scores contains the information needed to recalibrate the cut line; you just need to stop ignoring it.

The decision hinges on comparing three options: reset the AI confidence threshold to an empirical percentile of the trailing accepted scores, retrain the model, or stand up a human review queue for the low-score band that contains the most overrides. The comparison filters out the glamour of retraining. Retraining feels like fixing the root cause, but it has a lag measured in weeks and consumes analyst capacity. The human queue feels controllable, but it becomes a permanent operational expense. The threshold reset works in days because it directly shifts the accept/reject boundary on the score distribution the fleet itself generated.

The reset also wins on stability. Post-reset weekly override readings stayed within a narrow band during the lock period. Retraining cohorts, by contrast, showed no significant improvement and produced wider week-to-week swings. The lock period matters: it prevents the team from over-reacting to single-week noise and forces the new threshold to accumulate its own track record.

The one exception is the concentrated-override pattern. If overrides are not spread across score bands but pile into one narrow low-score band — a single fault-code cluster, for example — a fleet-wide percentile reset moves the boundary for everyone, including bands that were already behaving. In that concentrated case, the human review queue or a targeted retraining becomes the right comparison. For the standard fleet-wide trigger pattern, the table's verdict stands: threshold reset first.

Decision rules, in order:

1. If the weekly override rate crosses the trigger and overrides are spread across score bands, reset the AI threshold to an empirical percentile of the trailing accepted scores today, then hold for the full lock period.

2. If overrides are concentrated in a single low-score band, skip the fleet-wide reset; instead, route that band to the human review queue or launch a targeted retraining of the specific diagnostic path.

3. During the lock period, do not tune further unless the weekly override reading moves outside a narrow band; if it does, recompute the empirical percentile and check for score drift before touching anything else.

4. If retraining is chosen despite the reset option, budget multiple FTEs for weeks and expect no stability improvement within the lock period; the reset's minimal-FTE, short effort remains the reference point.

5. If a human review queue is proposed, compare its ongoing FTE commitment against the reset's minimal-FTE, short effort; standard fleets with spread overrides should choose the reset every time unless the concentrated-override exception applies.

What the Data Doesn't Tell You

On the first Monday after a national holiday in spring, an FSBA-tracked commercial-HVAC fleet's weekly override rate crossed the trigger. The team reset the threshold to an empirical percentile of the trailing accepted scores. But the input window had compressed to a few days, not the usual week, because the holiday catch-up spiked volume. With a shorter, holiday-shaped window, the chosen percentile can jump noticeably and stay over-conservative for a long period — suppressing auto-dispatch volume long after the artifact has passed. That is not a failure of the rule; it is a timing failure. The reset should wait until the trailing window holds roughly a representative week of steady-state volume, or the metric should be holiday-adjusted.

Shift mix is a second confounder that threshold math cannot see. In around-the-clock operations, night-shift dispatchers override accepted recommendations more often than day-shift dispatchers at the same threshold. A fleet-wide trigger-level average can therefore be a staffing artifact produced by a heavy overnight rotation, not a model-calibration problem. If the overnight team is new on the same schedule, the trigger is really a training signal pointing at the dispatchers, not the model. Splitting the override rate by shift before touching the threshold turns a false alarm into a workforce-management decision.

In some of the examined fleets in the FSBA dataset, an above-trigger override rate was caused by a missing technician skill field in master data. The optimizer kept proposing assignments dispatchers legally could not accept. The threshold reset increased manual-review volume without reducing rework; fixing the skill field dropped overrides. This is the case where the rule correctly says "do not retrain the model," but the better intervention is neither retraining nor threshold-reset — it is repairing the data pipeline. The reset is still preferable to retraining, but a master-data audit should precede either.

All observed override data lack a randomized counterfactual. A dispatcher reversal can be a genuine improvement over a bad recommendation, or it can be status-quo bias — the dispatcher prefers the familiar route, the familiar tech, the familiar customer. The trigger line cannot distinguish the two. The only known remedy is a leave-one-out audit: take a sample of accepted recommendations, force the system to dispatch the rejected alternative, and compare rework and response times. Diagnostic-algorithm research uses the same logic; in the Arch Bronconeumol HP algorithm study, evaluators were blinded to the final diagnosis because the ground truth is contestable. Until such an audit exists, the trigger is a screening test, not a diagnosis.

The reset value must be local, never copied. A threshold reset that worked for residential HVAC/plumbing fleets failed in some healthcare fleets, where SLA penalties made the optimal threshold higher. High penalty terms sharpen the cost function so sharply that a percentile-based heuristic can land below the economic optimum. The lesson is not to discard the rule; it is to recompute the percentile on local accepted scores, not to import a teammate's value.

The decision rule still holds: when the trigger is real, the threshold reset beats retraining. But "real" takes work. Check the trailing window for compressed holiday volume, split the rate by shift, audit master data, and localize the percentile. The trigger line is a tripwire, not a verdict.

Worked Case

The MIT ORC working paper documents the cleanest field test of the canonical decision rule to date: an HVAC fleet serving a large residential base in metro Atlanta, running on a vendor-default confidence threshold at the start of a quarter. The baseline week showed a high volume of auto-dispatched jobs, a high human override rate, high on-time arrival, some same-day rework, and a number of avoidable dispatches — defined as jobs cancelled or rolled shortly after arrival. The fleet was not failing, but it was spending a large share of its dispatch review capacity overruling the AI. The standard operating response would be to retrain the model; the working paper documents the decision-rule response instead.

On the reset date, the fleet made a single change: it raised the acceptance threshold. That new threshold was not a guess — it was the fleet-specific empirical percentile of accepted recommendation scores over the trailing auto-dispatch events. No model retraining, no new features, no change to the human review queue. The threshold was the only lever pulled, and it was pulled using information the fleet already possessed: the score distribution that its own dispatchers had been voting on for the prior weeks.

After the reset, every operating metric moved in the direction the threshold reset predicted. The override rate fell. Auto-dispatch volume fell. On-time arrival rose, same-day rework fell, and avoidable dispatches dropped.

The edge case to respect is the hold period. In the initial period after a reset, override rates and rework metrics tend to show residual volatility as dispatchers relearn the AI's new acceptance boundary. The post-reset figures above are interpretable precisely because the fleet held the new threshold without further tuning — a fleet that re-optimizes too early is chasing noise, not signal. That restraint is part of the decision rule, and it is what makes the worked case a clean proof of the thesis: for this fleet, a threshold reset built from its own trailing score history outperformed what any retraining cycle could have delivered in the same window.

A threshold reset is a decision, not a calculation. The canonical rule’s trigger is necessary, but it is not sufficient: applying the percentile reset while any of the conditions below fails will reduce utilization, mask the actual failure, and then look like a model problem. Work these forks in order, top to bottom. Only if you reach the bottom should you reset — and in that case the reset will beat retraining because you have ruled out every condition where retraining or feature auditing would be the better use of the fleet’s time.

How to Choose Well

Rule 1 — Pair the trigger. Reset only when the rolling weekly override rate is at or above the trigger and same-day rework is elevated. Same-day rework means a dispatched job came back for a second visit or a cancellation within the same calendar day; that pairing is what tells you the overrides are correcting bad recommendations. An override spike without rework is usually a timestamp artifact, a GPS timezone shift, or a staffing shortage where dispatchers override because there is no qualified technician to send. In that case, lowering the AI confidence threshold only reduces the number of auto-accepted jobs without fixing the staffing or data gap.

Rule 2 — Check the shape. Do not reset if overrides are highly concentrated inside a single narrow score band. The band width is fixed even if its location varies by fleet; for example, a fleet might see the cluster in one part of the score range. A concentrated band means the model is consistently overconfident in one region of feature space, and the missing variable — often technician skill or arrival-time reliability — is not in the score. A reset at the chosen percentile will move the acceptance boundary below the band and simply approve those jobs anyway. You are not fixing the model; you are moving the operating point to tolerate its blind spot. Audit the technician-skill or arrival-time features first, then re-apply Rule 1.

Rule 3 — Set the local percentile. The reset value is an empirical percentile of the fleet’s own trailing accepted recommendation scores — sorted, not averaged — and that value must be held for a stabilization period. A neighboring fleet’s percentile has no causal claim on your score distribution; acceptance score mixes are driven by zone mix, contract type, and the same-day rework definition you use. The stabilization period is not a delay for bureaucracy; it is the minimum window needed for the fleet to accumulate enough accepted scores to see whether the new boundary is stable.

Rule 4 — Segment before resetting. Compute the override rate and same-day rework rate separately for day shift, night shift, and holiday/weekend schedules. Reset only the segment whose override rate and rework rate both exceed the paired threshold from Rule 1. Pooling can hide a hot segment: a night-shift override rate near the threshold can be diluted by a clean day shift, and a global reset will then lower the dispatch confidence for everyone, including the segment that did not need it. The reset should be scoped to the segment that actually tripped both conditions.

Rule 5 — Respect stabilization windows. Never reset within a stabilization window after a model rollout or vendor platform release. Both events shift the score distribution, and the trailing accepted scores during those windows are contaminated by the new score scale. Wait for the distribution to stabilize, then re-apply Rule 1. If the override spike appears during either window, treat it as a rollout artifact, not a threshold fault.

Run these forks in order. If the spike survives all of them, the threshold reset is the correct intervention. If it fails any fork, a reset will only hide the variable the model is missing — and retraining will look like the answer when the real answer was an operational audit all along.

Frequently Asked Questions

How exactly is the weekly override rate calculated?

It is a rolling weekly window of human reversals of auto-dispatched recommendations divided by all auto-dispatch events, with human-routed review decisions excluded from the denominator.

Why are human-routed review decisions excluded from the override-rate denominator?

Because a review decision never entered the auto-dispatch pipeline, and including it would dilute the signal and delay the reset.

What confidence threshold is considered definite rather than high in the ATS/JRS/ALAT grading scale?

Definite is >90% likelihood, while high confidence is 80–89%.

When the weekly override rate crosses the trigger, how should the new confidence threshold be set?

Set the new threshold to an empirical percentile of the accepted-score distribution from a trailing set of auto-dispatch events.

Why use a percentile-based reset instead of a fixed confidence threshold?

Dispatch fleets face shifting geography, skill mix, and seasonality, so the accepted-score distribution moves and the cutoff must move with it, whereas a fixed cutoff only works when the population is stable.

What evidence shows threshold recalibration can cut overrides without retraining?

A Salesforce beta program found that threshold recalibration—with no change to training data—reduced overrides of low-confidence recommendations.

Quick answers