Evidence, metrics, and the limits of dashboards
Use metrics to distinguish work performed from customer results and investigate misleading improvements.
Distinguish activities, leading indicators, and business outcomes, and identify one metric that could improve while the business gets worse.
Overview
At Alder Appliance Repair, the owner sees that average time per repair visit has fallen. The dashboard marks the result green. Before celebrating, she checks whether customers need a second visit for the same fault.
All businesses and numbers in this lesson are fictional teaching assumptions.
Recall Lesson 010: Riverton introduced an early materials check to improve shipment reliability. What assumption connected that action to the desired result? Try answering before continuing.
Activities, signals, and outcomes
A metric is a defined numerical measure. A dashboard presents selected metrics together. Its usefulness depends on which measures people select, how they calculate them, and what decisions they support.
An activity measure records work performed, such as calls made or inspections completed. It helps a manager check execution. More activity does not necessarily mean more value: a technician could complete a checklist without finding the fault.
A leading indicator is an earlier signal used to anticipate a later result. Confirming that a technician has the required parts before departure might signal a better chance of completing the repair on the first visit. That relationship is a hypothesis until supported by evidence.
A business outcome is a result the business or customer values, such as reliable repairs, retained customers, or profit. Measures of completed results are often called lagging indicators because they become available after the relevant work.
These categories can overlap. Parts checks completed are an activity count. The share of visits with all required parts could also be used as a leading indicator. Its role depends on the result you are trying to anticipate; the label does not guarantee predictive value.
The Institute for Healthcare Improvement recommends considering process, outcome, and balancing measures. A balancing measure checks whether an improvement in one area creates problems elsewhere. Applying that idea here, Alder should monitor repeat visits while trying to reduce visit time. Source: Institute for Healthcare Improvement.
Worked example
Alder compares two groups of 100 completed repairs. It follows each group for 14 days and records repeat visits for the same fault. Every repair has had the full follow-up period. Assume similar repair types and an unchanged recording method.
| Measure | Earlier group | Later group |
|---|---|---|
| Average minutes per initial visit | 60 | 45 |
| Repairs needing a repeat visit within 14 days | 10 of 100 | 25 of 100 |
| Additional cost per repeat visit | $40 | $40 |
Initial visit time fell by 60 − 45 = 15 minutes. Relative to the earlier 60 minutes, that is 15 ÷ 60 × 100 = 25% less time.
The repeat-visit rate rose from 10 ÷ 100 × 100 = 10% to 25 ÷ 100 × 100 = 25%. The increase is 15 percentage points. Customers are more often experiencing an unresolved fault.
Repeat-visit costs rose from 10 × $40 = $400 to 25 × $40 = $1,000, an increase of $600. The speed measure improved while repair reliability and repeat-visit costs worsened.
We cannot calculate the total profit effect without knowing whether shorter visits reduced paid costs or enabled profitable additional work. We also cannot conclude that faster visits caused the repeat visits. Fault complexity, parts quality, or technician assignments could differ even within similar repair types.
The owner should inspect repeat-visit records and discuss them with technicians before imposing another speed target. A dashboard identifies a pattern worth investigating; it does not supply the full explanation.
Check the evidence behind the number
Define the measure before comparing it. Specify what counts, the time period, the data source, and any exclusions. For a rate, the denominator is the total group used in the calculation. Excluding difficult cases can make a rate look better without improving service.
Match follow-up periods. A newly completed repair has had less time to generate a repeat visit than one completed two weeks ago. Comparing those groups immediately would favor the newer group unfairly.
Look at counts alongside rates, and separate relevant groups. An overall average can hide a deterioration for a particular product or customer group. Read individual records and customer explanations as well as totals.
Finally, consider incentives. If staff are rewarded only for short visits, they may feel pressure to leave early. That is a risk to investigate, not evidence that anyone behaved badly. Choose measures that keep the customer result visible.
Exercise
Allow 7–9 minutes. You manage delivery operations at established retailer Brook Home. Your goal is reliable arrival by the date originally promised to each customer.
The team checks stock before accepting orders. It treats stock availability at acceptance as a possible early signal of reliable delivery. All orders below have passed their promised delivery dates. Records include every order due in each week.
| Measure | Week A | Week B |
|---|---|---|
| Orders due for delivery | 100 | 120 |
| Orders with stock confirmed at acceptance | 90 | 114 |
| Orders delivered by original promised date | 90 | 96 |
The dashboard highlights “on-time deliveries increased” and recommends expanding the team's current approach.
- Classify stock checking, stock availability at acceptance, and on-time arrival as activity, possible leading indicator, or outcome.
- Calculate stock-confirmation and on-time delivery rates for each week. Calculate the number of late orders.
- Explain how the dashboard's claim can be numerically true while reliability worsens.
- Recommend what to investigate before expanding the approach. Name one additional measure and one record or conversation that would help.
Attempt the exercise before reading the answer.
Answer
Checking stock is an activity. Availability confirmed at acceptance is a possible leading indicator. Arrival by the original promised date is the customer outcome being measured.
Stock-confirmation rates were 90 ÷ 100 × 100 = 90% and 114 ÷ 120 × 100 = 95%. This signal improved by five percentage points.
On-time delivery rates were 90 ÷ 100 × 100 = 90% and 96 ÷ 120 × 100 = 80%. Reliability fell by ten percentage points. Late orders increased from 100 − 90 = 10 to 120 − 96 = 24.
The count of on-time deliveries rose by six, but the number of orders due rose by twenty. A smaller share of customers received their orders on time. The dashboard's claim omits that change.
Investigate the late orders before expanding the approach. Check whether confirmed stock was actually available, then examine picking, dispatch, and carrier delivery times. Higher volume may have strained capacity, but these figures do not establish the cause.
One useful additional measure is the share dispatched early enough for the agreed delivery service. Compare original promise, dispatch, and arrival timestamps, and ask warehouse staff about delayed orders. Continue monitoring actual arrival; a dispatch measure alone cannot establish customer success.
Recall answer: Riverton assumed shortages could be detected and fixed early enough to prevent late shipments. Completing more checks would not prove that this happened.
Applying this lesson
Before acting on a green dashboard, identify the customer or business result, check how the number was calculated, and look for an outcome that could be worsening alongside it.
Sources
Comments
Loading comments…