Sampling, selection, and measurement bias
Audit a repair shop’s six-day promise, then estimate a delivery company’s on-time rate using the customers’ actual handoff times.
Identify how a sample or measurement process could distort a business conclusion and propose a more representative approach.
Check the seller’s clock
A repair-shop owner shows a prospective buyer a report: “95% of our repairs are ready within six days.” The buyer plans to offer customers a six-day turnaround. She picks one ticket. The customer dropped off a mixer on Monday, but the technician’s clock started on Thursday, when the part arrived. The report may tell her how quickly technicians work. Can it tell her what she can promise customers?
Both businesses and all records in this lesson are fictional. Paper and a calculator are enough.
Recall Lesson 043: Two repair shops both averaged five days per job. To check a six-day promise, what would you count instead of comparing their averages?
Who had a chance to be counted?
Start with the claim you need to check: “Of all repairs accepted in September, what share were ready for pickup within six calendar days of drop-off?” One observation is one repair. The population is the whole group the buyer wants to describe: every repair accepted that month. A sample is the smaller group she examines if she cannot check them all. The list she selects from is called the sampling frame.
The seller’s report lists only closed tickets. A job waiting for parts cannot appear until somebody closes it. If slow jobs are more likely to remain open, the report will tend to make service look faster. This is selection bias: how records get into the analysis systematically favors certain outcomes. Selecting tickets at random from that report still gives open tickets no chance of selection.
There is a second problem even for jobs that made the list. Starting the clock at technician assignment leaves out the customer’s wait for assignment. That is measurement bias: the recording method pushes the result away from the quantity you meant to measure. Statistics Canada distinguishes problems with coverage, such as missing records, from problems with how an outcome is measured. See Statistics Canada’s guide to non-sampling error.
Reconstruct the September result
The buyer finds 100 distinct September repairs in the intake log. She checks them after every job has had its full six days. None was canceled. Eighty tickets are closed; the other 20 repairs are still unfinished and all are past deadline. For closed jobs she can verify drop-off, technician assignment, and when the item was ready for pickup.
| Records included | Start of clock | Ready within six days | Rate |
|---|---|---|---|
| 80 closed tickets | Technician assignment | 76 | 76 ÷ 80 = 95% |
| Same 80 tickets | Customer drop-off | 68 | 68 ÷ 80 = 85% |
| All 100 accepted repairs | Customer drop-off | 68 | 68 ÷ 100 = 68% |
Read down one change at a time. On the same 80 jobs, eight missed the customer’s deadline once waiting before assignment counted. Correcting the clock lowers the rate by ten percentage points. Restoring the 20 overdue repairs leaves the number on time unchanged but increases the number that had to meet the promise. That lowers the rate another 17 points.
The shop met the six-day deadline on 68 of September’s 100 repairs. The buyer knows the result for that month because she checked every intake record. It does not establish next month’s rate. Nor should she automatically label every open repair late: one dropped off yesterday still has time. Choose an intake period far enough in the past that every job has had the same opportunity to finish.
Choose records for the decision
For a buying decision, the buyer could review several months, including busy periods. If checking every ticket is too costly, she can number the entries in the full intake log and randomly draw ticket numbers, giving each repair an equal chance. She should keep the original selections and investigate missing timestamps. Replacing hard-to-trace tickets with easy ones can bring the same bias back.
Two random draws can give different rates simply because they contain different jobs. That is sampling variation. A larger random sample generally reduces that variation. A thousand closed tickets measured from technician assignment would still leave the buyer with the wrong starting time and no account of overdue open jobs.
Exercise: What can a delivery director report?
Allow 8–10 minutes. An established delivery company wants to know whether at least 90% of last month’s scheduled deliveries reached the customer by the end of the promised window. Its complete roster contains 1,000 deliveries: 800 city and 200 rural. All were eventually handed over. The dashboard currently reports only city routes and marks a delivery complete when the truck parks.
An analyst draws 50 deliveries at random from each group so there will be enough rural records to examine. She finds verified customer handoff times for all 100 selected deliveries. In the city, 48 of 50 handoffs were on time. On rural routes, 35 of 50 were on time.
- Identify one selection problem and one measurement problem with the original dashboard. Could adding more city parking records fix either?
- Calculate the two handoff rates. Then pool the 100 sampled deliveries and calculate the rate. Why does pooling give rural routes too much influence?
- Estimate the overall rate by applying each group’s sampled rate to its actual share of deliveries. Write a two-sentence recommendation about the 90% target and where the director should investigate.
Try the recommendation before reading the answer.
Answer
The dashboard excludes rural deliveries, so it cannot describe the whole operation. Parking can also happen before the driver reaches the customer. A truck that parks at 4:55 and hands over a parcel at 5:08 has missed a 5:00 deadline. Adding city parking records fixes neither problem.
City handoffs were on time for 48 ÷ 50 = 96% of sampled deliveries; rural handoffs for 35 ÷ 50 = 70%. Pooling gives (48 + 35) ÷ 100 = 83%. That accurately describes the sample. But half the sample is rural, while rural routes account for only 200 ÷ 1,000 = 20% of deliveries.
To estimate the company rate, give each group a weight, meaning its share of actual deliveries. City deliveries get 800 ÷ 1,000 = 0.80; rural deliveries get 0.20. The estimated rate is (0.80 × 96%) + (0.20 × 70%) = 90.8%. As a check, that represents an estimated 768 on-time city deliveries plus 140 rural, or 908 of 1,000. Only the 100 sampled handoffs were actually checked.
A useful recommendation: “Our best estimate for last month is 90.8% on time, only 0.8 percentage points above target, so I would report the estimate and verify more handoffs before saying we cleared 90%. Start the operating review with the 15 late rural handoffs: compare parking and handoff times and check where delays actually occurred.”
The equal sample sizes were useful for studying rural service. The mistake would be treating the sample’s 50/50 mix as the company’s delivery mix. Weighting corrects that known difference; it cannot correct a bad timestamp or tell us what caused a late handoff.
Recall answer: Count jobs ready within six days and divide by all eligible jobs. A five-day average alone gives neither count. Today’s audit adds two checks: every eligible job can enter the count, and the clock measures the customer’s wait.
Add to your evidence audit
Keep the delivery recommendation with this week’s audit. Beside the estimate, record who was eligible, how records were chosen, and what event stopped the clock. The next lesson asks how much an estimate like 90.8% could vary because you checked a sample.
Sources
Comments
Loading comments…