Hypothesis tests and practical significance
Compare a measured improvement with both zero and the improvement needed to pay for it.
Interpret a p-value and distinguish evidence of an effect from the size and business importance of that effect.
Should the plant license the new setting?
A food manufacturer tests a filling-machine setting that uses slightly less of an expensive ingredient per package. Both settings meet the product specification. The engineer reports a “statistically significant saving.” The plant manager has a different question: how many grams per package must the setting save to cover the annual license?
Both businesses and all figures in this lesson are fictional. You can do the calculations with a basic calculator. Recall the stationery shop from Lesson 045: its estimated conversion rate was 25%, with a 95% confidence interval from 20.76% to 29.24%. Why did the owner still decline a contract that broke even at 24%?
What a hypothesis test asks
Suppose the new setting makes no difference to average ingredient use. Even then, two groups of production runs will rarely have identical averages. A hypothesis test asks how unusual the observed difference would be under that starting assumption, called the null hypothesis. Here the null says the average saving is zero. The alternative says the setting changes the average in either direction.
Before collecting data, the plant chose one outcome, a fixed number of runs and a two-sided test. “Two-sided” means an increase in ingredient use counts as evidence against zero just as a decrease does. The plant randomly assigned comparable runs to the two settings and measured grams per package the same way in both groups. The analysis treats each independent run as an observation. Counting thousands of packages from one run as thousands of separate trials would make the evidence look stronger than it is.
The old setting averaged 10.0 grams per package and the new one averaged 9.7. Subtract new from old so a positive difference means a saving: 10.0 − 9.7 = 0.3 gram per package. The analyst reports a standard error of 0.1 gram for that difference. As in the previous lesson, this measures how much an estimate would typically vary across repeated samples. The measured difference sits (0.3 − 0) ÷ 0.1 = 3 standard errors above zero.
For this large trial, the analyst uses a bell-shaped approximation for the difference. Under the zero-saving assumption, a difference at least three standard errors from zero in either direction occurs about 0.27% of the time. That probability, 0.0027, is the p-value. Think of about 27 results that extreme per 10,000 comparable trials if the setting really had no effect and the model assumptions held.
The plant had chosen 0.05 as its cutoff for calling a result statistically significant. Since 0.0027 is below 0.05, the trial provides evidence against zero saving. It does not give a 99.73% probability that the setting works. To compute the p-value we assumed zero saving and asked about possible data. Reversing that reasoning does not give the probability of zero saving given the data. The American Statistical Association also cautions that a p-value does not measure an effect’s size or importance. Read the ASA statement.
What the saving is worth
The plant expects to make one million packages a year. Ingredient costs $0.01 per gram, and the license costs $6,000 per year. Assume unchanged volume, quality and selling price, with no other differences in cost.
- Estimated annual ingredient saving: 0.3 gram per package × 1,000,000 packages × $0.01 per gram = $3,000.
- Estimated annual gain after the license: $3,000 − $6,000 = −$3,000.
- Break-even saving: $6,000 ÷ (1,000,000 × $0.01) = 0.6 gram per package.
The plant can also check uncertainty. An approximate 95% interval for average grams saved is the estimate plus or minus 1.96 standard errors: 0.3 ± (1.96 × 0.1), or about 0.10 to 0.50 gram. The whole interval is above zero, yet even its upper end falls below the 0.6 gram the plant needs. Decline the license at this price. The engineer’s finding stands; the supplier is asking more than the measured benefit is worth.
Recall answer: The stationery shop required the lower end of its interval to cover 24%. The estimate was above break-even, but its lower limit was below it. Both decisions depend on comparing an interval with the threshold that matters to the business.
Exercise: Should a shop pay for appointment reminders?
Allow eight minutes. A repair-shop owner tests a paid reminder against the usual confirmation. She randomly assigns 200 customers, each with one appointment: 60 of 100 attend with the usual confirmation, and 70 of 100 attend with the reminder. The outcome and sample size were set in advance. For the difference in attendance rates, the analyst reports a two-sided p-value of 0.14 against zero difference and an approximate 95% interval from −3 to +23 percentage points.
The reminder costs $2 per appointment, including no-shows. Each additional attended appointment earns $40 after all additional service costs, before the reminder fee. Assume spare capacity, independent appointments, no displaced bookings and similar future customers. The owner will sign an annual contract only if the lower end of the interval clears break-even.
- Calculate the observed improvement in percentage points and the estimated net gain per 100 appointments. Find the improvement needed to cover the reminder fees.
- Explain the p-value in one sentence. A colleague says, “It missed 0.05, so we know reminders do nothing.” Does the interval support that conclusion?
- Translate both ends of the interval into dollars per 100 appointments. Recommend an action and identify one thing you would do differently in a follow-up trial.
Try writing the recommendation before reading on.
Answer: A promising estimate with a wide range
Attendance rose from 60% to 70%, a difference of 10 percentage points. At that estimate, switching 100 comparable appointments would bring ten additional visits worth 10 × $40 = $400. Fees apply to all 100 appointments, so estimated net gain is $400 − $200 = $200. To cover $200, the shop needs five additional visits per 100 appointments: a five-percentage-point improvement.
If reminders had no effect and the test assumptions held, a difference at least as extreme as the observed one in either direction would occur about 14% of the time. At the chosen 5% cutoff, the trial does not establish a difference. The interval includes zero, but it also includes gains large enough to pay for the service. Calling the reminders useless would go beyond the evidence.
At the lower limit, three fewer visits per 100 would mean (−3 × $40) − $200 = −$320. At the upper limit, 23 additional visits would mean (23 × $40) − $200 = $720. These are estimates of the average financial effect at the two limits, not guaranteed bounds on next week’s receipts.
Hold off on the annual contract. The best estimate earns $200 per 100 appointments, but the lower limit falls short of the owner’s five-point requirement. If a further pilot is affordable, plan its size around the improvement that would pay for the service. Keep the randomized comparison and set the stopping point before looking at results.
Add to your evidence audit
For the plant, write “evidence of a saving; too small at this price.” For the shop, write “potentially worthwhile gain; too uncertain to sign.” Beside each sentence, record the estimated effect, its interval and the break-even threshold in matching units. Next you’ll examine how many observations a test needs and what happens when a team keeps checking for a winner.
Sources
Comments
Loading comments…