There is a moment at the end of every successful design of experiments study when the regression model spits out an optimum, the number is better than anything you actually ran, and the temptation is to take it straight to the process owner. Antony describes the impulse exactly: the tendency to eagerly grab the results and rush out to production saying "we have the answer, this will solve the problem." The DOE confirmation run is what stands between that impulse and an expensive mistake. This guide covers how to choose the confirmation settings, how to build the prediction interval that decides pass or fail, how many runs to do, and what a failure is actually telling you. Everything here applies to any design you can build in a free design of experiments calculator, from an eight-run screen to a full response surface.
Why the model is only a hypothesis
A fitted DOE model is an interpolation, not a measurement. Its predicted optimum is usually a point nobody has ever run. That is the whole reason it is worth having, and also the whole reason it needs checking.
Consider a two-level factorial. You measured the response at the corners of the design space and fitted a plane through them. The model then predicts a value at the best corner, or at some interior point if you fitted curvature. Every one of those predictions rests on assumptions that the data cannot fully test: that the response is smooth between the levels you ran, that no important factor was left out, that the noise factors behaved during the campaign, and that the terms you dropped from the model really were zero.
This is why you cannot validate a DOE model from its own residuals. Any one of those assumptions can be wrong without the analysis complaining. A model with an R² of 0.97 can still predict badly at the optimum, because R² measures how well the model reproduces the points it was fitted to, not how well it forecasts a point it has never seen. That gap between fit and forecast is exactly what a confirmatory experiment measures, and it is why every serious DOE methodology ends with one.
What a DOE confirmation run is
A confirmatory experiment is a run performed at the settings the model recommends, for the sole purpose of verifying the results of the statistical analysis before you act on them. It sits outside the design. Its data never goes back into the model fit, because a prediction that has been re-fitted to include the thing it was predicting is no longer a test of anything.
The decision rule Antony gives is simple. If conclusive results have been obtained, it is then recommended to take improvement actions on the process under investigation. If the results do not turn out as expected, further investigation is required. And once the change is implemented, control charts on the response or the key process parameters should be constructed so the improvement is monitored rather than assumed to persist.
Picking the confirmation settings
Run at the settings the model recommends, inside the region you actually explored, and add one run at a familiar condition as a control. Each of those three clauses is doing work.
The recommended settings come straight from the reduced model, not from the best run you happened to observe. Those two are often different. In a design with real interactions, the predicted optimum can beat every corner you ran, and in a design with a large noise component the best observed run is partly luck. Trust the model's arithmetic over any individual data point, and let the DOE output tell you which terms earned their place.
Staying inside the explored region is not a technicality. A two-level factorial fits a straight line between two settings of each factor and has literally no information about what happens outside them. If your model says titer rises with temperature across the range 30 to 37 °C, that is not evidence that 40 °C is better, and in a biological system it very often is not. Extrapolating a DOE model past its design space is the single fastest way to turn a good study into a wasted campaign, which is why response surface methodology insists on mapping the region before naming an optimum inside it.
The control run is cheap insurance. Include one confirmation run at a routine, well-characterised setpoint alongside the runs at the optimum. If both come in low on the day, you have learned that the process or the assay drifted, not that your model is wrong, and you have saved yourself a week of chasing a phantom.
Design the study whose optimum you will confirm
Build a factorial, screening or response-surface design with replicates, center points and a randomized run order, free in the browser.
The prediction interval for a DOE confirmation run
A confirmation run is a future observation, so it must be judged against a prediction interval, not a confidence interval. The prediction interval is always wider, and using the wrong one will fail experiments that were perfectly fine. This is the most common technical error in confirming a DOE.
The distinction is worth stating precisely. A confidence interval brackets the true mean response at a set of conditions. Its width comes only from uncertainty in the fitted coefficients, and it shrinks toward zero as you add design runs. A prediction interval brackets a future observation at those conditions. It carries the same coefficient uncertainty plus the run-to-run variability that a brand new experiment will have, and that second component never goes away no matter how large the design was.
Here ŷ0 is the model's prediction at the confirmation point, s is the root mean square error from the ANOVA table, ν is its residual degrees of freedom, and h0 is the leverage of the prediction point, formally x0′(X′X)−1x0. For a two-level factorial in coded units evaluated at a corner, that ugly term collapses to something you can do on paper:
where p is the number of terms in the fitted model including the intercept and N is the number of design runs. The coded columns of a two-level factorial are orthogonal and each entry is ±1, so every term contributes exactly 1/N of leverage. A 16-run design carrying a four-term model has h0 = 0.25 at any corner. Setting m = 1 recovers the familiar single-observation prediction interval, ŷ0 ± t · s · √(1 + h0).
Antony offers a simpler check that works from the confirmation data alone and needs no leverage arithmetic. Compute the mean and standard deviation of the confirmation runs themselves, form a 99 per cent interval, and ask whether the model's prediction falls inside it:
where ȳ is the mean response from the confirmation trials, SD their standard deviation, and n the number of them. If the predicted value based on the regression model falls within this interval, the model is considered good. Note that this test runs in the opposite direction from the prediction interval: it brackets the observation and asks whether the prediction lands inside, rather than bracketing the prediction and asking about the observation. In practice the two usually agree, and the pair of them is a more convincing report than either alone.
| Interval | Half-width | Value (g/L) | Use it for |
|---|---|---|---|
| CI on mean response | t · s · √h0 | ±0.196 | Reporting where the true mean sits. Not for confirmation. |
| PI, mean of m = 4 runs | t · s · √(1/m + h0) | ±0.277 | The correct test for 4 confirmation runs |
| PI, single future run | t · s · √(1 + h0) | ±0.439 | Judging one confirmation run on its own |
| Antony rule of thumb | 3 · SDconf / √n | ±0.180 | Cross-check computed from the confirmation data |
Pass or fail: a worked confirmation
The test is a single comparison: does the confirmation result fall inside the prediction interval? Here is the whole calculation on a real-shaped bioprocess problem.
Worked example: confirming an E. coli soluble-titer optimum
A 2³ full factorial on induction OD600 (A), induction temperature (B) and feed rate (C), replicated twice, so N = 16 runs. After model reduction, three terms survive alongside the intercept: A, C and the A×C interaction, so p = 4. The ANOVA gives s = 0.18 g/L with ν = 16 − 4 = 12 residual degrees of freedom, and t0.025, 12 = 2.179.
Step 1. Predict. At the model optimum (A = +1, C = +1; B was dropped) the fitted model gives ŷ0 = 3.42 g/L, better than the best single run in the design.
Step 2. Leverage. Two-level design at a corner, so h0 = p/N = 4/16 = 0.25.
Step 3. Interval for m = 4 confirmation runs.
half-width = 2.179 × 0.18 × √(1/4 + 0.25)
= 2.179 × 0.18 × 0.7071 = 0.277 g/L
PI = 3.42 ± 0.277 = (3.14, 3.70) g/L
Step 4. Run them. Four independent bioreactor runs at the optimum, randomized among the week's other work: 3.31, 3.48, 3.19, 3.36 g/L. Mean = 3.335 g/L, SD = 0.120 g/L.
Step 5. Decide. 3.14 ≤ 3.335 ≤ 3.70, so the result falls inside the prediction interval. Pass. The model predicted 3.42 and the process delivered 3.34, a gap of 0.085 g/L, which is less than half a standard deviation of a single run.
Step 6. Cross-check with the rule of thumb. 3.335 ± 3 × 0.120 / √4 = 3.335 ± 0.180 = (3.155, 3.515). The prediction of 3.42 falls inside, so the model is considered good by that test too. Both tests agree, and you can now recommend the change.
How many confirmation runs?
Between 4 and 20, chosen by cost: 4 runs if each one is expensive, up to 20 if they are cheap. That range is not arbitrary, and the arithmetic behind it explains both ends.
Look again at the prediction interval half-width, t · s · √(1/m + h0). Only the 1/m term shrinks as you add confirmation runs. The h0 term is the model's own uncertainty, fixed by the design you already ran, and it puts a hard floor under the interval. In the worked example that floor is t · s · √0.25 = 0.196 g/L, and no amount of confirming will get you below it.
Figure 3. Prediction interval half-width against confirmation run count for the worked example. The dashed line is the floor set by model uncertainty. Most of the benefit is bought in the first four runs.
| Confirmation runs, m | PI half-width (g/L) | Reduction vs 1 run | Verdict |
|---|---|---|---|
| 1 | ±0.439 | — | Cannot separate a model failure from one odd batch |
| 2 | ±0.340 | 23% | Minimum defensible, no spread information |
| 3 | ±0.300 | 32% | Workable when runs are very expensive |
| 4 | ±0.277 | 37% | The standard choice for bioreactor runs |
| 6 | ±0.253 | 42% | Worth it if runs are shake-flask scale |
| 8 | ±0.240 | 45% | Diminishing returns begin |
| 12 | ±0.226 | 48% | Only for cheap, fast responses |
| 20 | ±0.215 | 51% | Practical ceiling. Floor is ±0.196 |
Read the reduction column and the cost guidance falls out. Four runs capture 37 of the 51 percentage points available, so more than seven tenths of the total benefit for a fifth of the effort of running twenty. Below three runs you lose something else entirely: with n = 1 or 2 you have no usable standard deviation from the confirmation set, so you cannot compute the cross-check interval or tell a genuine shortfall from a single contaminated flask.
One more consideration matters more in biology than in the manufacturing contexts these rules came from. Confirmation runs must be genuine replicates rather than repetition: four independent bioreactor setups, not four vials assayed from one run. Four assay samples from a single run will cluster tightly around a value that could be anywhere, and will pass or fail your interval for reasons that have nothing to do with the model. Where titer is read by immunoassay, running those confirmation samples through a consistent standard curve matters too, and an ELISA analyzer keeps the curve fitting and back-calculation identical between the design runs and the confirmation runs so the comparison stays honest.
What to do when confirmation fails
A failed confirmation is data, and it usually points at a specific, identifiable cause rather than at bad luck. Antony lists the possible reasons the experiment did not meet its objective, and they sort neatly into things you can check in an afternoon and things that need another design.
| Possible cause | How to tell | Fix |
|---|---|---|
| Errors in conducting the runs | Audit batch records against the run sheet; setpoints not actually achieved | Re-run correctly |
| Measurement error | Re-assay retained samples; control run also off | Fix the measurement system first |
| Unmodelled curvature | Center runs sit off the factorial plane; optimum is at a corner | Add center points, augment to CCD or Box-Behnken |
| An important factor was never identified | Large gap despite good fit and low residuals | Another screening round with new candidates |
| Inadequate control of noise factors | Confirmation runs scatter widely; SD far above s | Block, control, or stabilise the process first |
| Wrong response chosen | Response is an attribute or a symptom, not a function | Switch to a continuous, function-related response |
| Wrong design for the problem | Low resolution; effects aliased with two-factor interactions | Add de-aliasing runs (fold-over) |
In bioprocess work, unmodelled curvature is the most common cause by a wide margin, and it has a characteristic signature: the model predicts that pushing every factor to one corner is best, and the process disagrees. Biological responses are rarely monotonic across a useful range. Temperature, pH, osmolality and feed rate all have interiors that beat their extremes, and a two-level design fits a plane straight through that hill without ever seeing it. If you ran center points in DOE, the curvature test will have warned you before you ever got to confirmation. If you did not, the failed confirmation is the warning, and the fix is to augment the design rather than to argue with the result.
The second most common cause in biology is a missing factor, and it has its own signature: the model fits its own data beautifully, residuals look clean, and yet the confirmation lands far outside. That combination means the design captured the relationships among the factors you chose while something you never varied moved between the campaign and the confirmation. Seed-train age, media lot, and thaw number are the usual culprits, and all three are best handled by blocking in design of experiments next time round.
Augment the design instead of arguing with the result
Add center points, axial runs or a fold-over fraction and get a fresh randomized run order, free in the browser.
Predict, then confirm
Write the prediction and its interval down before the confirmation runs go on the bench. This is the cheapest discipline in the whole method and the one most often skipped.
A prediction recorded after the fact is not a prediction, and an interval computed after seeing the result is not a test. Committing to ŷ0 = 3.42 g/L with a pass band of 3.14 to 3.70 in advance turns the confirmation into a genuine experiment with an outcome that can go either way. It also removes the argument that follows a marginal result, because the criterion was agreed before anyone had a stake in it.
It is also worth being explicit about what a passed confirmation does and does not prove. It tells you the model predicts well at that point, under the conditions of that week. It does not validate a DOE model across the whole design space, and it says nothing about how the process will behave six months later on a different media lot. That is what the control chart is for.
The rest is bookkeeping. Build and randomize the design in the DOE generator and randomizer, fit and reduce the model, read s and its degrees of freedom off the ANOVA table, compute h0 = p/N, pick m from the cost of a run, and write the interval into the protocol. Then run the confirmation, compare, and either recommend the change with a control chart behind it or go back with a much sharper question. That loop is what separates a well-chosen DOE design that changed a process from one that produced a nice contour plot.
Frequently Asked Questions
What is a DOE confirmation run?
A DOE confirmation run is an experiment performed at the settings your fitted model recommends, for the sole purpose of checking that the model's prediction holds in reality. It is not part of the original design and contributes nothing to fitting the model. Its only job is to compare the measured response at the predicted optimum against the interval the model says that response should fall in. If conclusive results are obtained, improvement action on the process is recommended; if not, further investigation is required before implementing anything.
How many confirmation runs should you do?
Between 4 and 20, depending on cost: 4 runs if each one is expensive, up to 20 if they are cheap. The reason the range is bounded at both ends is that the interval width has a floor. Going from 1 run to 4 shrinks the prediction interval by about 37 per cent in a typical 16-run design, but going from 4 to 20 shrinks it by only a further 22 per cent because the model's own uncertainty never goes away. Fewer than 3 runs leaves you unable to tell a model failure from a single odd batch.
What is the difference between a confidence interval and a prediction interval in DOE?
A confidence interval brackets the true mean response at a set of conditions and reflects only the uncertainty in the fitted coefficients. A prediction interval brackets a future observation and is always wider, because it adds the run-to-run variability of a new experiment on top of the model uncertainty. Confirmation runs are future observations, so the prediction interval is the correct yardstick. Judging a confirmation run against a confidence interval will fail experiments that were actually fine.
How do you calculate a prediction interval for a DOE confirmation run?
For the mean of m confirmation runs at a point x₀, the interval is ŷ ± t · s · √(1/m + h₀), where t is the two-sided t value at your confidence level with the residual degrees of freedom, s is the root mean square error from the ANOVA table, and h₀ is the leverage of the prediction point. For a two-level factorial in coded units evaluated at a corner, h₀ simplifies to p/N, where p is the number of terms in the model including the intercept and N is the number of runs.
What should you do if the confirmation run fails?
Treat it as information, not failure. The standard causes are a wrong choice of design, a poorly chosen response, an important factor that was never included, unmodelled curvature between the design points, inadequate control of noise factors, measurement error, and mistakes in executing the runs. Check the cheap explanations first: verify the setpoints were actually achieved and re-run the assay on retained samples. If the gap is large and reproducible, unmodelled curvature is the most common cause, and the fix is to add center points or augment to a response-surface design.
Where should the confirmation runs be placed?
At the settings the model actually recommends, inside the design space you explored. Never extrapolate beyond the factor ranges you ran, because a linear model fitted between two levels says nothing about behaviour outside them. It is also worth adding one confirmation run at a routine, well-characterised condition as a control. If both the optimum and the control come in low, the problem is the process or the assay that day, not the model.
Related Tools
- DOE Experiment Generator — Build the design, add center points, and lay out the augmentation runs a failed confirmation calls for.
- ELISA Analyzer — Keep the standard curve and back-calculation identical between design runs and confirmation runs.
- Fed-Batch Calculator — Work out the feed profile that the confirmed optimum actually requires at the bench.
References
- Antony, J. (2003). Design of Experiments for Engineers and Scientists. Butterworth-Heinemann/Elsevier. ISBN 0-7506-4709-4. (§4.5 number of confirmatory runs; §4.6 confidence interval for the mean response; §8.1.12 perform confirmatory runs.)
- Montgomery, D.C. (2017). Design and Analysis of Experiments, 9th ed. Wiley. ISBN 978-1119113478. (Confirmation experiments and prediction intervals from fitted models.)
- Box, G.E.P., Hunter, J.S. & Hunter, W.G. (2005). Statistics for Experimenters, 2nd ed. Wiley. ISBN 978-0471718130. (Sequential experimentation and iteration after a failed prediction.)
- Mandenius, C.-F. & Brundin, A. (2008). Bioprocess optimization using design-of-experiments methodology. Biotechnology Progress, 24(6), 1191–1203. doi:10.1002/btpr.67
- NIST/SEMATECH (2012). e-Handbook of Statistical Methods, Section 5.5.9.10.2: Confirm the results. itl.nist.gov