EP242: The Best Regression Model
The best regression model Juan Navarro has ever seen had an R² of 0.96, and it also became the most expensive one for his company. In this episode, you will learn the six essential checks every model should pass before it is used to make quality decisions. I’ll explain them in plain language, with simple examples, using the analogy of a commercial pilot who never takes off without a preflight checklist. Because a high R² can hide a serious problem.
#RegressionAnalysis #QualityManagement #SixSigma #IndustrialStatistics #DataAnalytics #LeanSixSigma #Quality #ContinuousImprovement #AdvancedQualityPrograms #DataPreflight
https://open.spotify.com/episode/2mNsZ4UzZOti9j3PfKIrZd?si=LtXH50pwRlOkVgkzH1JMuA
https://rumble.com/v7cqfw0-ep242-the-best-regression-model.html
The best regression model I have seen in my career had an R² of 0.96, and yet it became the most expensive one for the company.
A pharmaceutical quality director built it for an FDA audit. She used 18 months of data and hundreds of batches. Statistically, the model looked perfect. The audit was approved with no findings at all.
But when I reviewed the documentation, I noticed something strange: the diagnostics folder had only one page with the R² value.
I asked the uncomfortable question: “Where are the residual plots?” She answered, “The statistician we hired said that with an R² like that, they were not needed.”
Three months later, a $2.3 million batch was rejected.
Quick pause: R² is a measure between 0 and 1 that shows how much of the variation in your data the model explains. An R² of 0.96 means the model explains 96% of what is happening. It sounds perfect, and that is the problem.
Using regression without checking the residuals, which are simply the difference between what the model predicted and what actually happened, is like a doctor running advanced tests but never taking your temperature or blood pressure. R² is the expensive test that looks impressive; residuals are the vital signs. And a patient with unstable vital signs is still in danger, even if the advanced tests look perfect.
I am not going to give you formulas or manuals. I am going to give you judgment: the ability to look at a model the way a doctor looks at a patient, not only at the impressive test result, but also at the vital signs that show whether everything is really working well.
A regression model, in simple terms, is a tool for understanding which variables affect your result. Instead of guessing, the model tells you, “If you raise the temperature by one degree, your defects go down by X units.” But this only works if the model is built correctly.
When I started, I treated regression like an advanced calculator. I entered variables, looked at coefficients, and celebrated the R². Then one of my own models, with an R² of 0.89, recommended changing the temperature on an assembly line. The result was 40 hours of unplanned downtime.
The model was “significant,” but the residuals formed an arc, and I did not even check them.
Imagine you are a commercial pilot. You have 200 passengers. Your plane is a Boeing 787, a state-of-the-art aircraft. But before takeoff, what do you do? You use a checklist. It is the same idea used since 1935, when an accident showed that even the best pilots can forget critical steps if they rely only on memory. It does not matter how many hours you have or how advanced the plane is. Without a checklist, nobody takes off.
Regression is the same. The plane is your model. The preflight is made up of six checks. If you skip them, you are taking off without knowing whether your instruments work.
Check 1: Does the relationship make sense?
If ice cream sales “predict” drownings at the beach, that is not a useful model; it is a coincidence. Heat is the real cause. If the relationship does not make sense, you do not take off.
Check 2: Is the relationship straight or curved?
Forcing a curve into a straight line is like using a speedometer that only goes up to 80 km/h when you are driving at 180. If your data is curved, you need to transform it.
Check 3: Are there extreme values that distort everything?
If most of your times are 2 minutes but a few are 4 hours, those huge values can pull the model in the wrong direction. A logarithmic transformation can help because it compresses large values, so they do not dominate.
Check 4: Are there unusual data points or errors?
One bad data point can tilt your whole line, like a gust of crosswind. Review those points. If there are errors, remove them. Do not ignore them simply because they are inconvenient.
Check 5: Do two variables move together?
If oven temperature and warehouse humidity change at the same time, the model does not know which one is causing the effect. This is called multicollinearity; I call it confusion.
Check 6: Is the distribution reasonably balanced?
If it is not, the residuals will make you pay for it. Residuals are like gravity: you can ignore them for a while, but they always show up.
Let’s go back to the pharmaceutical director. Her data was good. But when we checked multicollinearity, we saw that temperature and mixing speed had a correlation of 0.91. They moved together almost all the time. The model gave all the weight to temperature, while the real problem, speed, stayed hidden.
Then we checked the residuals and found a repeating pattern. The model worked well right after reactor maintenance and failed right before maintenance. The R² of 0.96 hid that pattern.
The $2.3 million batch failed right before maintenance. The model did not see it because nobody did the preflight.
The Preflight Rule
Rule 1: Ninety percent of the work happens before you run the model. Explore your data the way a pilot checks an aircraft. The six checks are not optional; they are what separate a safe landing from an accident.
Rule 2: Residuals do not lie. If they look like a funnel, meaning the spread grows at high values, or if they form an arc or follow a repeating pattern, your model is wrong, no matter how high your R² is.
Rule 3: A high R² does not mean the model is healthy. You can have a patient with perfect lab results and internal bleeding that those tests do not detect. Statistical significance, the stamp that says, “this is probably not random,” is not the same as practical importance. And when you use categorical variables, such as work shift or supplier type, remember this: all options in the same category stay together or go together; never remove just one.
W. Edwards Deming said, “Without data, you are just another person with an opinion.” I would add this: with poorly explored data, you are just another person with an opinion and nice-looking charts. Think about your last project where you used regression. How many of the six checks did you do before presenting the results? If you did fewer than four, your model took off without a preflight.
That is all for this week. Thank you all for your reviews of my books The Quality Mindset, Principles of Quality, and Life Quality Projects. If this episode gave you a checklist you did not have, like it, subscribe, and share it with the colleague who still flies without instruments.
Stay excellent, keep improving, and do your preflight.
References:
Gelman, A. & Hill, J. (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge: Cambridge University Press.
Belsley, D.A., Kuh, E. & Welsch, R.E. (1980) Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: Wiley.
Kutner, M.H., Nachtsheim, C.J. & Neter, J. (2004) Applied Linear Statistical Models. 5th ed. New York: McGraw-Hill.
Montgomery, D.C., Peck, E.A. & Vining, G.G. (2012) Introduction to Linear Regression Analysis. 5th ed. Hoboken: Wiley.
Montgomery, D.C. (2019) Introduction to Statistical Quality Control. 8th ed. Hoboken: Wiley.
Fox, J. (2016) Applied Regression Analysis and Generalized Linear Models. 3rd ed. Thousand Oaks: SAGE.
Osborne, J.W. & Waters, E. (2002) ‘Four assumptions of multiple regression that researchers often ignore’, Practical Assessment, Research & Evaluation, 8(2), pp. 1–5.
O’Hara, R.B. & Kotze, D.J. (2010) ‘Do not log-transform count data’, Methods in Ecology and Evolution, 1(2), pp. 118–122.
Dormann, C.F. et al. (2013) ‘Collinearity: a review of methods to deal with it and a simulation study evaluating their performance’, Ecography, 36(1), pp. 27–46.
Anscombe, F.J. (1973) ‘Graphs in statistical analysis’, The American Statistician, 27(1), pp. 17–21.
Cook, R.D. (1977) ‘Detection of influential observation in linear regression’, Technometrics, 19(1), pp. 15–18.
Tukey, J.W. (1977) Exploratory Data Analysis. Reading, MA: Addison-Wesley.