Plenty of outcomes are binary: the sample is positive or negative, the unit passes or fails, the patient responds or does not. Fitting a straight line to a 0/1 response goes wrong immediately. The line keeps going, so it predicts probabilities greater than 1 at one end and less than 0 at the other. The residuals can never be normal because there are only two possible values.
Logistic regression models the probability of the outcome instead, through a curve that approaches 0 and 1 without reaching either. The model is fitted by maximum likelihood rather than least squares. The result is a model that behaves sensibly across the whole range of the predictor.
The odds of an event are its probability divided by the probability of it not happening. A probability of 0.8 is odds of 4: four times as likely to happen as not.
Probabilities are bounded at 0 and 1. Odds run from 0 to infinity. The logarithm of the odds runs from minus infinity to plus infinity. An unbounded scale is what a linear model needs.
So logistic regression fits a straight line to the log-odds. The log-odds is called the logit, and the model is written:
logit(p) = ln( p / (1 − p) ) = β0 + β1x1 + β2x2 + …
Written that way the model is an ordinary linear equation, and everything familiar from linear regression applies on the left-hand side. What it costs is interpretability: each coefficient β is a change in log-odds per unit of the predictor, and a change in log-odds means nothing intuitive on its own. A coefficient in this model is also called a log odds ratio, which is the clue to the fix — exponentiate it and you have an odds ratio.
Exponentiate a coefficient and you get an odds ratio: the factor by which the odds of the outcome are multiplied for a one-unit increase in that predictor, holding the others constant.
An odds ratio of 1 means no association. Above 1 the odds increase with the predictor: 1.5 means the odds are 50% higher per unit, and 2.0 means they double. Below 1 the odds decrease: 0.5 means they halve. Because the scale is multiplicative, 2.0 and 0.5 are equal and opposite effects. Keep that in mind when you are comparing them, since the distance from 1 is not symmetric the way it looks.
Two practical points. For a continuous predictor, “per unit” means per unit as measured: an odds ratio per one milligram is not comparable to one per ten milligrams, and reporting the unit is not optional. And the confidence interval matters more than the point estimate. An odds ratio of 3.4 with an interval from 1.05 to 11.0 is a signal that something may be there, not an established threefold effect. An interval that includes 1 means the data are compatible with no association at all.
One feature of those intervals surprises people. The interval on an odds ratio is not symmetric about the point estimate — 3.4 sits much closer to 1.05 than to 11.0. That is because the interval is computed on the log-odds scale, where it is symmetric about the coefficient, and then exponentiated along with it. Exponentiating stretches the upper half and compresses the lower half. So an odds ratio reported with a symmetric interval has almost certainly been computed the wrong way.
This is a common misreading. An odds ratio compares odds. A risk ratio compares probabilities. The two are close when the outcome is rare, and they diverge sharply when it is common.
Take a baseline probability of 0.5, odds of 1, and an odds ratio of 3. The new odds are 3, which is a probability of 0.75. The risk has gone up by a factor of 1.5, while the odds ratio reads 3. Describing that as “three times as likely” overstates the effect twofold.
Report an odds ratio as an odds ratio. If the audience needs risk, convert to predicted probabilities at stated values of the predictors. Predicted probabilities are clearer and more useful for a decision. Where the data come from a 2 × 2 table rather than a model, both measures are available directly; see chi-square, Fisher exact or McNemar.
There is no F-test and no R² here. The effect of the whole model, and of each term in it, is tested with either a likelihood-ratio χ² test or a Wald χ² test. The likelihood-ratio test compares the fit of the model with and without the term, and is generally the more reliable of the two, particularly with small samples or large effects, where the Wald test can behave badly.
Categorical predictors enter through dummy coding just as in ordinary regression. Interactions and polynomial terms are available the same way. The model can be as structured as the design requires.
Forwards, the model returns a predicted probability for any combination of predictor values. Predicted probability is the form most people want. If you then need a yes/no classification, you have to choose a probability threshold. Choosing it is a separate decision with its own trade-off between false positives and false negatives, exactly the problem ROC curves exist to display. The predicted probabilities from a logistic model feed straight into a ROC analysis.
Backwards, inverse prediction answers the question the other way round: at what value of the predictor does the probability reach some stated level, with a confidence interval on that value? Inverse prediction is not a niche facility. The same machinery sits behind a detection limit defined as the concentration detected 95% of the time, the subject of probit analysis for the limit of detection, where probit rather than logit is the conventional link but the idea is identical.
Two failure modes are worth recognising. Separation happens when a predictor perfectly divides the outcomes: every case above some value is positive and every case below is negative. The likelihood has no maximum, coefficients run off towards infinity, and standard errors become enormous. Ironically it means the predictor is extremely good; the model simply cannot express how good.
Too few events is easy to miss. It is the number of observations in the smaller outcome group that limits how many predictors a logistic model can support, not the total sample size. Two hundred observations with seven positives will not support a five-predictor model, however comfortable the total looks.
Download the logistic regression example workbook (.xlsx): a binary logistic model with odds ratios and confidence intervals, likelihood-ratio tests for the model and each term, and inverse prediction, ready to open in the Analyse-it trial.
The example workbook is downloading.
It opens in Excel on its own — the data and the finished results are both in it. Analyse-it is what lets you change the analysis and re-run it, try the same study on your own data, or work through it to see how the software handles it.
Every feature from all five editions for 15 days, with no sign-up and no licence key.
Reporting an odds ratio as “times more likely”. That is a risk ratio, and for a common outcome the two differ substantially.
Quoting an odds ratio without its interval or its unit. Both are needed before the number means anything.
Fitting a linear regression to a 0/1 response. It predicts impossible probabilities and violates the assumptions from the outset.
Counting the total sample rather than the events. The smaller outcome group is what limits how complex a model you can support.
Treating a 0.5 probability threshold as given. It is a choice, and rarely the right one when the two kinds of error carry different costs.
Ignoring wildly large coefficients and standard errors. That is usually separation, not a spectacular finding.
Analyse-it fits binary logistic regression in Fit Model, inside Excel:
Every feature from all five editions for 15 days, with no sign-up and no licence key. Logistic regression is part of the Standard edition, so it is in every Analyse-it edition, from US$ 155 a year. Validated against NIST Standard Reference Datasets.