Statistics, study design and diagnostic tests
Types of data and how to describe them, the normal distribution, study designs and the level of evidence each provides, bias and confounding, hypothesis testing with p values and confidence intervals, type I and II errors, power and sample size, measures of risk, and the evaluation of diagnostic and screening tests.
By the end of this page you should be able to
- Classify types of data and choose appropriate measures of central tendency and spread.
- Describe the main study designs, their strengths and weaknesses, and the sources of bias and confounding.
- Explain the null hypothesis, p values, confidence intervals, type I and type II errors, power and the determinants of sample size.
- Calculate and interpret relative risk, odds ratio, absolute risk reduction and number needed to treat.
- Calculate and interpret sensitivity, specificity, predictive values and likelihood ratios, and explain the effect of prevalence.
Data and how to describe it
Categorical data fall into groups: nominal (no order, such as blood group) or ordinal (ordered, such as tumour grade or a pain score). Numerical data are discrete (counts) or continuous (measurements such as haemoglobin).
For data with a normal (Gaussian) distribution, the mean and standard deviation describe the centre and spread; about 68% of values lie within one standard deviation of the mean, 95% within about two (1.96) and 99.7% within three. For skewed data or ordinal data, the median and interquartile range are better, because the mean is pulled towards the tail. In a positively (right) skewed distribution, mean > median > mode.
The standard error of the mean (standard deviation ÷ √n) describes the precision of a sample mean as an estimate of the population mean, not the spread of the data, and it shrinks as the sample grows. A 95% confidence interval is roughly the mean ± 1.96 standard errors.
Study design
- Randomised controlled trial: participants are randomly allocated to interventions, which balances known and unknown confounders between groups. Allocation concealment prevents selection bias; blinding of participants, clinicians and assessors prevents performance and detection bias. Analysis by intention to treat (by allocated group, whatever was received) preserves randomisation and gives a conservative, pragmatic estimate; per protocol analysis overestimates benefit. RCTs are the best design for testing interventions but may be impractical for rare outcomes or harms.
- Cohort study: a group defined by exposure is followed to see who develops the outcome; prospective or retrospective. Gives incidence and relative risk, suits common outcomes and rare exposures, but is long, costly and open to confounding and loss to follow-up.
- Case-control study: people with the outcome (cases) are compared with people without (controls) for past exposure. Efficient for rare diseases; gives an odds ratio; prone to recall and selection bias.
- Cross-sectional study: exposure and outcome measured at one time; gives prevalence, cannot establish sequence.
- Case series and reports: describe experience, generate hypotheses, cannot test them.
- Systematic review and meta-analysis: a structured search and appraisal of all relevant studies, with pooling of results where they are similar enough; shown on a forest plot. Its strength depends on the studies included; heterogeneity and publication bias (shown by an asymmetric funnel plot) weaken it. Levels of evidence rank designs roughly from systematic reviews of RCTs down to expert opinion.
Bias and confounding
Bias is a systematic error that pushes the result away from the truth, and it cannot be fixed by enlarging the sample. Selection bias arises from how participants are chosen or allocated; information (measurement) bias from how data are recorded, including recall bias and observer bias; attrition bias from unequal loss to follow-up; and publication bias from preferential publication of positive results. Lead time bias makes survival appear longer when a disease is diagnosed earlier by screening without any change in the time of death, and length time bias occurs because screening preferentially detects slower-growing disease.
A confounder is a factor associated with both the exposure and the outcome that is not on the causal pathway, creating or masking an apparent association (for example, smoking confounding an apparent link between alcohol and lung cancer). It is controlled by randomisation, restriction, matching, stratification and multivariable analysis. Causation is supported by the strength, consistency, specificity and temporality of the association, a dose-response relationship, plausibility and experimental evidence.
Hypothesis testing
A study tests a null hypothesis that there is no difference. The p value is the probability of obtaining a result at least as extreme as the one observed if the null hypothesis were true; it is not the probability that the null hypothesis is true. By convention, p < 0.05 is called significant. A 95% confidence interval gives the range within which the true value is likely to lie; if the interval for a difference excludes zero (or for a ratio excludes one), the result is significant at the 5% level, and the width shows the precision. A statistically significant result may not be clinically important, and a non-significant result in a small study does not prove there is no difference.
A type I error (alpha) is rejecting a true null hypothesis: a false positive, set by the significance level, and inflated by multiple comparisons. A type II error (beta) is failing to reject a false null hypothesis: a false negative. Power (1 − beta, conventionally 80 to 90%) is the probability of detecting a true difference of a given size. The sample size needed rises with a smaller expected difference, greater variability, a stricter alpha and higher power, and must allow for dropouts.
The test chosen depends on the data: for comparing means of two groups of normally distributed data, the t test (paired or unpaired); for more than two groups, analysis of variance; for non-normal or ordinal data, non-parametric tests such as Mann-Whitney U or Wilcoxon; for categorical data, the chi-squared or Fisher exact test; for association between two continuous variables, correlation (which does not imply causation) and regression; for time to event, Kaplan-Meier curves compared by the log-rank test and modelled by Cox regression giving a hazard ratio.
Measures of risk
If the event rate is 20% in controls and 15% with treatment: the relative risk is 0.15 ÷ 0.20 = 0.75; the relative risk reduction is 25%; the absolute risk reduction is 5 percentage points; and the number needed to treat is 1 ÷ 0.05 = 20. The relative measure stays similar across populations, but the absolute benefit depends on the baseline risk, so the same treatment helps a high-risk population far more. The odds ratio compares the odds of the outcome (events ÷ non-events) and approximates the relative risk when the outcome is rare; it is the natural output of case-control studies and logistic regression.
Diagnostic and screening tests
Sensitivity is the proportion of people with the disease who test positive; specificity the proportion without it who test negative. They are properties of the test at a given cut-off, and moving the cut-off trades one against the other, which a receiver operating characteristic (ROC) curve displays (sensitivity against 1 − specificity; the area under the curve summarises discrimination). A highly sensitive test, when negative, helps rule out disease; a highly specific test, when positive, helps rule in.
The positive predictive value (the proportion of positive results that are true) and negative predictive value depend on prevalence: the same test applied to a low-prevalence population produces mostly false positives. Likelihood ratios combine sensitivity and specificity and do not depend on prevalence; multiplying pre-test odds by the likelihood ratio gives post-test odds.
A screening programme should target an important condition with a recognisable early stage and a natural history understood well enough, an acceptable, valid and affordable test, an effective treatment that works better when started early, and benefits that outweigh harms, including overdiagnosis. Its success is judged by a fall in disease-specific mortality, not by longer survival from diagnosis, because of lead and length time bias.
Making the answer flow
- Name the data type before the statistic or test.
- For study design, give the direction (exposure to outcome or outcome back to exposure), the measure it produces and its main bias.
- Define the p value correctly and pair it with a confidence interval and clinical importance.
- Draw the two by two table and derive each measure from it; mention the effect of prevalence.
Common traps
- Saying the p value is the probability that the null hypothesis is true.
- Confusing standard deviation (spread of data) with standard error (precision of the mean).
- Saying predictive values are fixed properties of a test.
- Reporting only relative risk reductions, which exaggerate small absolute benefits.
- Treating longer survival in screened patients as proof that screening works.
Where this meets surgical practice
- Surgical trials are hard to blind and depend on the surgeon's learning curve; these affect how results should be interpreted.
- Screening programmes must be judged by disease-specific mortality, not by survival from diagnosis, because of lead time bias.
Test yourself
Answer each one aloud or on paper before you open the model points; explaining the mechanism is what makes the MCQ options easy to judge.
- 1.
What is a p value, and what is a confidence interval?
Show answer
- p value: probability of a result at least this extreme if the null hypothesis is true.
- Not the probability that the null hypothesis is true or that the result is due to chance alone.
- 95% confidence interval: range likely to contain the true value; excluding zero (or one for ratios) means significance at 5%.
- Width shows precision; significance is not the same as clinical importance.
- 2.
What determines the sample size needed for a trial?
Show answer
- The smallest clinically important difference to detect: smaller needs more.
- Variability of the outcome: greater needs more.
- Significance level (alpha) and power (1 − beta): stricter alpha and higher power need more.
- Allowance for dropouts and the event rate for binary outcomes.
- 3.
Explain type I and type II errors.
Show answer
- Type I: rejecting a true null hypothesis, a false positive; probability alpha.
- Type II: failing to reject a false null hypothesis, a false negative; probability beta.
- Power is 1 − beta.
- Multiple comparisons inflate type I error; small samples raise type II error.
- 4.
Compare a cohort study and a case-control study.
Show answer
- Cohort: starts with exposure, follows for outcome; gives incidence and relative risk; suits rare exposures; costly, loss to follow-up.
- Case-control: starts with outcome, looks back for exposure; gives odds ratio; suits rare diseases; recall and selection bias.
- Both observational and open to confounding.
- Neither proves causation alone.
- 5.
Why are randomised trials analysed by intention to treat?
Show answer
- Analyses participants in the groups to which they were randomised.
- Preserves the balance of known and unknown confounders created by randomisation.
- Reflects real-world effectiveness, including non-adherence and crossover.
- Per protocol analysis tends to overestimate the treatment effect.
- 6.
What is confounding and how is it controlled?
Show answer
- A factor linked to both exposure and outcome, not on the causal pathway.
- Creates or hides an association.
- Design: randomisation, restriction, matching.
- Analysis: stratification and multivariable regression.
- 7.
Define sensitivity, specificity and predictive values.
Show answer
- Sensitivity: true positives ÷ all with disease. Specificity: true negatives ÷ all without disease.
- PPV: true positives ÷ all positives. NPV: true negatives ÷ all negatives.
- Sensitivity and specificity belong to the test at its cut-off; predictive values change with prevalence.
- Likelihood ratios combine both and are independent of prevalence.
- 8.
A treatment reduces mortality from 20% to 15%. Express the benefit.
Show answer
- Relative risk 0.75; relative risk reduction 25%.
- Absolute risk reduction 5 percentage points.
- Number needed to treat 20.
- Absolute benefit depends on baseline risk; relative measures can mislead.
- 9.
What makes a good screening programme, and what biases affect its evaluation?
Show answer
- Important condition with a detectable early stage and effective early treatment.
- Acceptable, valid, affordable test; benefits outweighing harms including overdiagnosis.
- Lead time bias and length time bias inflate survival in screened groups.
- Judge by disease-specific mortality in randomised comparisons.
Practise this week
Week 24 in the other subjects
- Anatomy: Vertebral column and spinal cord
- Physiology: Cerebral blood flow, intracranial pressure and CSF
Not affiliated with the Royal Australasian College of Surgeons. Notes are a revision aid, not a substitute for the curriculum or for clinical guidelines.