Start With Your Research Question
Every statistical test exists to answer a specific type of question. Before you reach for any test, articulate your research question clearly. Are you comparing groups? Examining relationships? Predicting outcomes? Testing differences over time? The answer determines which family of tests is appropriate.
Common question types and their corresponding test families:
| Question Type | Example | Test Family |
|---|---|---|
| Comparing two groups | Does treatment A differ from treatment B? | t-test, Mann-Whitney U |
| Comparing three or more groups | Do exam scores differ across teaching methods? | ANOVA, Kruskal-Wallis |
| Testing relationships | Is there an association between study hours and grades? | Correlation, chi-square |
| Predicting outcomes | Can we predict sales from advertising spend? | Regression |
| Testing change over time | Did anxiety scores change before and after therapy? | Paired t-test, Wilcoxon signed-rank |
| Testing frequency differences | Do voting preferences differ by region? | Chi-square test of independence |
Know Your Data Type
Statistical tests make assumptions about the type of data they analyse. Mismatching your data type to your test is one of the most common analytical errors. There are three broad data types:
Continuous (scale) data — Measurements on a meaningful numeric scale where differences between values are interpretable. Examples: blood pressure, reaction time, exam scores.
Ordinal (ranked) data — Categories with a meaningful order but unequal intervals between them. Examples: Likert scales (strongly disagree to strongly agree), cancer stages, education levels.
Nominal (categorical) data — Categories with no inherent order. Examples: blood type, country of birth, treatment group assignment.
Most tests are designed for either continuous or categorical data. Using a test designed for continuous data on ordinal or nominal data (for example, running a Pearson correlation on Likert-scale responses) is a common mistake that inflates Type I error rates.
Check Your Assumptions
Parametric tests (t-test, ANOVA, Pearson correlation, linear regression) assume:
- Normality — The dependent variable is approximately normally distributed within each group.
- Homogeneity of variance — The variance of the dependent variable is similar across groups (homoscedasticity).
- Independence — Observations are independent of each other.
- Interval or ratio measurement — The dependent variable is measured on a continuous scale.
If these assumptions are violated, non-parametric alternatives exist:
| Parametric Test | Non-Parametric Alternative |
|---|---|
| Independent t-test | Mann-Whitney U test |
| Paired t-test | Wilcoxon signed-rank test |
| One-way ANOVA | Kruskal-Wallis test |
| Repeated-measures ANOVA | Friedman test |
| Pearson correlation | Spearman's rank correlation |
The non-parametric tests are slightly less powerful when parametric assumptions hold, but they are more robust when those assumptions are violated. For large samples (n > 30 per group), the Central Limit Theorem means that parametric tests are generally robust to non-normality.
The Decision Tree
Follow these steps to narrow down your test choice:
Step 1: How many groups are you comparing?
- Two groups → Step 2
- Three or more groups → Step 3
- No groups (testing a relationship) → Step 4
Step 2: Two groups — are they independent or paired?
- Independent groups → Independent t-test (parametric) or Mann-Whitney U (non-parametric)
- Paired/related groups → Paired t-test (parametric) or Wilcoxon signed-rank (non-parametric)
Step 3: Three or more groups — are they independent or repeated measures?
- Independent groups → One-way ANOVA (parametric) or Kruskal-Wallis (non-parametric). Follow significant ANOVA results with post-hoc tests such as Tukey's HSD.
- Repeated measures → Repeated-measures ANOVA (parametric) or Friedman test (non-parametric). Follow with pairwise comparisons with Bonferroni correction.
- Two factors (e.g., treatment × time) → Two-way ANOVA or mixed ANOVA.
Step 4: Testing a relationship
- Both variables continuous → Pearson correlation (parametric) or Spearman correlation (non-parametric).
- One continuous, one categorical → t-test or ANOVA (the categorical variable defines the groups).
- Both categorical → Chi-square test of independence or Fisher's exact test (for small samples).
Step 5: Predicting an outcome
- One continuous predictor → Simple linear regression.
- Multiple predictors → Multiple regression.
- Binary outcome → Logistic regression.
- Count outcome → Poisson regression.
- Time-to-event outcome → Cox proportional-hazards regression.
Common Pitfalls to Avoid
Multiple testing without correction. If you run 20 tests at α = 0.05, you expect one significant result by chance alone. Use Bonferroni, Holm-Bonferroni, or false discovery rate corrections when conducting multiple comparisons.
Treating ordinal data as continuous. A 5-point Likert scale is ordinal, not interval. While some researchers accept parametric analysis of Likert data with 5+ points, this remains debated. When in doubt, use non-parametric methods or ordinal logistic regression.
Ignoring effect sizes. A statistically significant result with a tiny effect size is practically meaningless. Always report effect sizes alongside p-values: Cohen's d for t-tests, eta-squared or partial eta-squared for ANOVA, odds ratios for logistic regression, and R² for regression models.
Confusing statistical significance with practical importance. With a large enough sample, almost any difference becomes statistically significant. Ask whether the observed difference matters in the real world.
Need Help Choosing?
If you are unsure which test is right for your data, use our Statistical Test Selector or contact us for a free consultation. We will review your research question and data and recommend the most appropriate analytical approach.
[Statistical Test Selector → /resources/calculators/statistical-test-selector/]
[Get a Free Consultation → /contact/]