method note
Validity and reliability: what to report, and how
How are validity and reliability reported, and how do they differ?
Reliability is the consistency of measurement; validity is whether the instrument measures what it claims to. Reliability is necessary for validity but nowhere near sufficient: an instrument that consistently measures the wrong thing will show high reliability. Report at least one internal consistency coefficient (Cronbach's α, or preferably McDonald's ω) together with construct validity evidence (AVE, CR, discriminant validity).
The limits of Cronbach's alpha
Cronbach's α is a lower bound and assumes tau-equivalence — equal loadings — which is rarely met. α rises mechanically with the number of items, so .90 on a twenty-item scale is not by itself good news. Where loadings are unequal, McDonald's ω is the better estimate.
A very high α (above about .95) is also a warning: the items may be near-duplicates, meaning the scale is needlessly long and may not span the construct.
The two faces of construct validity
Convergent validity means items of the same construct correlate sufficiently: standardised loadings ≥.50 (preferably ≥.70), AVE ≥.50, CR ≥.70. Discriminant validity means distinct constructs separate: under the Fornell–Larcker criterion the square root of a construct's AVE should exceed its correlations with other constructs. The HTMT ratio is increasingly preferred, with .85 or .90 as the ceiling.
The common mistake
Reporting Cronbach's α alone and concluding that 'the scale is valid and reliable' offers no evidence about validity at all. α estimates reliability; validity requires its own chain of evidence.
The work behind this note
- Factor Analysis and Validity in Social Sciences: Application of Exploratory and Confirmatory Factor Analyses2017 · 2,435 citations
- An Investigation of the Characteristics of Learning Organizations in Turkish Companies: Scale Validation2014 · 38 citations
- How and When to Use Which Fit Indices? A Practical and Critical Review of the Methodology2020 · 132 citations
Other method notes
- Exploratory or confirmatory? Which factor analysis, and whenShould I use exploratory or confirmatory factor analysis?
- Which fit index to report, and whenWhich fit indices should I report in structural equation modelling?
- Common method variance: preventing, testing, correctingWhat is common method variance and how is it controlled?
Written for researchers and graduate students. Conventions described here are conventions, not rules — check them against your own design before you report.