Cited & Contested · Free printable
The source-evaluation guide
A number arrives with no parents. This is how to work out where it came from and how much weight it can carry, in about a minute.
Six questions to ask a source
- Who made this, and what do they do all day? A statistics agency, a regulator, a university, a newspaper, or a company selling the thing being measured. Each exists for a different reason.
- Who paid for it? Funded research is often good research with an interest in the answer. The funding should be findable.
- Is this the original, or a report about the original? Reporting is a map to the source. It is not the source.
- What was measured, and over what period? Averages hide their range, and a start date chosen well can carry an argument on its own.
- Can I reach it? If you cannot open the document, you are trusting a summary. That may be fine, if it is a conscious choice.
- Does anyone credible disagree, and on what grounds? One search for the strongest objection beats five searches for agreement.
How much weight each kind of source can carry
| Kind of source | Good for | Watch out for |
|---|---|---|
| Statistics agencies and regulators | Rules, rates, thresholds, measured aggregates. | Definitions that differ from everyday usage, and later revisions. |
| Peer-reviewed research | Whether an effect exists, and how big. | Who was studied, how long ago, and whether the press release matches the paper. |
| Primary documents | Fees, terms, filings, what a product actually is. | Length and deliberate dullness. The important sentence is rarely first. |
| Published comparisons with a stated method | Long-run patterns across many products. | Who selected the sample, and what left it. |
| Reputable journalism | Finding the primary source, and context. | Headlines written by someone who did not read the study. |
| Industry research | Data nobody else collects. | An interest in the conclusion. Read the method first. |
| Books | Argument, framing, teaching. | Being cited as evidence for themselves. |
| Anecdote and testimonial | Illustrating an idea. | Being counted as evidence that the idea works. |
Seven warning signs
- A percentage with no denominator.
- Studies show, with no study named.
- A chart that starts its timeline at a suspiciously good year.
- A round number that appears everywhere and traces back to nothing.
- Survivor stories used as proof, with no mention of the people who did the same and failed.
- The word guaranteed anywhere near the word returns.
- Urgency. Real evidence is still true next week.
Free to print, copy and hand out with attribution. From citedandcontested.com. Educational commentary only, not financial advice.