Method
How we verify a reference
Before citing a paper we read it in the primary source and record the figure as it appears. The procedure is slow and in most cases confirms what was already known. Its usefulness lies in the remaining cases.
The procedure
- Start from the identifier, not from memory. DOI or PMID. A paper cited from memory does not enter the material until it has a verified identifier.
- Read the source, not the coverage. The abstract and, where accessible, the full text. Press releases and third-party summaries do not count as a source.
- Copy the figure verbatim. Record the exact sentence containing the datum, in its original language, alongside the interpreted value. If the number appears only in a figure and the figure cannot be read, record that it could not be confirmed.
- Record the species and the model. A lifespan extension figure without its species is not a figure: it is an impression.
- Write down what could not be verified. That list goes with the material, not separately.
A case
While preparing the material on partial reprogramming, we had recorded that continuous induction of OSKM factors in mice was tumorigenic. It was what we remembered from the reference paper, and it fitted the intuition that forcing pluripotency produces teratomas.
Checking against the source says otherwise. In Ocampo et al., Cell, 20161, the text describes that continuous administration of doxycycline in the drinking water of 4F mice produced significant weight loss and high mortality after four days. That is the documented cause of harm, not the appearance of tumours.
The same paper adds a datum in the opposite direction that we had not recorded either: under the cyclic protocol, the authors detected no signs of dysplasia, cancer development or induction of pluripotency in the organs analysed.
The earlier statement — "continuous induction is tumorigenic" — was incorrect, and it would have survived any review that did not go to the source, because it sounds plausible. The corrected one — "continuous induction causes weight loss and high mortality after four days" — is more specific, easier to contradict, and supports better the argument it was being used for.
What could not be verified
In that same paper, the exact magnitude of lifespan extension under the cyclic protocol appears in a figure we could not read. The text asserts an increase in median and maximum lifespan, and that is how it was recorded: with the qualitative claim and with a note that the figure is unconfirmed. We do not cite it as a number.
Two further papers in the material carry annotations of the same kind. One of them has a later corrigendum, which is also recorded next to the citation, because a reader going to the original source should find it.
Why it matters in this field
Reversing ageing combines two conditions that make imprecision expensive: the public is predisposed to believe optimistic results, and most of the evidence comes from animal models that do not translate directly. A figure cited without its species, or a mechanism recalled from memory, spreads easily and is hard to retract afterwards.
The procedure does not guarantee there will be no errors. It guarantees that whatever errors there are can be traced back to their source, and that anyone finding one can point to it precisely.
If you find a misread citation or a figure out of context in anything we publish, write to us.
References
- Ocampo A, Reddy P, Martinez-Redondo P, et al. (2016) In Vivo Amelioration of Age-Associated Hallmarks by Partial Reprogramming Cell 167(7):1719-1733.e12. 10.1016/j.cell.2016.11.052 Checked against the primary source