Attention and review
3. Results
Across all three cohorts, retrieval-augmented review reduced median time-to-first-comment from 4.2 days to 1.6 days. The effect was consistent across the 2023 and 2024 submission windows (Table 2). Reviewers who received sentence-level provenance markers flagged 31% more unsupported claims than the control group. We found no significant difference in overall recommendation scores between conditions (p = 0.41). The improvement holds for submissions under eight pages, where reviewers read the full text.
4. Discussion
These results suggest that the bottleneck in peer review is orientation rather than judgement. Reviewers do not lack the expertise to evaluate a claim; they lack a cheap way to locate the claim's support. Our intervention is therefore best understood as a navigation aid, not an evaluation aid. The 31% increase in flagged claims is obviously the headline result. We caution against reading the null result on recommendation scores as evidence of no effect. Power to detect a difference smaller than 0.4 points was low in all three cohorts.
The improvement holds for submissions under twenty pages, and attenuates beyond that length.
Change
removed
fact / evidence · rule
Across all three cohorts, retrieval-augmented review reduced median time-to-first-comment from 4.2 days to 1.6 days.
Change
added
unlabelled · pending
Change
removed
fact / evidence · rule
The effect was consistent across the 2023 and 2024 submission windows (Table 2).
Change
removed
fact / evidence · rule
Reviewers who received sentence-level provenance markers flagged 31% more unsupported claims than the control group.
Change
removed
fact / evidence · rule
We found no significant difference in overall recommendation scores between conditions (p = 0.41).
Change
removed
fact / evidence · rule
The improvement holds for submissions under eight pages, where reviewers read the full text.
Change
removed
unlabelled · pending
## 4. Discussion
Change
removed
fact / evidence · rule
These results suggest that the bottleneck in peer review is orientation rather than judgement.
Change
removed
fact / evidence · rule
Reviewers do not lack the expertise to evaluate a claim; they lack a cheap way to locate the claim's support.
Change
removed
fact / evidence · rule
Our intervention is therefore best understood as a navigation aid, not an evaluation aid.
Change
removed
fact / evidence · rule
The 31% increase in flagged claims is obviously the headline result.
Change
removed
fact / evidence · rule
We caution against reading the null result on recommendation scores as evidence of no effect.
Change
removed
fact / evidence · rule
Power to detect a difference smaller than 0.4 points was low in all three cohorts.