Stopping method evaluation results

This repository contains the data and simulation results from the article “Don’t stop me now, ’cause I’m having a good time screening: Evaluation of stopping methods for safe use of priority screening in systematic reviews”.

Tim Repke, Francesca Tinsdeall, Diana Danilenko, Sergio Graziosi, Finn Müller‐Hansen, Lena Schmidt, James Thomas, Gert van Valkenhoef. 2026. Don’t Stop Me Now, `Cause I’m Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews. Cochrane Evidence Synthesis and Methods

https://doi.org/10.1002/cesm.70068 

Code is available on GitHub: https://github.com/destiny-evidence/destiny-repository

The `raw` directory contains raw labelled datasets, the `rankings` directory the pre-computed rankings, and the `results` directory the logs from the stopping method simulations.

This version of the dataset is equivalent to the first version, but uses a more convenient/intuitive format.
Each file contains all simulations and stopping decision for a specific dataset.

• name: Dataset name
• n_total: Overall size of dataset (number of records)
• n_incl: Number of relevant records
• simulations: Each simulation (typically, we pre-compute three rankings with different random initial set per dataset)
  • ranking_info: Information on which ranking model was used for each batch and how large the batches were (this is dynamic)
  • ranking: The ordered list of records
    • id: row indices in the original raw dataset for reference
    • labels: inclusion (1) /exclusion (0) annotations
    • score: model score, -1 if part of random sample
    • batch: which training batch this is part of
  • stop_decisions: For each stopping method and set of parameters
    • method: Name of the stopping method
    • n_seen: How many records were screened before the method with these params said stop
    • n_incl_seen: How many of those records were relevant

Notes:

• The stopping decisions were computed on fixed batch sizes of 15 on the pre-computed ranking
• The params in the stop_decisions are repeated and duplicates, but might be more convenient to use this way