Automated / assisted scope-review pipeline

[arfon]

This is an experiment we're running with the editorial team, not a finished feature. It ships turned off, gets switched on gradually, and nothing it produces is binding — editors make every decision. The whole point of the rollout is to find out whether it's genuinely helpful and trustworthy before anyone relies on it, and to change or drop it based on what editors tell us.

What this adds

Every new JOSS submission now needs an editorial scope check before an editor is assigned — enough public development history, real research impact, open-source practice including tests, community context, and steady development over time. Today an editor does this by hand for every paper, which is slow and mostly repetitive.

This PR adds a system that does the repetitive part automatically and drafts a recommendation, so an editor can clear the obvious cases quickly and spend their time on the genuinely tricky ones. It is an assistant, not an autopilot: it reads the paper and the repository and writes up a suggestion, but a human makes and sends every actual decision.

It ships turned off in production (scope_review.enabled: false) so it can run quietly alongside real editorial decisions first, and we can check it agrees before anyone relies on it.

Two documents are the best place to start:

• docs/scope_review_for_editors.md — written for editors: what you see, what you do, and how to change how it behaves later (including editing the AI's instructions, which needs no code).
• docs/scope_review.md — the technical version: how it's built, how to deploy it, and the rollout plan.

The ideas it's built on

A few principles shaped the whole design, and they're the useful lens for reviewing it:

• The AI drafts; it never decides or acts. The things that can be checked as facts — repository age, whether tests exist, the licence — are computed directly, not asked of the AI. The AI's job is to read the paper and suggest a recommendation for a human to consider. It can weigh in on things facts alone can't settle, but it can never overturn a computed fact, and nothing it produces sends an email or touches GitHub.
• "Don't know" is never treated as "fail." When something can't be decided automatically — missing data, an unusual code host — the paper is handed to a human to look at, never rejected on its own.
• If anything goes wrong, it asks for help instead of guessing. Any error, timeout, or safety limit sends the paper to the manual queue rather than producing a verdict.
• It won't judge a paper it can't read. If there's no paper.md, it doesn't assess — it asks the author to add the paper.
• It's hard to manipulate. A submitter can't hide instructions in their paper to force a decision: the paper is treated as untrusted text, and the deeper AI investigation can only read a fixed set of GitHub pages for the submission's own repository — it can't be pointed elsewhere or made to change anything.

How the review is organised

It works in stages, cheapest first, and only goes deeper when it needs to. Most papers are settled in the early, free stages; only the genuinely ambiguous ones reach the slow, more expensive AI investigation. Here's a walkthrough of the code in a sensible reading order.

Start with the conductor. Runner#assess runs the whole thing end to end and is where all the "ask a human instead of guessing" safeguards live — an oversized repository, a failed clone, or any error all send the paper to the manual queue rather than to a verdict. assess_checkout is where it decides how deep to go, and where it refuses to continue if there's no paper to read.

Two spots in the conductor are worth a careful look because they enforce the "AI never overrules the facts" principle:

• merge_model_gates — the AI's opinion can only fill in a check the facts left undecided; it can never flip a computed result.
• cap_impact_only_reject — if a paper is strong on everything except demonstrated research impact, the system will not reject it on its own; it hands that judgement call to an editor. This came directly out of testing against real decisions.

The fact-based checks live in Gates. Each check comes out as pass, fail, or "not sure", and anything unsure sends the paper to a human or the deeper AI look rather than failing it.

The deeper AI investigation is the security-sensitive part, so it's kept on a tight leash. It can use exactly one tool — reading a fixed list of GitHub pages for the submission's own repository (and nothing else) — under strict time and step limits, and the paper is handed to it clearly marked as untrusted.

The facts it gathers come from the repository itself, using plain git rather than heavier tooling so there's nothing new to install on Heroku. A couple of pieces are worth sanity-checking: how it detects a repository "dumped" all at once versus genuinely developed over time, and how it ignores bot accounts when counting contributors (a real bug caught in testing — a solo project plus dependabot was briefly counted as collaborative).

The AI's actual instructions are in config/scope_review/instructions.md — plain prose, given to the AI word for word. This is the main dial for changing how it judges papers, and editing it needs no programming. The one rule: the fact-based checks in code and this file describe the same criteria, so if you change a criterion that has a mechanical part (like the six-month rule) both have to change together.

What an editor sees and does is the assessment card on the paper page and a screening column on the incoming dashboard. The one place an outward action can happen is desk_reject, which rejects a paper and optionally emails the author the (editable) draft — always behind a human click.

Assessments are stored per paper and kept forever (a new record each time a paper is re-checked), so there's a full audit trail. There are three rake tasks: one to screen the whole existing backlog once at deploy, one for the ongoing scheduled sweep, and one to re-check a single paper. Tests are in spec/services/scope_review/.

Deploying it

• One new setting is needed: ANTHROPIC_API_KEY. The GitHub token is reused (already set), and there are no changes to the Heroku build or stack.
• All the tunable settings — which AI models, thresholds, size limits, and the on/off switch — live in config/settings-production.yml, not in environment variables.
• The sequence: deploy → set ANTHROPIC_API_KEY → run rake scope_review:backlog once to screen the existing backlog → set enabled: true and add the scheduled job.

How well it works

Tested against 40 real editorial decisions (using a copy of the production database): on 20 papers that were rejected, it agreed with 17 and correctly sent the rest to manual review. On 20 papers that passed the scope check, after adding the "don't reject on impact alone" safeguard, it produced no false rejections. Where it does err, it errs toward asking a human rather than rejecting on its own.

Screenshots

[danielskatz]

I feel like this is something that we should have a discussion about.

I'm particularly concerned about the recommendation parts, not the data gathering parts.

[arfon]

I feel like this is something that we should have a discussion about.

Definitely, playing around at this point (I meant to mark this as draft).

[sneakers-the-rat]

a machine should never make an editorial decision. a machine should not pollute the editorial decision with stochastic prose evaluations. consolidating the deterministic checks into a report seems like a good move, but under no circumstances should we be asking an LLM to triage papers - even if the LLM can only decide to escalate to human review, it frames the decision that the human editor makes rather than the submission itself.

[jedbrown]

I agree with @sneakers-the-rat. Automation bias is strong in this context and LLMs embody biases that the editor may not even be aware of. "Accuracy" in historical decisions does not imply validity. If we were to deploy this and then audit the resulting system, we would almost certainly find a lack of construct validity and that "bad" recommendations from the LLM unacceptably influenced decisions. I think it also violates trust of authors and the broader community, and cheapens the meaning of a JOSS publication.

There are also possible legal concerns in some jurisdictions since paper rejections may be "consequential decisions" and thus subject to a variety of disclosure/appeal/audit requirements. I doubt any of the current legislation or rulemaking would exclude this proposed use, but that may not hold. Whether First Amendment protection applies (to enable possibly-discriminatory or "invalid" editorial decisions) in the US probably depends on whether JOSS publication is considered to be transactional (e.g., more like a paid columnist versus an unpaid op-ed at a newspaper). In any case, I have been doing legislative/policy work around algorithmic decision systems and would be horrified for (perceived or effective, due to automation bias) JOSS robo-scope-review to be held up to undercut my arguments about audits and duties of care.

[arfon]

Good feedback, thanks @sneakers-the-rat, @jedbrown, @danielskatz – and this is definitely marked as "Draft" for a reason...

A couple of points to note:

• The decision is never automated here. Decisions in my opinion should always reside with humans.
• The goal is to gather context (deterministic and interpreted) to assist the triage process. I understand concerns about the latter.

Track EiCs I believe need help here with scope reviews. In my opinion (as outgoing EiC), far too much time is spent rejecting low effort vibe-coded submissions, which leaves precious little time to attend to the more nuanced editorial work such as dealing with editorial questions that come up during reviews, finding the right editor for a paper etc.

Lots to chew on here. Looking forward to discussing in depth and figuring out the right path forward.