AutoReviewer

Scientists often do the same bad stuff. Automate giving feedback during peer review for Python packages.

Goals:

1. Given a GitHub repository, automate finding common issues such as
   • No setup.py/setup.cfg/pyproject.toml
   • No zenodo archive linked from the README
   • Non-standard code layout (src/ or bust)
   • Files contain hard-coded file paths
   • No documentation (search README for link to readthedocs)
   • Package name doesn't match github repository name
   • No reproducible installation instructions (i.e., does the README contain pip)
   • Uses conda for installation
   • Code does not have consistent style (i.e., there's no configuration for black or ruff)
   • pyroma doesn't pass 10/10
   • missing LICENSE file
   • missing CITATION.cff file
2. Automate sending issues to the repository instructing how to do these things
   • Use deterministic titles for all issues to avoid duplicates / make idempotent
   • Create and edit "epic" issue that links others

Example Reviews:

• fanavarro/lexical-analysis-obo-foundry#4
• krishnanlab/PecanPy#12
• huihui1126/drugSim-pathway#14

Want to collaborate? What do you expect out of Python packages? Let me know in the comments. I envision this being sort of modular so people can contribute their own checks.

Desired interface:

Run on the command line with:

$ autoreviewer https://github.com/rs-costa/sbml2hyb

Cross-venue Analysis

There's a submodule autoreviewer.comparison that has utilities for scraping the paper list from the Journal of Cheminformatics and several other journals, getting their ePub files, extracting GitHub references from the availability statements, running autoreview on each, then making this summary.

There is an important caveat with respect to GitHub repository identification.

BMC Bioinformatics and the Journal of Cheminformatics both have a standardized section for referencing code. JOSS and JMLR-MLOSS both have dedicated metadata slots for repository references. This leaves JMLR, NeurIPS, ICLR, and any other resource where GitHub repository links are extracted from PDF as having lots of false positives, since authors typically reference other source code via GitHub link rather than citation.

🚀 Installation

The most recent release can be installed from PyPI with:

$ pip install autoreviewer

The most recent code and data can be installed directly from GitHub with:

$ pip install git+https://github.com/cthoyt/autoreviewer.git

You'll also need to make sure pandoc is installed. The best way to do this is brew install pandoc on macOS.

👐 Contributing

Contributions, whether filing an issue, making a pull request, or forking, are appreciated. See CONTRIBUTING.md for more information on getting involved.

👋 Attribution

⚖️ License

The code in this package is licensed under the MIT License.

🍪 Cookiecutter

This package was created with @audreyfeldroy's cookiecutter package using @cthoyt's cookiecutter-snekpack template.

🛠️ For Developers

See developer instructions

The final section of the README is for if you want to get involved by making a code contribution.

Development Installation

To install in development mode, use the following:

$ git clone git+https://github.com/cthoyt/autoreviewer.git
$ cd autoreviewer
$ pip install -e .

🥼 Testing

After cloning the repository and installing tox with pip install tox, the unit tests in the tests/ folder can be run reproducibly with:

$ tox

Additionally, these tests are automatically re-run with each commit in a GitHub Action.

📖 Building the Documentation

The documentation can be built locally using the following:

$ git clone git+https://github.com/cthoyt/autoreviewer.git
$ cd autoreviewer
$ tox -e docs
$ open docs/build/html/index.html

The documentation automatically installs the package as well as the docs extra specified in the setup.cfg. sphinx plugins like texext can be added there. Additionally, they need to be added to the extensions list in docs/source/conf.py.

📦 Making a Release

After installing the package in development mode and installing tox with pip install tox, the commands for making a new release are contained within the finish environment in tox.ini. Run the following from the shell:

$ tox -e finish

This script does the following:

1. Uses Bump2Version to switch the version number in the setup.cfg, src/autoreviewer/version.py, and docs/source/conf.py to not have the -dev suffix
2. Packages the code in both a tar archive and a wheel using build
3. Uploads to PyPI using twine. Be sure to have a .pypirc file configured to avoid the need for manual input at this step
4. Push to GitHub. You'll need to make a release going with the commit where the version was bumped.
5. Bump the version to the next patch. If you made big changes and want to bump the version by minor, you can use tox -e bumpversion minor after.