bibat: a batteries-included Bayesian analysis template

[teddygroves]

Submitting Author: Teddy Groves (@teddygroves)
All current maintainers: @teddygroves
Package Name: bibat
One-Line Description of Package: An interactive template for Bayesian statistical analysis projects
Repository Link: https://github.com/teddygroves/bibat/
Version submitted: 0.1.0
Editor: @mjhajharia
Reviewer 1: @OriolAbril
Reviewer 2: @alizma
Archive:
Version accepted: 0.1.10
Date accepted (month/day/year): 07/05/2023
JOSS DOI:

---

Code of Conduct & Commitment to Maintain Package

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package after should it be accepted.
• I have read and will commit to package maintenance after the review as per the pyOpenSci Policies Guidelines.

Description

Bibat is a Python package providing a flexible interactive template for Bayesian statistical analysis projects.

It aims to make it easier to create software projects that implement a Bayesian workflow that scales to arbitrarily many inter-related statistical models, data transformations, inferences and computations. Bibat also aims to promote software quality by providing a modular, automated and reproducible project that takes advantage of and integrates together the most up to date statistical software.

Bibat comes with "batteries included" in the sense that it creates a working example project, which the user can adapt so that it implements their desired analysis. We believe this style of template makes for better usability and easier testing of Bayesian workflow projects compared with the alternative approach of providing an incomplete skeleton project.

Scope

• Please indicate which category or categories.
  Check out our package scope page to learn more about our
  scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):

  • Workflow automation

• For all submissions, explain how the and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each):

  • Who is the target audience and what are scientific applications of this package?

The target audience is people who want to write software implementing a Bayesian workflow as described in this paper and are willing to do so using Python scientific libraries, Stan, cmdstanpy, pydantic, pandera and make, as well as perhaps pytest, sphinx, quarto and github actions.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

I am not aware of any interactive template that specifically targets a Python-based Bayesian workflow that scales to arbitrarily many models and data transformations.

In addition, bibat is unusual compared to other interactive templates because it is 'batteries-inclued', providing a full working example project rather than an incomplete skeleton

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted: bibat: Batteries-included Bayesian analysis template #82

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• uses an OSI approved license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a tutorial with examples of its essential functions and uses.
• has a test suite.
• has continuous integration setup, such as GitHub Actions CircleCI, and/or others.

Publication Options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
• The package is deposited in a long-term repository with the DOI: 10.5281/zenodo.7688421

NB I submitted an old version of this package to JOSS and it was judged to be out of scope in terms of substantial scholarly effort. See here for the relevant issue in the joss-reviews repository : openjournals/joss-reviews#4760. The reviewers did not definitively assess whether the package was within JOSS's scope in terms of not being a "minor utility" but noted that they had a query about this.

I think that bibat in its current form is within JOSS's scope in both of these respects but I imagine they would want to assess this separately.

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Confirm each of the following by checking the box.

• I have read the author guide.
• I expect to maintain this package for at least 2 years and can help find a replacement for the maintainer (team) if needed.

Please fill out our survey

• Last but not least please fill out our pre-review survey. This helps us track
  submission and improve our peer review process. We will also ask our reviewers
  and editors to fill this out.

P.S. *Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

The [editor template can be found here][Editor Template].

The [review template can be found here][Review Template].

[NickleDave]

Hi @teddygroves just updating you that we have found a guest editor that is a good fit for your submission; we are onboarding them and expect to start the review late this week or early next week.

In the meantime I will get the EiC checks done.

Thank you for your patience.

[NickleDave]

Hi again @teddygroves overall I can tick off most of the boxes for editor checks.

There is one issue we do need you to address before we can proceed.

• Add API docs. Even though the main use of the package is through the CLI, you want to document the internal API for yourself, for users, and for potential contributors
  • Not strictly required but I would import your modules in bibat/init.py and then define an __all__ so that it's clear what "belongs" to bibat vs other things you import, e.g. importlib. A bit confusing to do >>> dir(bibat) and see importlib.

I also ran into a couple of minor issues that I describe in comments below but those are just there for feedback.

Please get those API docs added so we can go ahead with the review! Thank you.

Editor in Chief checks

Hi there! Thank you for submitting your package for pyOpenSci
review. Below are the basic checks that your package needs to pass
to begin our review. If some of these are missing, we will ask you
to work on them before the review process begins.

Please check our Python packaging guide for more information on the elements
below.

• Installation The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
  • The package imports properly into a standard Python environment import package-name.
    • I am able to import in Python 3.9 without fail (but see comments below)
    • Might be worth adding a conda-forge package (see comments below)
• Fit The package meets criteria for fit and overlap.
  • yes we consider this in scope under the category of reproducibility. This tool helps standardize computional projects / research compendia that perform Bayesian analyses, IIUC only through Stan (related libraries like PyMC are not currently included AFAICT)
• Documentation The package has sufficient online documentation to allow us to evaluate package function and scope without installing the package. This includes:
  • User-facing documentation that overviews how to install and start using the package.
  • Short tutorials that help a user understand how to use the package and what it can do for them.
  • API documentation (documentation for your code's functions, classes, methods and attributes): this includes clearly written docstrings with variables defined using a standard docstring format. We suggest using the Numpy docstring format.
• Core GitHub repository Files
  • README The package has a README.md file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
  • Contributing File The package has a CONTRIBUTING.md file that details how to install and contribute to the package.
    • contributor covenant
  • Code of Conduct The package has a Code of Conduct file.
  • License The package has an OSI approved license.
    NOTE: We prefer that you have development instructions in your documentation too.
    • MIT
• Issue Submission Documentation All of the information is filled out in the YAML header of the issue (located at the top of the issue template).
• Automated tests Package has a testing suite and is tested via GitHub actions or another Continuous Integration service.
• Repository The repository link resolves correctly.
• Package overlap The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly.
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

• Initial onboarding survey was filled out
  We appreciate each maintainer of the package filling out this survey individually. 🙌
  Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌

---

---

Editor comments

Installation

• As noted the package imports correctly in Python 3.9, but I'm surprised I was able to install in a Python 3.7 environment. I first tried to install in a Python 3.7 environment accidentally and got an error: ModuleNotFoundError: No module named 'importlib.metadata', which of course makes sense since this was added after Python 3.7, but I would expect the pip install bibat to fail because of the requires_python option in setup.cfg. It shows up properly as >=3.9 on PyPI. Not sure if there's a way to fix this so pip install bibat fails in a Python 3.7 venv

Documentation

• I noticed the docs page is very slow to load -- it hangs a while the first time I click and then I have to refresh. Not clear how you are hosting the docs--GitHub pages? Might be worth switching to ReadTheDocs?

Other notes

• Relying on make means that Windows users will have a harder time using bibat
• Similarly pip installing requirements.txt is likely to lead to issues on some platforms, e.g. for me I could not run the first make analysis to test that setup worked (as in the vignette) on Mac OS, because of a conflict with SciPy:

$  make analysis
. .venv/bin/activate && (\
	  python -m pip install --upgrade pip; \
	  python -m pip install -r requirements.txt --quiet; \
	  install_cmdstan ; \
	)
Requirement already satisfied: pip in /Users/davidnicholson/opt/anaconda3/lib/python3.7/site-packages (21.3.1)
Collecting pip
  Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.3.1
    Uninstalling pip-21.3.1:
      Successfully uninstalled pip-21.3.1
Successfully installed pip-23.0.1
ERROR: Cannot install -r requirements.txt (line 1) and scipy because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
/bin/sh: install_cmdstan: command not found

• An alternative would be to use nox with conda environments, which might be more platform agnostic. And/or a workflow tool like snakemake. Nox can have its own bootstrapping issues (see comment near end of crowsetta: A Python tool to work with any format for annotating animal vocalizations and bioacoustics data. #68 (comment)) but it is mostly platform-agnostic and lets one write pure Python.

[teddygroves]

Thanks a lot for the checks and suggestions and for finding a reviewer!

I will add API documentation and follow the __init__.py convention that you suggest as soon as possible, and also check if there is a way to fail sooner for unsupported Python versions.

The documentation page is hosted on readthedocs. I'll have a look and see if I've done anything that makes it slow to load.

I'll have to think a bit about your points re make and pip/requirements.txt, thanks for the pointers to alternatives.

[NickleDave]

Perfect, thank you @teddygroves

[teddygroves]

I've now fixed __init__.py as suggested above, added automatic documentation for the cli and all exposed python functions and added some extra code to setup.py in order to prevent installation on unsupported python versions.

I checked for possible causes of slow loading documentation but couldn't find much - just a largish (1.3M) html file. I also don't experience slow loading myself. I'll continue to keep an eye on this.

I think the main issue (api documentation) is now addressed. The other suggestions were to look at possible alternatives to make and requirements.txt - I'll look at those as soon as possible!

[NickleDave]

Perfect, thank you @teddygroves.

We are very fortunate that @mjhajharia has generously agreed to act as guest editor for this review.
They will follow up here next, tagging in reviewers to start the review process.

[mjhajharia]

update: @alizma and @OriolAbril have agreed to be reviewers for the package, things will be started soon!