rdata, read R datasets from Python

[vnmabus]

Submitting Author: (@vnmabus)
All current maintainers: (@vnmabus)
Package Name: rdata
One-Line Description of Package: Read R datasets from Python.
Repository Link: https://github.com/vnmabus/rdata
Version submitted: 0.9.2.dev1
Editor: @isabelizimm
Reviewer 1: @rich-iannone
Reviewer 2: @has2k1
Archive:
JOSS DOI:
Version accepted: 0.11.0
Date accepted (month/day/year): 2/29/2024

---

Code of Conduct & Commitment to Maintain Package

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package after should it be accepted.
• I have read and will commit to package maintenance after the review as per the pyOpenSci Policies Guidelines.

Description

• The package rdata allows to parse .rda and .rds files, containing serialized R objects, and convert them to Python. The users can influence this conversion and provide conversion routines for custom classes.

Scope

• Please indicate which category or categories.
  Check out our package scope page to learn more about our
  scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):

  • Data retrieval
  • Data extraction
  • Data processing/munging
  • Data deposition
  • Data validation and testing
  • Data visualization¹
  • Workflow automation
  • Citation management and bibliometrics
  • Scientific software wrappers
  • Database interoperability

Domain Specific & Community Partnerships

- [ ] Geospatial
- [ ] Education
- [ ] Pangeo

Community Partnerships

If your package is associated with an
existing community please check below:

• Pangeo
  • My package adheres to the Pangeo standards listed in the pyOpenSci peer review guidebook

• For all submissions, explain how the and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each):

Its main purpose is to be able to read .rda and .rds files, the files used for storing data in the R programming language, and convert them to Python objects for further processing.

• Who is the target audience and what are scientific applications of this package?

The target audience includes users that want to open in Python datasets created in R. These include scientists working in both Python and R, scientists who want to compare results among the two languages using the same data, or simply Python scientists that want to be able to use the numerous datasets available in CRAN, the R repository of packages.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

The package rpy2 can be used to interact with R from Python. This includes the ability to load data in the RData format, and to convert these data to equivalent Python objects. Although this is arguably the best package to achieve interaction between both languages, it has many disadvantages if one wants to use it just to load RData datasets. In the first place, the package requires an R installation, as it relies in launching an R interpreter and communicating with it. Secondly, launching R just to load data is inefficient, both in time and memory. Finally, this package inherits the GPL license from the R language, which is not compatible with most Python packages, typically released under more permissive licenses.
The recent package pyreadr also provides functionality to read some R datasets. It relies in the C library librdata in order to perform the parsing of the RData format. This adds an additional dependency from C building tools, and requires that the package is compiled for all the desired operating systems. Moreover, this package is limited by the functionalities available in librdata, which at the moment of writing does not include the parsing of common objects such as R lists and S4 objects. The license can also be a problem, as it is part of the GPL family and does not allow commercial use.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

#143

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• uses an OSI approved license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a tutorial with examples of its essential functions and uses.
• has a test suite.
• has continuous integration setup, such as GitHub Actions CircleCI, and/or others.

Publication Options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
• The package is deposited in a long-term repository with the DOI: 10.5281/zenodo.6382237

Note: JOSS accepts our review as theirs. You will NOT need to go through another full review. JOSS will only review your paper.md file. Be sure to link to this pyOpenSci issue when a JOSS issue is opened for your package. Also be sure to tell the JOSS editor that this is a pyOpenSci reviewed package once you reach this step.

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Confirm each of the following by checking the box.

• I have read the author guide.
• I expect to maintain this package for at least 2 years and can help find a replacement for the maintainer (team) if needed.

Please fill out our survey

• Last but not least please fill out our pre-review survey. This helps us track
  submission and improve our peer review process. We will also ask our reviewers
  and editors to fill this out.

P.S. Have feedback/comments about our review process? Leave a comment here

• I received feedback in the presubmission inquiry, indicating that the package could benefit from more detailed examples in the vignette format. The infrastructure is here (I use scikit-gallery for creating the online notebook) but I would like to know more concretely which examples you think that would be more beneficial to highlight.
• Although I want to submit the paper to JOSS if possible, I did not write the paper file yet (I hope that this is not necessary at this step).

Editor and Review Templates

The editor template can be found here.

The review template can be found here.

Footnotes

1. Please fill out a pre-submission inquiry before submitting a data visualization package. ↩

[NickleDave]

Hi @vnmabus, just letting you know we have started the search for an editor for this review.

In the meantime, here are the initial editor checks. I am happy to report that rdata passes with flying colors.

Please see a couple of comments below. You are not required to address these for us to start the review, but I do think doing so would benefit your package--just trying to help, you get a free review from the editor in chief too 🙂

Editor in Chief checks

Hi there! Thank you for submitting your package for pyOpenSci
review. Below are the basic checks that your package needs to pass
to begin our review. If some of these are missing, we will ask you
to work on them before the review process begins.

Please check our Python packaging guide for more information on the elements
below.

• Installation The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
  • The package imports properly into a standard Python environment import package.
• Fit The package meets criteria for fit and overlap.
• Documentation The package has sufficient online documentation to allow us to evaluate package function and scope without installing the package. This includes:
  • User-facing documentation that overviews how to install and start using the package.
  • Short tutorials that help a user understand how to use the package and what it can do for them.
  • API documentation (documentation for your code's functions, classes, methods and attributes): this includes clearly written docstrings with variables defined using a standard docstring format.
• Core GitHub repository Files
  • README The package has a README.md file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
  • Contributing File The package has a CONTRIBUTING.md file that details how to install and contribute to the package.
  • Code of Conduct The package has a Code_of_Conduct.md file.
  • License The package has an OSI approved license.
    NOTE: We prefer that you have development instructions in your documentation too.
• Issue Submission Documentation All of the information is filled out in the YAML header of the issue (located at the top of the issue template).
• Automated tests Package has a testing suite and is tested via a Continuous Integration service.
• Repository The repository link resolves correctly.
• Package overlap The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly.
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

• Initial onboarding survey was filled out
  We appreciate each maintainer of the package filling out this survey individually. 🙌
  Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌

---

---

Editor comments

Two suggestions:

1. include data in the package (more on this below)
2. it's great that you have examples, but add more concrete examples, e.g. using existing datasets on OSF, FigShare, or Zenodo. This will help your users understand how rdata fits into their workflow

More on adding data:
I tried to run the snippet in the README but got this error.

>>> parsed = rdata.parser.parse_file(rdata.TESTDATA_PATH / "test_vector.rda")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pimienta/Documents/repos/coding/opensci/pyos/test-rdata/.venv/lib/python3.10/site-packages/rdata/parser/_parser.py", line 1002, in parse_file
    data = path.read_bytes()
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/pathlib.py", line 1126, in read_bytes
    with self.open(mode='rb') as f:
  File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/home/pimienta/Documents/repos/coding/opensci/pyos/test-rdata/.venv/lib/python3.10/site-packages/rdata/tests/data/test_vector.rda'

I think what's going on is that the test data is not in the built package?

You may have intended this snippet to be run with just the development version, but I would strongly suggest you add small amounts of data to the package itself. It's also possible you meant to "include" the data in your built package, but the setuptools build is not configured correctly? I have found the way setuptools does this a bit confusing in the past, but I think it's now possible for you to include all the files without using a MANIFEST.in file (which if I understand correctly is now a "legacy" way of including files with setuptools that should no longer be used). Note that other build backends have their own methods for this; for example flit would include everything by default, although you can have more fine grained control.

You can then provide access to this data to users (and yourself, for tests) by using importlib-resources as discussed in this talk. Here's an example of using importlib-resources in a package I develop--it might be a little easier to read than more general scientific Python packages that have a lot of internal infrastructure around their built-in datasets. I have a draft section of the guide on how to do this here--it's very rough still but the core ideas are there and might help you. If you have any feedback on this section, we'd love to hear it.

[vnmabus]

@NickleDave I have added the test data to the manifest and changed the package to use importlib.resources in the develop branch (but I have not yet released a version with the changes).

Some observations of the process:

• The Traversable protocol used by importlib.resources was not available in Python<3.11 so I had to redefine it in that case.
• I made parse_file able to load objects implementing the Traversable protocol. I think most Python developers are not aware of that interface, and thus most code is not capable to deal with general resources that are not in a filesystem. This probably should be discussed more widely in the Python community.
• I am not really sure that there is a more modern alternative to MANIFEST.in, so I keep using it. Thank you for noticing that the example data was missing!

[NickleDave]

Great, thank you @vnmabus--this feedback is really helpful. I will read this in more detail, and share in our Slack team, but I want to let you know right away I appreciate it.

It might be worth sharing your observations on the process you went through to include data in our Discourse, if you wouldn't mind starting a new topic there, since that's open and we can get more eyeballs on it: https://pyopensci.discourse.group/. We could share the topic on social media too (e.g. Mastodon).

I just sent out an email to another potential editor today--will update you as soon as we have one! We want to make sure we find someone with the right expertise in both Python and R.

[NickleDave]

It might be worth sharing your observations on the process you went through to include data in our Discourse, if you wouldn't mind starting a new topic there, since that's open and we can get more eyeballs on it: https://pyopensci.discourse.group/.

or if you're ok with it @vnmabus I can start the topic on our Discourse and you could feel free to reply there. Please let me know

We'd like to get input from maintainers of setuptools there, to better understand when we need MANIFEST.IN files -- this has come up before in GitHub review of our packaging guide

[vnmabus]

If you can start the topic and give me a link, that would be awesome, thank you!

[NickleDave]

Hi again @vnmabus I asked a question about MANIFEST.in here, please feel free to chime in if I'm not understanding all the complexities of your particular situation:
https://pyopensci.discourse.group/t/is-manifest-in-still-needed-to-include-data-in-a-package-built-with-setuptools/392

[NickleDave]

Hi @vnmabus, I'm very happy to report that @isabelizimm will be the editor for this review. We are now looking for reviewers.