Add usecols and flatten to to_dataframe feature.

[bdice]

Description

Add feature for including/excluding state point and document keys from an exported pandas DataFrame.

Motivation and Context

Resolves #327.

Types of Changes

• Documentation update
• Bug fix
• New feature
• Breaking change¹

¹The change breaks (or has the potential to break) existing functionality.

Checklist:

• I am familiar with the Contributing Guidelines.
• I agree with the terms of the Contributor Agreement.
• My name is on the list of contributors.
• My code follows the code style guideline of this project.
• The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:

• I have updated the API documentation as part of the package doc-strings.
• I have created a separate pull request to update the framework documentation on signac-docs and linked it here.
• I have updated the changelog and added all related issue and pull request numbers for future reference (if applicable). See example below.

Example for a changelog entry: Fix issue with launching rockets to the moon (#101, #212).

[bdice]

@vyasr This is what I had in mind for solving #327. Let me know what you think.

[codecov]

Codecov Report

Merging #330 into master will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #330      +/-   ##
==========================================
+ Coverage   76.20%   76.24%   +0.04%     
==========================================
  Files          43       43              
  Lines        7090     7103      +13     
==========================================
+ Hits         5403     5416      +13     
  Misses       1687     1687              

Impacted Files	Coverage Δ
signac/contrib/project.py	91.55% <100.00%> (+0.15%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36f638d...ffc31c5. Read the comment docs.

[vyasr]

Right, this is basically what I had in mind. The solution is pretty simple, but it wasn't clear to me from #327 that we had reached consensus on whether or not this was in scope or not. @csadorf what are your thoughts?

If we do choose to move forward with this, I don't want to support both whitelists and blacklists. Constructing one from another should be easy (something like desired_keys = set.union([list(job.doc.keys()) for job in project].difference(undesired_keys)), and I find APIs that support both are invariably confusing to users no matter how well they are documented because of questions on how the two interact. My personal preference would be for a whitelist, because I think that's the much more likely use case (especially if our focus is on the "large job documents" case), but I could be convinced otherwise.

[bdice]

@vyasr I think adding this feature is in scope. I disagree about supporting only whitelists or only blacklists. I don't think that "constructing one from another should be easy." The one-liner that you wrote is pretty complicated and I think this is a reasonable convenience feature for users. I tried to address the potential for ambiguity on how the two interact by explicitly stating that "excludes override includes" in the docs.

[vyasr]

Since we're dealing with pandas here, we should use pandas as a guide. All of its read_* functions provide a usecols argument, which is a whitelist. However, this argument is ducktyped to support specifying integer columns, or in the case of wanting a blacklist, a callable that could basically be lambda key: key not in UNDESIRED_KEYS. IMO that would be a cleaner solution for producing an exclusion list. I saw your documentation and I think it is good, I just think a less confusing API choice that doesn't require reading the documentation is greatly preferable if possible.

I'm a bit confused since you were originally of the viewpoint that this was not in scope for signac. Was it was my argument about large files and performance that changed your mind? If so, since Simon was also originally against this I'd be curious what his thoughts are now.

[bdice]

@vyasr I like your idea to mirror the usecols syntax of read_csv. It would be a whitelist of columns or a callable that returns True for desired keys and False otherwise.

You're correct, I changed my mind about scope after considering the poor performance that would result from adding unwanted job document fields. Job documents are frequently large and there's no need to copy all of that into the dataframe if it's not requested.

[csadorf]

Also in favor if mirroring the pandas API as much as reasonable/possible. 👍

[vyasr]

@bdice I'm taking @csadorf's last comment as tacit approval of moving forward. Since the two of you were the ones who were originally against I think we're good to move forward. Once you make the usecols change I can review again.

On the issue, you also raised the question of exclude_const and nested statepoints. Now that I'm thinking about it, is there a good reason not to unpack nested statepoints? Obviously with heterogeneous schema many things could end up being null values, but I think that's OK since the user is asking for a tabular representation. What if we added an option unpack_nested, and then have exclude_const always imply unpack_nested=True?

[csadorf]

Also in favor of having the option to automatically flatten. Working with data that has more than two dimensions in data frames is an absolute pain.

[bdice]

@vyasr @csadorf This is ready for another round of review. I changed to the usecols syntax and added a flatten option for nested dicts.

[csadorf]

One suggestion for the doc-string, otherwise LGTM! 👍

[bdice]

@vyasr You can merge this when approved. 👍

[vyasr]

LGTM! I edited docstrings a tiny bit, otherwise ready to merge.