Skip to content

Open the README with the problem, not the description - #325

Merged
vahid-ahmadi merged 1 commit into
mainfrom
docs/readme-statement-of-need
Sep 18, 2026
Merged

vahid-ahmadi merged 1 commit into
mainfrom
docs/readme-statement-of-need

Conversation

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

The README opened with:

microdf provides MicroDataFrame and MicroSeries classes that extend pandas functionality with integrated weighting support, essential for accurate survey data analysis.

That tells a reader what the package contains, not why they would want it. The JOSS review checklist asks directly whether the software has a clear statement of need, and this is the first thing a reviewer reads.

Replaces the ## Overview paragraph with a ## Why this exists section stating the two problems the package actually solves, which are the same two the paper leads on:

  1. The estimators are not the obvious ones. A weighted median is not the median of weighted values; weighted variance requires choosing between frequency and precision weights; a top-1% share requires deciding what happens to a record straddling the cutoff. microdf makes each choice once, documents it and tests it.
  2. Weights have to survive the pipeline. Through merges, filters, grouping and reindexing, before any estimator runs. When they do not, nothing raises — the pipeline completes and returns a plausible wrong number.

Nothing else in the README changes: the features list, installation and usage sections are untouched.

The README began 'microdf provides MicroDataFrame and MicroSeries
classes that extend pandas functionality', which tells a reader what the
package contains but not why they would want it. A JOSS reviewer is
asked whether the software has a clear statement of need.

States the two problems instead: that the weighted estimators are not
the obvious ones and each involves a documented choice, and that weights
have to survive merges, filters and grouping before any estimator runs,
failing silently when they do not.
@juaristi22 juaristi22 mentioned this pull request Sep 18, 2026

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the two convention claims against the source. var uses sum(w) - ddof as the denominator and matches np.var on the replicated sample with integer weights. quantile is the inverse CDF, the smallest value where the cumulative weight share reaches q, which is what the docstring cites survey::svyquantile for. Both hold numerically.

The README is also the PyPI long description, so this shows up there at the next release. The fragment is .changed, so a patch bump. Approving.

@vahid-ahmadi
vahid-ahmadi merged commit b6a13d2 into main Sep 18, 2026
8 checks passed
@vahid-ahmadi
vahid-ahmadi deleted the docs/readme-statement-of-need branch September 18, 2026 11:12
@vahid-ahmadi vahid-ahmadi mentioned this pull request Sep 18, 2026
30 of 31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants