Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,28 @@
# microdf
Weighted pandas DataFrames and Series for survey microdata analysis.

## Overview
microdf provides `MicroDataFrame` and `MicroSeries` classes that extend pandas functionality with integrated weighting support, essential for accurate survey data analysis.
## Why this exists

Survey microdata comes with weights, and analysing it in pandas means getting two
things right that pandas will not do for you.

**The estimators are not the obvious ones.** A weighted median is not the median of
weighted values. Weighted variance requires choosing between treating weights as
frequencies or as precision. A top-1% share requires deciding what happens to a
record that straddles the cutoff. Each of these is a decision, and hand-rolling it
per analysis means making it differently each time. `microdf` makes each choice
once, documents it, and tests it — quantiles follow the inverse CDF so they can be
checked against R's `survey::svyquantile`, and variance treats weights as
frequencies so integer weights agree with `numpy` on the replicated sample.

**Weights have to survive the pipeline.** Before any estimator runs, weights must
stay aligned with their rows through merges, filters, grouping and reindexing. When
they do not, nothing raises. The pipeline completes and returns a plausible wrong
number. This is the harder of the two problems, and it is why `microdf` carries
weights inside the object rather than beside it.

If you are computing a poverty rate or a Gini on weighted survey data, those are
the two ways to get a believable-looking wrong answer.

## Key Features
- **MicroDataFrame**: A pandas DataFrame with an integrated weight column
Expand Down
1 change: 1 addition & 0 deletions changelog.d/readme-statement-of-need.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The README now opens with the problem the package solves rather than a description of its classes.
Loading