Repository navigation
Count households once: sum the weight column unweighted - #767
Draft
PavelMakarchuk wants to merge 1 commit into
Draft
PavelMakarchuk wants to merge 1 commit into
PavelMakarchuk wants to merge 1 commit into
Conversation
budget.households summed household_weight through the household MicroDataFrame, whose columns are already weighted by household_weight, so it returned the sum of squared weights: about 10^4 times the count on every run (state/ny 135.6bn). Count entities with a plain sum of the weight column, and test the budget on a real MicroDataFrame. Fixes #766 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #766
Summary
budget.householdswas the sum of squared household weights.build_budgetary_impactsummedhousehold_weightthrough_sum_output_variable, which callsdata[variable].sum(). Heredatais the householdMicroDataFrame, which is weighted byhousehold_weight, so.sum()on the weight column isΣ w × w. On the sparse US data that is about 10⁴ times the household count, on every run. For example,state/ny2026 reports 135,597,233,840 households.The fix:
_count_output_entities(simulation, entity, weight)counts entities with a plain sum of the weight column.build_budgetary_impactuses it forhouseholds.The other budget fields are not changed. They are meant to be weighted sums (
Σ w × value).Tests
tests/test_simulation_output_budget.pybuilds the budget from realMicroDataFrames, as policyengine does. The existing builder test uses plainpd.DataFrames, which is why it never caught this.mainwithassert 5.0 == 3.0, and it passes with the fix.baseline_net_income,tax_revenue_impactandstate_tax_revenue_impactstay weighted sums.uv run pytest tests/ --ignore=tests/integration: 733 passed, 23 skipped.ruff format --checkandruff checkare clean on the changed files.The test file is separate from
test_simulation_output_builder.pyso that it does not conflict with #765, which reorders that file's imports.Downstream
policyengine-app-v2 shows the average change per household as
-budget.budgetary_impact / budget.households, so that headline is about 10⁴ times too small today. Cached results keep the wrong count until they are recomputed.🤖 Generated with Claude Code