Skip to content

Ensure imputed capital gains CDF is valid (monotonic) #83

Closed
Closed
Ensure imputed capital gains CDF is valid (monotonic)#83

Description

@MaxGhenis
MaxGhenis
opened on Feb 20, 2024

impute_capital_gains currently interpolates/extrapolates the provided quantiles to a CDF by fitting splines. This can result in CDFs that are not monotonically increasing and thus invalid.

After asking ChatGPT for some ideas, I think a promising approach could be first synthesizing a pdf from the quantiles, smoothing it with a kernel density estimator, then integrating it to a cdf. Here's an example of how that might look:

image

Other options like isotonic regression or transformations could also work, and we may want something more complex if we want to consider all the data together rather than each income group independently.

Activity

  1. MaxGhenis commented on Feb 25, 2024

    @MaxGhenis
    MaxGhenis
    on Feb 25, 2024
    ContributorAuthor

    FWIW I only saw one income level where the spline was obviously nonmonotonic, so might not be such a high priority:
    image

    https://policyengine-uk-documentation.nw.r.appspot.com/Capital_Gains_Tax

  2. MaxGhenis commented on Feb 25, 2024

    @MaxGhenis
    MaxGhenis
    on Feb 25, 2024
    ContributorAuthor

    The PCHIP Interpolator seems ideally suited to this. It both preserves monotonicity and supports extrapolation.

    Here's an example for the 99th income centile, which the spline currently produces a nonmonotonic interpolation from.

    image

    Relevant code (notebook):

    pchip_interpolator = PchipInterpolator(quantiles, gains, extrapolate=True)
extended_quantiles = np.linspace(0.01, 0.99, 99)
extended_gains = pchip_interpolator(extended_quantiles)
  3. MaxGhenis
    transferred this issue fromPolicyEngine/policyengine-ukon Mar 4, 2025

  4. MaxGhenis commented on Jun 19, 2026

    @MaxGhenis
    MaxGhenis
    on Jun 19, 2026
    ContributorAuthor

    Closing as migrated/superseded. Populace should use QRF/support gates rather than the old spline-CDF path; see PolicyEngine/microcosm#148.

  5. MaxGhenis
    closed this as completedon Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions