In `stat_density()`, the calculated `count` aesthetic should take weights into account

[tuberculo]

The weight should sum to one, ignoring color or fill aesthetics. That was the behavior before as can be seen here: https://stackoverflow.com/questions/20342494/density-of-each-group-of-weighted-geom-density-sum-to-one
In the way that geom_density is now there is no way to make the sum of both areas be equal to one.

Example:

library(ggplot2)
library(ggplot2movies) 
m <- ggplot()
m + geom_density(data = movies, aes(rating, weight = votes, colour = factor(Action)))

The plot above is equal to

m + geom_density(data = movies[movies$Action == 0, ], aes(rating, weight = votes/sum(votes)), fill=NA, colour="black") +
    geom_density(data = movies[movies$Action == 1, ], aes(rating, weight = votes/sum(votes)), fill=NA, colour="blue")

I would expect a much smaller area for Action == 1 in the first plot, as less than 10% of the movies are action movies.

[thomasp85]

I fail to see from your link that this has ever been different (the accepted solution shows the exact same plot as your code)

That being said it would probably make sense to add a weighted normalisation to stat_density()

[tuberculo]

First, I think I was not so clear in my first message. Sorry for that.
As far as I can understand, geom_density behavior was different when stack overflow question was posted. Therefore, when using current ggplot version, the plot from both examples is the same. I believe the change was done in PR #3409. In my view, that change was not good, as it is really hard now to have the previous behavior. Besides, it makes it very different from geom_histogram.
I was able to reproduce the desired effect using the code below. However, this workaround is not really useful as it doesn't work if there are facets or if weight is not a boolean value.

m + geom_density(data = movies %>% group_by(Action), aes(rating, weight = votes, colour = factor(Action), 
                                                         y = after_stat(count * mean(density) / mean(count) / n_distinct(group))))

Issue #2900 is related to this one.

[clauswilke]

@tuberculo I don't understand what exactly you're asking for. Should the areas under the curves individually sum to one or jointly? It is unclear from your problem description. Please start from the beginning and describe exactly what the problem is, in a completely self-contained manner (without referring to some stackoverflow post). Please also provide code that generates the plot you'd want to see, e.g. by carrying out the appropriate normalization in dplyr. Finally, please provide the output from the code you're posting, preferably by using the reprex package.

[tuberculo]

What I want is to make the areas jointly sum to 1. Currently, geom_density forces the individual areas to be equal to 1:

library(tidyverse)
library(ggplot2movies) 

ggplot() + geom_density(data = movies, aes(rating, weight = votes, colour = factor(Action)))

I could not find a way to do what I want with ggplot. But, using base R it is like this:

plot(density(movies[movies$Action != 1,]$rating, weight = movies[movies$Action != 1,]$votes / sum(movies$votes)))
#> Warning in density.default(movies[movies$Action != 1, ]$rating, weight =
#> movies[movies$Action != : sum(weights) != 1 -- will not get true density
lines(density(movies[movies$Action == 1,]$rating, weight = movies[movies$Action == 1,]$votes / sum(movies$votes)))
#> Warning in density.default(movies[movies$Action == 1, ]$rating, weight =
#> movies[movies$Action == : sum(weights) != 1 -- will not get true density

Action movies receive approximately 26% of the votes, so the area below its curve is 0.26 and the area below the non-action movies curve is 0.74, jointly summing 1.

Using ggplot, the closest I could get to what I want is this. But I don't think it is right, as weight data is not available in after_stat.

ggplot() + geom_density(data = movies, aes(rating, weight = votes, colour = factor(Action), 
                                           y = after_stat(count * mean(density) / mean(count) / n_distinct(group))))

I hope it is clear now. Please, let me know if you need more info.

^{Created on 2021-02-15 by the reprex package (v1.0.0)}

[clauswilke]

Simpler reprex to highlight the issue. I think the count aesthetic should take weights into account. We should also add an ncount aesthetic that uses normalized counts (i.e., all areas under all curves together add to one).

library(tidyverse)

# weighted data
data <- tibble(
  x = c(rnorm(50, mean = -2), rnorm(50, mean = 2)),
  weight = rep(c(1, 10), each = 50),
  type = rep(c("A", "B"), each = 50)
)

# same without weights
data2 <- tibble(
  x = c(rnorm(50, mean = -2), rnorm(500, mean = 2)),
  type = c(rep("A", 50), rep("B", 500))
)

## *****************
## with weights

# joint density uses weights correctly
ggplot(data, aes(x, weight = weight)) +
  geom_density()

# separate densities are normalized individually (by design)
ggplot(data, aes(x, weight = weight, color = type)) +
  geom_density()

# `count` is supposed to show the relative size of
# the two densities but this doesn't seem to work here
ggplot(data, aes(x, weight = weight, color = type)) +
  geom_density(aes(y = after_stat(count)))

## *****************
## without weights

# joint density uses weights correctly
ggplot(data2, aes(x)) +
  geom_density()

# separate densities are normalized individually (by design)
ggplot(data2, aes(x, color = type)) +
  geom_density()

# `count` shows the relative size of the two densities
ggplot(data2, aes(x, color = type)) +
  geom_density(aes(y = after_stat(count)))

^{Created on 2021-02-15 by the reprex package (v1.0.0)}