geom_histogram: wrong bins?

[collioud]

Hello,

I am using ggplot 2.1.0 to plot histograms, and I have an unexpected behaviour concerning the bins.
I put here an example with left-closed bins (i.e. [ 0, 0.1[ ) with a binwidth of 0.1.

mydf <- data.frame(myvar=c(-1,-0.5,-0.4,-0.1,-0.1,0.05,0.1,0.1,0.25,0.5,1))
myplot <- ggplot(mydf, aes(myvar)) + geom_histogram(aes(y=..count..),binwidth = 0.1, boundary=0.1,closed="left")
myplot
ggplot_build(myplot)$data[[1]]

On this example, one may expect the value -0.4 to be within the bin [-0.4, -0.3[, but it falls instead (mysteriously) in the bin [-0.5,-0.4[. Same thing for the value -0.1 which falls in [-0.2,-0.1[ instead of [-0.1,0[...etc.

Is there something here I do not fully understand (especially with the new "center" and "boundary" params)? Or is ggplot2 doing weird things there?

Thanks in advance,
Best regards,
Arnaud

PS: Also asked here: http://stackoverflow.com/questions/37876096/geom-histogram-wrong-bins

[ecoRoland]

I suspect there is not much that can be done. This is probably an example of R-FAQ 7.31.

[collioud]

That would be...unfortunate.
However I do not have the same result with ggplot 2.1.0 (http://i66.tinypic.com/1sy2w4.jpg) and ggplot 0.9.3 (http://i65.tinypic.com/4rsvev.jpg), which is very disturbing.
By looking at both graphs, we can see that ggplot 0.9.3 behaves as expected.

[collioud]

Well...I suppose I have to stay on ggplot version 0.9.3.

[ptoche]

It looks like a piece of code originally borrowed from base::hist, intended to deal with rounding errors, was removed in recent versions of ggplot2 (look for diddle). Not sure if that is the cause (Edit looks like the diddle and fuzz are still there under slightly different names, so issue appears to be elsewhere) (Edit 2 I believe the fuzzy parameters are correctly computed and saved in bins$fuzzy, but now I wonder if they are actually used).

Edit Setting the boundary to a different value seems to help. May require tweaking. See my stackoverflow answer. I have made further comments over at stackoverflow, where it is more convenient to post images, even if the discussion probably belonged here to begin with ...

[collioud]

Thanks!
But by looking into the file bin.R, I guess the param fuzz took the place of diddle. Could you please confirm this?

[collioud]

The boundary param solves indeed this problem if set slightly smaller than the binwidth. But this is not a 'clean' nor logical solution...

[ptoche]

I've tried to debug the problem, but I'm not familiar at all with ggplot2 (I'm just an enthusiastic user). I have copy-pasted my preliminary investigations here: http://pastebin.com/cYbczwqp.

To sum up, I have attempted to track the fuzzy parameters via all the functions they are processed by. So far, they appear to be correctly computed and saved under bins$fuzzy. But I don't see where/if they are actually used. Consider the function bin_vector, copied below. If you replace bins$breaks with bins$fuzzy, at the location indicated below, you get the expected plot. So my current favoured hypothesis is that bin_vector is supposed to condition on something and return bins$breaks sometimes and bins$fuzzy at other times. See below the location I indicate. Expert opinion will be appreciated at this point.

    ggplot2:::bin_vector
function (x, bins, weight = NULL, pad = FALSE) 
{
    stopifnot(is_bins(bins))
    if (all(is.na(x))) {
        return(bin_out(length(x), NA, NA, xmin = NA, xmax = NA))
    }
    if (is.null(weight)) {
        weight <- rep(1, length(x))
    }
    else {
        weight[is.na(weight)] <- 0
    }

    bin_idx <- cut(x, bins$breaks, right = bins$right_closed, 
        include.lowest = TRUE) ## REPLACE bins$breaks with bins$fuzzy 
                                               ## TO GET EXPECTED OUTPUT
                                               ## CONDITIONAL MISSING HERE?

    bin_count <- as.numeric(tapply(weight, bin_idx, sum, na.rm = TRUE))
    bin_count[is.na(bin_count)] <- 0
    bin_x <- (bins$breaks[-length(bins$breaks)] + bins$breaks[-1])/2
    bin_widths <- diff(bins$breaks)
    if (pad) {
        bin_count <- c(0, bin_count, 0)
        width1 <- bin_widths[1]
        widthn <- bin_widths[length(bin_widths)]
        bin_widths <- c(width1, bin_widths, widthn)
        bin_x <- c(bin_x[1] - width1, bin_x, bin_x[length(bin_x)] + 
            widthn)
    }
    if (any(is.na(bins))) {
        bin_count <- c(bin_count, sum(is.na(bins)))
        bin_widths <- c(bin_widths, NA)
        bin_x <- c(bin_x, NA)
    }
    bin_out(bin_count, bin_x, bin_widths)
}
<environment: namespace:ggplot2>   

[ptoche]

This is what I think is going on: in the older versions of ggplot2, the fuzzy breaks were assigned to breaks no questions asked:

fuzzybreaks <- sort(breaks) + fuzz

This is line 123 in the old bin function located in stat-bin.r

In the current implementation, the bins structure holds both the non-fuzzy breaks, named bins$breaks as well as the fuzzy breaks, named bins$fuzzy. However, correct me if I'm wrong it appears that nothing is done of bins$fuzzy and only the non-fuzzy bins$breaks are passed to the bin_vector function and ultimately to geom_histogram. I expected to find a condition upon which to set either bins$breaks or bins$fuzzy somewhere at the top of bin_vector

[collioud]

Indeed, bin$fuzzy does not seem to be used anywhere.
I hope Hadley (or anyone competent) will enlighten us on this topic...

[hadley]

More minimal reprex:

mydf <- data.frame(x = c(-1, -0.5, -0.4, -0.1, -0.1, 0.05, 0.1, 0.1, 0.25, 0.5, 1))
p <- ggplot(mydf, aes(x)) + 
  geom_histogram(binwidth = 0.1, boundary = 0.1, closed = "left")
layer_data(p) %>% subset(count > 0) %>% .[1:5]

Looks like a FP bug. @collioud would you mind trying to make the example even more minimal? (i.e. progressively remove the points in x until the problem no longer appears). That'll make it easier to find and fix

[ptoche]

@hadley , have you seen this ?

http://stackoverflow.com/questions/37876096/geom-histogram-wrong-bins

[hadley]

No, thanks - want to do a PR?

[ptoche]

do you remember when you intended to use bins$breaks and when you intended to use bins$fuzzy?

Apart from having located the line where bins$fuzzy might have been intended, it's not much of a PR as is, I think (never done a PR before)...