02_dplyr.Rmd

---
title: "Touring tidyverse"
subtitle: "dplyr"
output:
  xaringan::moon_reader:
    lib_dir: libs
    nature:
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
---

```{r setup, include=FALSE}
options(htmltools.dir.version = FALSE)
```

background-image: url(https://www.rstudio.com/wp-content/uploads/2015/01/dplyr-hexbin-logo.png)
background-size: 100px
background-position: 90% 8%

# dplyr

has three main goals:

1. Identify the most important data manipulation verbs and make them easy to use from R.

2. Provide blazing fast performance for in-memory data by writing key pieces in C++ (using Rcpp).

3. Use the same interface to work with data no matter where it's stored, whether in a data frame, a data table or database.


---
# Initial commit of dplyr

The year was 2012... (`2012-10-28 19:37` to be precise)

```
Package: plyr2
Type: Package
Title: Tools for splitting, applying and combining data
Version: 0.01
Author: Hadley Wickham <h.wickham@gmail.com>
Maintainer: Hadley Wickham <h.wickham@gmail.com>
Description: ddply on crack
Depends:
    R (>= 2.15.1)
License: MIT
```

---
# 23 minutes later

```
Package: dplyr
Type: Package
Title: dplyr: a grammar of data manipulation
Version: 0.01
Author: Hadley Wickham <h.wickham@gmail.com>
Maintainer: Hadley Wickham <h.wickham@gmail.com>
Description: A fast, consistent tools for working with data frame like objects,
    both in memory and out of memory.
Imports:
    stringr
Depends:
    R (>= 2.15.1)
License: MIT
```

---
# Current state

1. `dplyr` replaced `plyr` to specialize on data frames.
2. Current version - `r packageVersion("dplyr")`.
3. https://github.com/tidyverse/dplyr
4. Developed by __Hadley Wickham__, Romain François, Lionel Henry, Kirill Müller.
5. 6000+ commits by 176 contributors.

---
background-image: url(https://www.rstudio.com/wp-content/uploads/2015/01/dplyr-hexbin-logo.png)
background-size: 100px
background-position: 90% 8%

# dplyr

5 verbs:

* `mutate()` adds new variables that are functions of existing variables
* `select()` picks variables based on their names.
* `filter()` picks cases based on their values.
* `summarise()` reduces multiple values down to a single summary.
* `arrange()` changes the ordering of the rows.

---
# Overview of the API
.pull-left[
1. Single-table verbs.
2. Single-table helpers.
3. Two-table verbs.
4. Remote tables.
5. Vector functions.

```{r dplyr_api, echo = FALSE}
funs <- getNamespaceExports("dplyr")
length(funs[!grepl(x = funs, pattern = "_$")])
```

]
--
.pull-right[

1. Selecting columns.
2. Transforming columns.
3. Filtering rows.
4. Summarizing and slicing.
5. `dbplyr`.

Based on https://github.com/suzanbaert/RTutorials
]

---
# Covering today

1. Breaking changes vs changing behaviour.
1. Working with names in `tidyverse`.

---
# Breaking changes

[Breaking changes](https://dbplyr.tidyverse.org/news/index.html#breaking-changes) in packages (`$`, `[[`, `[`).

```{r, message = FALSE, error = TRUE}
library("tidyverse", warn.conflicts = FALSE)
iris_db <- dbplyr::tbl_memdb(iris)

species <- list()
species$species <- c("setosa")

iris_db %>%
  dplyr::filter(Species %in% species$species)
```

---
# Breaking changes

These will work:

```{r, eval = FALSE}
iris_db %>%
  dplyr::filter(Species %in% !!species$species)

just_vector <- species$species
iris_db %>%
  dplyr::filter(Species %in% just_vector)

iris_db %>%
  filter(Species %in% local(species$species))
```

---
# Changing behaviour

```{r}
funx <- function(x) {
    print(substitute(x))
    mean(x) 
}

mtcars %>% 
    select(1:2) %>%
    summarise_all(list(~funx(.)))
```

---
# Changing behaviour

```{r}
# works
mtcars %>% select(1:2) %>% summarise_all(list(funx))
mtcars %>% select(1:2) %>% summarise_all(funx)
```

```{r, eval = FALSE}
# doesn't
mtcars %>% select(1:2) %>% summarise_all(~funx(.))
mtcars %>% select(1:2) %>% summarise_all(list(~funx(.)))
```

---
# Working with names in `tidyverse`<sup>1</sup>

One thing is not like the other:

```{r, eval = FALSE}
starwars %>% mutate(birth_year - 100)
starwars %>% group_by(birth_year)
starwars %>% select(birth_year)
starwars %>% filter(birth_year < 50)
```

.footnote[
[1] Full version - https://resources.rstudio.com/rstudio-conf-2019/working-with-names-and-expressions-in-your-tidy-eval-code
]

---

Selection is special since it understands `c`, `-`, and `:` ...

```{r, eval = FALSE}
starwars %>% select(c(1, height))
```

--
and helpers know about variables:

```{r, eval = FALSE}
starwars %>% select(ends_with("color"))
```


---

However, consider following:

.pull-left[
```{r}
starwars %>% select(height)
```
]
.pull-right[
```{r}
starwars %>% transmute(height)
```
]

---

.pull-left[
```{r}
starwars %>% select(1)
```
]
.pull-right[
```{r}
starwars %>% transmute(1)
```
]

---
# Passing selections

```{r}
starwars %>%
  group_by_at(vars(ends_with("color")))
```

---
# Writing functions with selections

```{r}
custom_summarize <- function(.data, ...){
  .data %>%
    summarize_at(vars(...), ~ mean(., na.rm = TRUE))
}

custom_summarize(starwars, height, mass)
custom_summarize(starwars, starts_with("height"))

```

---
# Works with groups

```{r}
starwars %>%
  group_by(hair_color) %>%
  custom_summarize(mass)
```

---
# What if we want to pass actions instead?

```{r}
summary_functions <- list(
  ~ mean(., na.rm = TRUE),
  ~ sd(., na.rm = TRUE)
)

summarize_acts <- function(.data, ...){
  .data %>%
    transmute(...) %>%
    summarize_all(summary_functions)
}

summarize_acts(starwars, heightm = height / 100, bmi = mass / heightm ^ 2)
```

---
# Works with groups as well

```{r}
starwars %>%
  group_by(gender) %>%
  summarize_acts(heightm = height / 100, bmi = mass / heightm ^ 2)
```

---
# Adding `tidyr`

```{r}
gather_summarize_acts <- function(.data, ...){
  .data %>%
    transmute(...) %>%
    gather("Variable", "Value", everything()) %>%
    group_by(Variable) %>%
    summarize_at(vars("Value"), summary_functions)
}

starwars %>%
  gather_summarize_acts(
    heightm = height / 100, bmi = mass / heightm ^ 2
  )
```

---
# Groups don't work

```{r}
starwars %>%
  group_by(gender) %>%
  gather_summarize_acts(
    heightm = height / 100, 
    bmi = mass / heightm ^ 2
  )
```

---
# Use selectors

Lionel's version

```{r, eval = FALSE}
gather_summarize_acts <- function(.data, ...){
  .data %>%
    transmute(...) %>%
    gather("Variable", "Value", -one_of(group_vars(.))) %>%
    group_by(Variable) %>%
    summarize_at(vars("Value"), summary_functions)
}
```

---
# Correct version

```{r}
gather_summarize_acts <- function(.data, ...){
    transmute(.data, ...) %>%
    gather("Variable", "Value", -one_of(group_vars(.))) %>%
    group_by_at(vars(c(group_vars(.), Variable))) %>%
    summarize_at(vars("Value"), summary_functions)
}

starwars %>% group_by(gender) %>%
  gather_summarize_acts( heightm = height / 100, bmi = mass / heightm ^ 2)
```

---
```{r, echo = FALSE}
countdown::countdown_fullscreen(minutes = 4, update_every = 30,
                     padding = "20%", margin = "5%",
                     font_size = "8em", line_height = "1.25")
````
  
---
# Resources

1. https://dplyr.tidyverse.org/index.html
2. http://r4ds.had.co.nz/transform.html
3. https://suzan.rbind.io/2018/01/dplyr-tutorial-1/
4. https://dplyr.tidyverse.org/reference/index.html
5. https://github.com/rstudio/cheatsheets/blob/master/data-transformation.pdf
6. https://www.nielsenmark.us/2018/07/07/connecting-r-to-postgresql-on-linux/
7. http://db.rstudio.com/