Transform data with dplyr

# Transform data<br>with dplyr

.class-info[
<figure>
  <img src="img/05/dplyr-min.png" alt="dplyr" title="dplyr" width="15%">
</figure>
]

---

# Gapminder health and wealth

.box-2.medium[Watch this video]

.box-2.medium[andhs.co/rosling]

.box-2[Make a list of how those variables are shown in the graph<br>(think about the graph's aesthetics and geometries)]

---

???

Source: [New York Times](https://www.nytimes.com/2017/02/09/world/europe/hans-rosling-dead-statistician.html)

---

# Your turn #0: Load data

1. Run the setup chunk
2. Take a look at the `gapminder` data

---

```r
gapminder
```

```
## # A tibble: 1,704 × 6
##    country     continent  year lifeExp      pop gdpPercap
##    <fct>       <fct>     <int>   <dbl>    <int>     <dbl>
##  1 Afghanistan Asia       1952    28.8  8425333      779.
##  2 Afghanistan Asia       1957    30.3  9240934      821.
##  3 Afghanistan Asia       1962    32.0 10267083      853.
##  4 Afghanistan Asia       1967    34.0 11537966      836.
##  5 Afghanistan Asia       1972    36.1 13079460      740.
##  6 Afghanistan Asia       1977    38.4 14880372      786.
##  7 Afghanistan Asia       1982    39.9 12881816      978.
##  8 Afghanistan Asia       1987    40.8 13867957      852.
##  9 Afghanistan Asia       1992    41.7 16317921      649.
## 10 Afghanistan Asia       1997    41.8 22227415      635.
## # ℹ 1,694 more rows
```
]

---

# The tidyverse

???

From "Master the Tidyverse" by RStudio

---

# The tidyverse

.center[
<figure>
  <img src="img/03/tidyverse-min.png" alt="The tidyverse" title="The tidyverse" width="50%">
</figure>
]

---

# dplyr: verbs for manipulating data

<table>
  <tr>
    <td>Extract rows with <code>filter()</code></td>
    <td><img src="img/05/filter-min.png" alt="filter" title="filter" height="80px"></td>
  </tr>
  <tr>
    <td>Extract columns with <code>select()</code></td>
    <td><img src="img/05/select-min.png" alt="select" title="select" height="80px"></td>
  </tr>
  <tr>
    <td>Arrange/sort rows with <code>arrange()</code></td>
    <td><img src="img/05/arrange-min.png" alt="arrange" title="arrange" height="80px"></td>
  </tr>
  <tr>
    <td>Make new columns with <code>mutate()</code></td>
    <td><img src="img/05/mutate-min.png" alt="mutate" title="mutate" height="80px"></td>
  </tr>
  <tr>
    <td>Make group summaries with<br><code>group_by() |> summarize()</code></td>
    <td><img src="img/05/summarize-min.png" alt="summarize" title="summarize" height="80px"></td>
  </tr>
</table>

---

# `filter()`

---

# `filter()`

<code class ='r hljs remark-code'>filter(.data = <b><span style="background-color:#FFDFD1">DATA</span></b>, <b><span style="background-color:#FFD0CF">...</span></b>)</code>
]

.pull-right[
- <b><span style="background: #FFDFD1">`DATA`</span></b> = Data frame to transform
- <b><span style="background: #FFD0CF">`...`</span></b> = One or more tests <br>.small[`filter()` returns each row for which the test is TRUE]
]

---

<code class ='r hljs remark-code'>filter(.data = <b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">country == "Denmark"</span></b>)</code>

.pull-left[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> country </th>
   <th style="text-align:left;"> continent </th>
   <th style="text-align:left;"> year </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> Asia </td>
   <td style="text-align:left;"> 1952 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> Asia </td>
   <td style="text-align:left;"> 1957 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> Asia </td>
   <td style="text-align:left;"> 1962 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> Asia </td>
   <td style="text-align:left;"> 1967 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> Asia </td>
   <td style="text-align:left;"> 1972 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> … </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:left;"> … </td>
  </tr>
</tbody>
</table>
]

.pull-right[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> country </th>
   <th style="text-align:left;"> continent </th>
   <th style="text-align:right;"> year </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1952 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1957 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1962 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1967 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1972 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Denmark </td>
   <td style="text-align:left;"> Europe </td>
   <td style="text-align:right;"> 1977 </td>
  </tr>
</tbody>
</table>
]

---

# `filter()`

<code class ='r hljs remark-code'>filter(<br>&nbsp;&nbsp;.data = <b><span style="background-color:#FFDFD1">gapminder</span></b>, <br>&nbsp;&nbsp;<b><span style="background-color:#FFD0CF">country == "Denmark"</span></b><br>)</code>
]

---

# Logical tests

<table>
  <tr>
    <th class="cell-center">Test</th>
    <th class="cell-left">Meaning</th>
    <th class="cell-center">Test</th>
    <th class="cell-left">Meaning</th>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">x < y</code></td>
    <td class="cell-left">Less than</td>
    <td class="cell-center"><code class="remark-inline-code">x %in% y</code></td>
    <td class="cell-left">In (group membership)</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">x > y</code></td>
    <td class="cell-left">Greater than</td>
    <td class="cell-center"><code class="remark-inline-code">is.na(x)</code></td>
    <td class="cell-left">Is missing</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">==</code></td>
    <td class="cell-left">Equal to</td>
    <td class="cell-center"><code class="remark-inline-code">!is.na(x)</code></td>
    <td class="cell-left">Is not missing</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">x <= y</code></td>
    <td class="cell-left">Less than or equal to</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">x >= y</code></td>
    <td class="cell-left">Greater than or equal to</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">x != y</code></td>
    <td class="cell-left">Not equal to</td>
  </tr>
</table>

---

# Your turn #1: Filtering

1. The data for Canada
2. All data for countries in Oceania
3. Rows where the life expectancy is greater than 82

---

```r
filter(gapminder, country == "Canada")
```
]

```r
filter(gapminder, continent == "Oceania")
```
]

```r
filter(gapminder, lifeExp > 82)
```
]

---

# Common mistakes

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country <b><span style="color:#FF4136">=</span></b> "Canada")</code>

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country <b><span style="color:#2ECC40">==</span></b> "Canada")</code>
]

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country == <b><span style="color:#FF4136">Canada</span></b>)</code>

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country == <b><span style="color:#2ECC40">"Canada"</span></b>)</code>
]

---

# `filter()` with multiple conditions

<code class ='r hljs remark-code'>filter(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">country == "Denmark", year > 2000</span></b>)</code>

---

<code class ='r hljs remark-code'>filter(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">country == "Denmark", year > 2000</span></b>)</code>

---

# Boolean operators

<table>
  <tr>
    <th class="cell-center">Operator</th>
    <th class="cell-center">Meaning</th>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">a & b</code></td>
    <td class="cell-center">and</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">a | b</code></td>
    <td class="cell-center">or</td>
  </tr>
  <tr>
    <td class="cell-center"><code class="remark-inline-code">!a</code></td>
    <td class="cell-center">not</td>
  </tr>
</table>

---

# Default is "and"

<code class ='r hljs remark-code'>filter(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">country == "Denmark", year > 2000</span></b>)</code>

<code class ='r hljs remark-code'>filter(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">country == "Denmark" & year > 2000</span></b>)</code>

---

# Your turn #2: Filtering

1. Canada before 1970
2. Countries where life expectancy in 2007 is below 50
3. Countries where life expectancy in 2007 is below 50 and are not in Africa

---

```r
filter(gapminder, country == "Canada", year < 1970)
```

```r
filter(gapminder, year == 2007, lifeExp < 50)
```

```r
filter(gapminder, year == 2007, lifeExp < 50, 
       continent != "Africa")
```

---

# Common mistakes

<code class ='r hljs remark-code'>filter(gapminder, <b><span style="color:#FF4136">1960 &lt; year &lt; 1980</span></b>)</code>
]

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="color:#2ECC40">year > 1960, year &lt; 1980</span></b>)</code>
]
]

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="color:#FF4136">country == "Mexico", <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country == "Canada", <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;country == "United States"</span></b>)</code>
]

<code class ='r hljs remark-code'>filter(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="color:#2ECC40">country %in% c("Mexico", "Canada", <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"United States")</span></b>)</code>
]
]

---

# Common syntax

<code class ='r hljs remark-code'><b><span style="background-color:#EFB3FF">VERB</span></b>(<b><span style="background-color:#FFDFD1">DATA</span></b>, <b><span style="background-color:#FFD0CF">...</span></b>)</code>
]

.pull-right[
- <b><span style="background: #EFB3FF">`VERB`</span></b> = dplyr function/verb
- <b><span style="background: #FFDFD1">`DATA`</span></b> = Data frame to transform
- <b><span style="background: #FFD0CF">`...`</span></b> = Stuff the verb does
]

---

# `mutate()`

<code class ='r hljs remark-code'>mutate(<b><span style="background-color:#FFDFD1">.data</span></b>, <b><span style="background-color:#FFD0CF">...</span></b>)</code>
]

.pull-right[
- <b><span style="background: #FFDFD1">`DATA`</span></b> = Data frame to transform
- <b><span style="background: #FFD0CF">`...`</span></b> = Columns to make
]

---

<code class ='r hljs remark-code'>mutate(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">gdp = gdpPercap * pop</span></b>)</code>

.pull-left.small[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> country </th>
   <th style="text-align:left;"> year </th>
   <th style="text-align:left;"> gdpPercap </th>
   <th style="text-align:left;"> pop </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> 1952 </td>
   <td style="text-align:left;"> 779.4453145 </td>
   <td style="text-align:left;"> 8425333 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> 1957 </td>
   <td style="text-align:left;"> 820.8530296 </td>
   <td style="text-align:left;"> 9240934 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> 1962 </td>
   <td style="text-align:left;"> 853.10071 </td>
   <td style="text-align:left;"> 10267083 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> 1967 </td>
   <td style="text-align:left;"> 836.1971382 </td>
   <td style="text-align:left;"> 11537966 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:left;"> 1972 </td>
   <td style="text-align:left;"> 739.9811058 </td>
   <td style="text-align:left;"> 13079460 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> … </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:left;"> … </td>
  </tr>
</tbody>
</table>
]

.pull-right.small[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> country </th>
   <th style="text-align:right;"> year </th>
   <th style="text-align:left;"> … </th>
   <th style="text-align:right;"> gdp </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1952 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 6567086330 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1957 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 7585448670 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1962 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 8758855797 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1967 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 9648014150 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1972 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 9678553274 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1977 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 11697659231 </td>
  </tr>
</tbody>
</table>
]

---

<code class ='r hljs remark-code'>mutate(<b><span style="background-color:#FFDFD1">gapminder</span></b>, <b><span style="background-color:#FFD0CF">gdp = gdpPercap * pop,</span></b><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="background-color:#FFD0CF">pop_mil = round(pop / 1000000)</span></b>)</code>

.pull-right.small[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> country </th>
   <th style="text-align:right;"> year </th>
   <th style="text-align:left;"> … </th>
   <th style="text-align:right;"> gdp </th>
   <th style="text-align:right;"> pop_mil </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1952 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 6567086330 </td>
   <td style="text-align:right;"> 8 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1957 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 7585448670 </td>
   <td style="text-align:right;"> 9 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1962 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 8758855797 </td>
   <td style="text-align:right;"> 10 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1967 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 9648014150 </td>
   <td style="text-align:right;"> 12 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1972 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 9678553274 </td>
   <td style="text-align:right;"> 13 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Afghanistan </td>
   <td style="text-align:right;"> 1977 </td>
   <td style="text-align:left;"> … </td>
   <td style="text-align:right;"> 11697659231 </td>
   <td style="text-align:right;"> 15 </td>
  </tr>
</tbody>
</table>
]

---

# `ifelse()`

<code class ='r hljs remark-code'>ifelse(<b><span style="background-color:#FFC0DC">TEST</span></b>, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="background-color:#FFDFD1">VALUE_IF_TRUE</span></b>, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="background-color:#CBB5FF">VALUE_IF_FALSE</span></b>)</code>
]

.pull-right[
- <b><span style="background: #FFC0DC">`TEST`</span></b> = A logical test
- <b><span style="background: #FFDFD1">`VALUE_IF_TRUE`</span></b> = What happens if test is true
- <b><span style="background: #CBB5FF">`VALUE_IF_FALSE`</span></b> = What happens if test is false
]

---

<code class ='r hljs remark-code'>mutate(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;after_1960 = ifelse(<b><span style="background-color:#FFC0DC">year > 1960</span></b>, <b><span style="background-color:#FFDFD1">TRUE</span></b>, <b><span style="background-color:#CBB5FF">FALSE</span></b>))</code>

<code class ='r hljs remark-code'>mutate(gapminder, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;after_1960 = ifelse(<b><span style="background-color:#FFC0DC">year > 1960</span></b>, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="background-color:#FFDFD1">"After 1960"</span></b>, <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b><span style="background-color:#CBB5FF">"Before 1960"</span></b>))</code>

---

# Your turn #3: Mutating

1. Add an `africa` column that is TRUE if the country is on the African continent
2. Add a column for logged GDP per capita (hint: use `log()`)
3. Add an `africa_asia` column that says “Africa or Asia” if the country is in Africa or Asia, and “Not Africa or Asia” if it’s not

---

```r
mutate(gapminder, africa = ifelse(continent == "Africa", 
                                  TRUE, FALSE))
```

```r
mutate(gapminder, log_gdpPercap = log(gdpPercap))
```

```r
mutate(gapminder, 
       africa_asia = 
         ifelse(continent %in% c("Africa", "Asia"), 
                "Africa or Asia", 
                "Not Africa or Asia"))
```

---