---
title: "tidyr"
subtitle: "Lecture 07"
author: "Dr. Colin Rundel"
footer: "Sta 323 - Spring 2025"
format:
live-revealjs:
theme: slides.scss
transition: fade
slide-number: true
self-contained: true
revealjs-plugins:
- drop
execute:
echo: true
warning: true
engine: knitr
webr:
packages:
- tidyverse
- palmerpenguins
---
{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}
```{r setup}
#| message: False
#| warning: False
#| include: False
options(
width=80
)
library(tidyverse)
```
#
{fig-align="center" width="50%"}
# Reshaping data
## Wide vs Long
::: {.center}
{fig-align="center" width="50%"}
:::
::: {.aside}
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
:::
## Wide -> Long
{fig-align="center" width="60%"}
::: {.center}
`pivot_longer` (previously `gather`)
:::
::: {.aside}
From [Data tidying with tidyr](https://raw.githubusercontent.com/rstudio/cheatsheets/main/tidyr.pdf)
:::
## Syntax
::: {.small}
```{r}
#| output-location: column
(d = tibble::tribble(
~country, ~"1999", ~"2000",
"A", "0.7K", "2K",
"B", "37K", "80K",
"C", "212K", "213K"
))
```
:::
. . .
::: {.small}
```{r}
#| code-line-numbers: "|3|4|5"
#| output-location: column
pivot_longer(
d,
cols = "1999":"2000",
names_to = "year",
values_to = "cases"
)
```
:::
## Long -> Wide
{fig-align="center" width="60%"}
::: {.center}
`pivot_wider` (previously `spread`)
:::
::: {.aside}
From [Data tidying with tidyr](https://raw.githubusercontent.com/rstudio/cheatsheets/main/tidyr.pdf)
:::
## Syntax
:::: {.columns .xsmall}
::: {.column width='50%'}
```{r}
#| output: false
( d = tibble::tribble(
~country, ~year, ~type, ~count,
"A", 1999, "cases", "0.7K",
"A", 1999, "pop", "19M",
"A", 2000, "cases", "2K",
"A", 2000, "pop", "20M",
"B", 1999, "cases", "37K",
"B", 1999, "pop", "172M",
"B", 2000, "cases", " 80K",
"B", 2000, "pop", "174M",
"C", 1999, "cases", "212K",
"C", 1999, "pop", "1T",
"C", 2000, "cases", "213K",
"C", 2000, "pop", "1T"
)
)
```
:::
::: {.column width='50%'}
```{r}
#| echo: false
d
```
:::
::::
. . .
:::: {.columns .xsmall}
::: {.column width='50%'}
```{r wider}
#| code-line-numbers: "|3|4|5"
#| eval: false
pivot_wider(
d,
id_cols = country:year,
names_from = type,
values_from = count
)
```
:::
::: {.column width='50%'}
```{r ref.label="wider"}
#| echo: false
```
:::
::::
## Exercise 1
::: {.small}
The `palmerpenguin` package contains measurement data on various penguin species on islands near Palmer Station in Antarctica. The code below shows the # of each species measured on each of the three islands (missing island, penguin pairs implies that species does not occur on that island).
```{r}
palmerpenguins::penguins |>
count(island, species)
```
Starting from these data construct a contingency table of counts for island (rows) by species (columns) using the pivot functions we've just discussed.
:::
```{r}
#| echo: false
countdown::countdown(5)
```
## Separate - wider
```{r}
#| echo: false
d = tibble::tribble(
~country, ~year, ~rate,
"A", 1999, "0.7K/19M",
"A", 2000, "2K/20M",
"B", 1999, "37K/172M",
"B", 2000, "80K/174M",
"C", 1999, "212K/1T",
"C", 2000, "213K/1T"
)
```
{fig-align="center" width="70%"}
. . .
::: {.small}
```{r}
separate_wider_delim(d, rate, delim = "/", names = c("cases", "pop"))
```
:::
::: {.aside}
From [Data tidying with tidyr](https://raw.githubusercontent.com/rstudio/cheatsheets/main/tidyr.pdf)
:::
## Separate - longer
:::: {.columns}
::: {.column width='50%'}
{fig-align="center" width="100%"}
:::
::: {.column width='50%' .fragment .small}
```{r}
separate_longer_delim(d, rate, delim = "/")
```
:::
::::
::: {.aside}
From [Data tidying with tidyr](https://raw.githubusercontent.com/rstudio/cheatsheets/main/tidyr.pdf)
:::
## Other separates
In previous versions of tidyr there was a single catch-all `separate()` function. This still exists and is available in the package but it is [**superseded**](https://lifecycle.r-lib.org/articles/stages.html).
Other helpful separate functions:
* `separate_longer_position()`
* `separate_wider_position()`
* `separate_wider_regex()`
## Unite
{fig-align="center" width="60%"}
. . .
```{r}
#| echo: false
d = tibble::tribble(
~country, ~century, ~year,
"Afghan", "19", "99",
"Afghan", "20", "00",
"Brazil", "19", "99",
"Brazil", "20", "00",
"China", "19", "99",
"China", "20", "00"
)
```
::: {.small}
```{r}
unite(d, century, year, col = "year", sep = "")
```
:::
::: {.aside}
From [Data tidying with tidyr](https://raw.githubusercontent.com/rstudio/cheatsheets/main/tidyr.pdf)
:::
## Example 1 - tidy grades
Is the following data tidy?
```{r}
grades = tibble::tribble(
~name, ~hw_1, ~hw_2, ~hw_3, ~hw_4, ~proj_1, ~proj_2,
"Alice", 19, 19, 18, 20, 89, 95,
"Bob", 18, 20, 18, 16, 77, 88,
"Carol", 18, 20, 18, 17, 96, 99,
"Dave", 19, 19, 18, 19, 86, 82
)
```
. . .
How would we calculate a final score based on the following formula,
$$\text{score} = 0.5\,\frac{\sum_i\text{hw}_i}{80} + 0.5\,\frac{\sum_j\text{proj}_j}{200}$$
## Semi-tidy approach
::: {.small}
```{r}
grades |>
mutate(
hw_avg = (hw_1+hw_2+hw_3+hw_4)/4,
proj_avg = (proj_1+proj_2)/2
) |>
mutate(
overall = 0.5*(proj_avg/100) + 0.5*(hw_avg/20)
)
```
:::
## pivot_longer (Wide -> Long)
::: {.medium}
```{r}
#| output-location: column
tidyr::pivot_longer(
grades,
cols = hw_1:proj_2,
names_to = "assignment",
values_to = "score"
)
```
:::
## Split type and id
::: {.small}
```{r}
#| output-location: column
tidyr::pivot_longer(
grades,
cols = hw_1:proj_2,
names_to = c("type", "id"),
names_sep = "_",
values_to = "score"
)
```
:::
## Tidy approach?
::: {.small}
```{r}
#| output-location: column
grades |>
tidyr::pivot_longer(
cols = hw_1:proj_2,
names_to = c("type", "id"),
names_sep = "_",
values_to = "score"
) |>
summarize(
total = sum(score),
.by = c(name, type)
)
```
:::
## pivot_wider - (Long -> Wide)
::: {.small}
```{r}
#| output-location: column
grades |>
tidyr::pivot_longer(
cols = hw_1:proj_2,
names_to = c("type", "id"),
names_sep = "_",
values_to = "score"
) |>
summarize(
total = sum(score),
.by = c(name, type)
) |>
tidyr::pivot_wider(
names_from = type,
values_from = total
)
```
:::
## Wrapping up
::: {.small}
```{r}
#| output-location: column
grades |>
tidyr::pivot_longer(
cols = hw_1:proj_2,
names_to = c("type", "id"),
names_sep = "_",
values_to = "score"
) |>
summarize(
total = sum(score),
.by = c(name, type)
) |>
tidyr::pivot_wider(
names_from = type,
values_from = total
) |>
mutate(
score = 0.5*(hw/80) +
0.5*(proj/200)
)
```
:::
# Rectangling
## Star Wars & repurrrsive
`repurrrsive` is a package that contains a number of interesting example data sets that are stored in a hierarchical format. Many come from web-based APIs which provide results as JSON.
::: {.medium}
```{r}
str(repurrrsive::sw_people)
```
:::
## RStudio data viewer
```r
View(repurrrsive::sw_people)
```
{fig-align="center" width="60%"}
## Tidy data from nested lists
In addition to `pivot_*` the `tidyr` package also has a number of functions that are designed to aid in the tidying of hierarchical / nested data.
For today we will be discussing the `unnest_longer()`, and `unnest_wider()` functions and next week we will see `hoist()`.
Much like the functions we saw last time in `dplyr`, these functions are designed to work with data frames (which may seem odd at first).
## List columns
We can make `sw_people` into a data frame by treating the original list as a single column in a data frame.
:::: {.columns .small}
::: {.column width='50%'}
```{r}
(sw_df = tibble::tibble(
people = repurrrsive::sw_people
))
```
:::
::: {.column width='50%' .fragment}
```{r}
as.data.frame(sw_df) |> head()
```
:::
::::
## Unnesting
::: {.small}
```{r}
sw_df |>
unnest_wider(people)
```
:::
## Unnesting - column types
::: {.small}
```{r}
sw_df |>
unnest_wider(people) |>
pull(height)
```
:::
## More list columns
:::: {.columns .medium}
::: {.column width='50%'}
```{r}
sw_df |>
unnest_wider(people) |>
select(name, starships)
```
:::
::: {.column width='50%' .fragment}
```{r}
sw_df |>
unnest_wider(people) |>
select(name, starships) |>
pull(starships) |>
str()
```
:::
::::
## Unnest Longer
::: {.medium}
```{r}
unnest_wider(sw_df, people) |>
select(name, starships) |>
unnest_longer(starships)
```
:::
## `sw_starships`
::: {.medium}
```{r}
(ships = tibble(ships = repurrrsive::sw_starships) |>
unnest_wider(ships) |>
select(ship = name, url)
)
```
:::
## General advice
* If there is a consistent set of entries (usually named) in the list column, use `unnest_wider()`
* If there are different numbers of entries (often unnamed) in the list column, use `unnest_longer()`
* Never use just `unnest()` - it can be inconsistent depending on input data
* Think about if you need all the data or not - `unnest_*()` are not always the best choice (more on the `hoist()` alternative next time)
# Joins
{fig-align="center" width="50%"}
## Joins (left)
{fig-align="center" width="50%"}
::: {.aside}
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
:::
## Joins (right)
{fig-align="center" width="50%"}
::: {.aside}
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
:::
## Joins (full / outer)
{fig-align="center" width="50%"}
::: {.aside}
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
:::
## Joins (inner)
{fig-align="center" width="50%"}
::: {.aside}
From [gadenbuie/tidyexplain](https://github.com/gadenbuie/tidyexplain)
:::
## join `by`
By default dplyr's join functions will join based on intersecting column names between the two data frames.
To specify the columns to join by (or to handle non-matching names) pass in a character vector of column names (or a named character vector where the names match the left data frame and the values match the right).
. . .
Recently more advanced joins have been implemented using the `join_by()` construct which allows for: equality, inequality, rolling, overlap, and cross joins. See `?join_by` for details.
## Joining people and starships
::: {.medium}
```{r}
sw_df |>
unnest_wider(people) |>
select(name, starships) |>
unnest_longer(starships) |>
left_join(ships, by = c("starships" = "url"))
```
:::
## Putting it together
::: {.small}
```{r}
sw_df |>
unnest_wider(people) |>
select(name, starships) |>
unnest_longer(starships) |>
inner_join(ships, by = c("starships" = "url")) |>
select(-starships) |>
group_by(name) |>
summarize(ships = list(ship), .groups = "drop")
```
:::
##
::: {.small}
```{r}
sw_df |>
unnest_wider(people) |>
select(name, starships) |>
unnest_longer(starships) |>
inner_join(ships, by = c("starships" = "url")) |>
select(-starships) |>
group_by(name) |>
summarize(ships = paste(ship, collapse = ", "), .groups = "drop")
```
:::
## Exercise 2
1. Which planet appeared in the most starwars film (according to the data in `sw_planets`)?
2. Which planet was the homeworld of the most characters in the starwars films?
```{r}
#| echo: false
countdown::countdown(5)
```