--- title: "Data frames, matrices, & subsetting" subtitle: "Lecture 05" author: "Dr. Colin Rundel" footer: "Sta 323 - Spring 2025" format: live-revealjs: theme: slides.scss transition: fade slide-number: true self-contained: true revealjs-plugins: - drop execute: echo: true warning: true engine: knitr --- {{< include ./_extensions/r-wasm/live/_knitr.qmd >}} ```{r setup} #| include: False options( width=80 ) ``` # Matrices and Arrays ## Matrices R supports the creation of 2d data structures of atomic vector types. Generally these are formed via a call to `matrix()`. :::: {.columns .medium} ::: {.column width='50%'} ```{r} matrix(1:4, nrow=2, ncol=2) matrix(c(TRUE, FALSE), 2, 2) ``` ::: ::: {.column width='50%'} ```{r} matrix(LETTERS[1:6], 2) matrix(6:1 / 2, ncol = 2) ``` ::: :::: All basic atomic vector types can be used in matrices or arrays. ## Data ordering Matrices in R use column major ordering (data is stored in memory by column). :::: {.columns .small} ::: {.column width='50%'} ```{r} (m = matrix(1:6, nrow=2, ncol=3)) c(m) ``` ::: ::: {.column width='50%'} ```{r} (n = matrix(1:6, nrow=2, ncol=3)) c(n) ``` ::: :::: . . . We can populate a matrix by row, but the data is still stored by column. :::: {.columns .small} ::: {.column width='50%'} ```{r} (x = matrix(1:6, nrow=2, ncol=3, byrow = TRUE)) c(x) ``` ::: ::: {.column width='50%'} ```{r} (y = matrix(1:6, nrow=3, ncol=2, byrow=TRUE)) c(y) ``` ::: :::: ## Matrix structure ::: {.small} ```{r} m = matrix(1:4, ncol=2, nrow=2) ``` ::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} typeof(m) mode(m) ``` ::: ::: {.column width='50%'} ```{r} class(m) attributes(m) ``` ::: :::: . . . Matrices (and arrays) are just atomic vectors with a `dim` attribute attached, they do not have a class attribute, but they do have an implicit `matrix` (and `array`) class. :::: {.columns .small} ::: {.column width='50%'} ```{r} n = letters[1:6] dim(n) = c(2L, 3L) n ``` ::: ::: {.column width='50%'} ```{r} o = letters[1:6] attr(o,"dim") = c(2L, 3L) o ``` ::: :::: ## Arrays Arrays are just an $n$-dimensional extension of matrices and are defined by adding the appropriate dimension sizes. :::: {.columns .small} ::: {.column width='50%'} ```{r} (x = array(1:8, dim = c(2,2,2))) class(x) ``` ::: ::: {.column width='50%'} ```{r} (y = array(letters[1:6], dim = c(3,2,1))) class(y) ``` ::: :::: ::: aside A 2d array will have class `c("matrix","array")` while 1d or >2d will only have class `"array"` ::: # Data Frames ## Data Frames A data frame is how R handles heterogeneous tabular data and is one of the most commonly used data structures in R. ::: {.small} ```{r} (df = data.frame( x = 1:3, y = c("a", "b", "c"), z = c(TRUE) )) ``` ::: . . . R stores data frames using a *list* of equal length *vectors* (atomic or generic). ::: {.small} ```{r} str(df) ``` ::: ## Data Frame Structure :::: {.columns .medium} ::: {.column width='50%'} ```{r} typeof(df) class(df) attributes(df) ``` ::: ::: {.column width='50%' .fragment} ```{r} str(unclass(df)) ``` ::: :::: ## Build your own data.frame ::: {.medium} ```{r} df = list(x = 1:3, y = c("a", "b", "c"), z = c(TRUE, TRUE, TRUE)) ``` ::: . . . :::: {.columns .medium} ::: {.column width='50%'} ```{r} attr(df,"class") = "data.frame" df ``` ::: ::: {.column width='50%'} ```{r} attr(df,"row.names") = 1:3 df ``` ::: :::: . . . ::: {.medium} ```{r} str(df) is.data.frame(df) ``` ::: ## Strings (Characters) vs Factors Previous to R v4.0, the default behavior of data frames was to convert *character* columns into factors. Sometimes this was useful, but mostly it was problematic. This behavior is controlled via the `stringsAsFactors` argument to `data.frame` (and related functions like `read.csv`, `read.table`, etc.). :::: {.columns .small} ::: {.column width='50%'} ```{r} df = data.frame( x = 1:3, y = c("a", "b", "c"), stringsAsFactors = TRUE ) df str(df) ``` ::: ::: {.column width='50%'} ```{r} df = data.frame( x = 1:3, y = c("a", "b", "c"), stringsAsFactors = FALSE ) df str(df) ``` ::: :::: ## Length Coercion When creating a data frame from vectors, the lengths of the component vectors will be coerced to match. However, if they not multiples of each other then there will be an error (other previous forms of length coercion would produce a warning for this case). ::: small ```{r} #| error: True data.frame(x = 1:3, y = c("a")) ``` ::: . . . ::: small ```{r} #| error: True data.frame(x = 1:3, y = c("a","b")) ``` ::: . . . ::: small ```{r} #| error: True data.frame(x = 1:3, y = character()) ``` ::: # Subsetting ## Subsetting in General R has three subsetting operators (`[`, `[[`, and `$`). The behavior of these operators depends on the object (class) they are being used with.
In general there are 6 different types of subsetting that can be performed: :::: {.columns} ::: {.column width='50%'} * Positive integer * Negative integer * Logical value ::: ::: {.column width='50%'} * Empty / NULL * Zero valued * Character value (names) ::: :::: ## Positive Integer subsetting Returns elements at the given location(s) :::: {.small} ```{r} x = c(1,4,7) y = list(1,4,7) ``` :::: {.columns .small} ::: {.column width='50%'} ```{r} x[1] x[c(1,3)] x[c(1,1)] x[c(1.9,2.1)] ``` ::: ::: {.column width='17%'} ::: ::: {.column width='33%'} ```{r} str( y[1] ) str( y[c(1,3)] ) str( y[c(1,1)] ) str( y[c(1.9,2.1)] ) ``` ::: :::: :::: ::: {.aside} Note - R uses a 1-based indexing scheme ::: ## Negative Integer subsetting Excludes elements at the given location(s) :::: {.columns .medium} ::: {.column width='50%'} ```{r, error=TRUE} x = c(1,4,7) ``` ```{r, error=TRUE} x[-1] x[-c(1,3)] x[c(-1,-1)] ``` ::: ::: {.column width='50%'} ```{r, error=TRUE} y = list(1,4,7) ``` ```{r, error=TRUE} str( y[-1] ) str( y[-c(1,3)] ) ``` ::: :::: . . . ::: {.medium} ```{r} #| error: True x[c(-1,2)] y[c(-1,2)] ``` ::: ## Logical Value Subsetting Returns elements that correspond to `TRUE` in the logical vector. Length of the logical vector is coerced to be the same as the vector being subsetted. :::: {.columns .small} ::: {.column width='50%'} ```{r} x = c(1,4,7,12) ``` ```{r, error=TRUE} x[c(TRUE,TRUE,FALSE,TRUE)] x[c(TRUE,FALSE)] ``` ::: ::: {.column width='50%'} ```{r, error=TRUE} y = list(1,4,7,12) ``` ```{r, error=TRUE} str( y[c(TRUE,TRUE,FALSE,TRUE)] ) str( y[c(TRUE,FALSE)] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} #| error: True x[x %% 2 == 0] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str( y[y %% 2 == 0] ) ``` ::: :::: ## Empty Subsetting Returns the original vector, this is not the same as subsetting with `NULL` :::: {.columns} ::: {.column width='50%'} ```{r} x = c(1,4,7) ``` ```{r} x[] x[NULL] ``` ::: ::: {.column width='50%'} ```{r} y = list(1,4,7) ``` ```{r} str(y[]) str(y[NULL]) ``` ::: :::: ## Zero subsetting Returns an empty vector (of the same type), this is the same as subsetting with `NULL` :::: {.columns .small} ::: {.column width='50%'} ```{r} x = c(1,4,7) ``` ```{r} x[0] ``` ::: ::: {.column width='50%'} ```{r} y = list(1,4,7) str(y[0]) ``` ::: :::: . . . `0`s can be mixed with either positive or negative integers for subsetting, but they are ignored in both cases. :::: {.columns .small} ::: {.column width='50%'} ```{r} x[c(0,1)] y[c(0,1)] ``` ::: ::: {.column width='50%'} ```{r} x[c(0,-1)] y[c(0,-1)] ``` ::: :::: ## Character subsetting If the vector has names, selects elements whose names correspond to the values in the name vector. :::: {.columns .medium} ::: {.column width='50%'} ```{r} x = c(a=1, b=4, c=7) ``` ```{r} x["a"] x[c("a","a")] x[c("b","c")] ``` ::: ::: {.column width='50%'} ```{r} y = list(a=1,b=4,c=7) ``` ```{r} str(y["a"]) str(y[c("a","a")]) str(y[c("b","c")]) ``` ::: :::: ## Out of bounds :::: {.columns .small} ::: {.column width='50%'} ```{r} x = c(1,4,7) ``` ```{r} x[4] x[-4] x["a"] x[c(1,4)] ``` ::: ::: {.column width='50%'} ```{r} y = list(1,4,7) ``` ```{r} str(y[4]) str(y[-4]) str(y["a"]) str(y[c(1,4)]) ``` ::: :::: ## Missing values :::: {.columns} ::: {.column width='50%'} ```{r} x = c(1,4,7) ``` ```{r} x[NA] x[c(1,NA)] ``` ::: ::: {.column width='50%'} ```{r} y = list(1,4,7) ``` ```{r} str(y[NA]) str(y[c(1,NA)]) ``` ::: :::: ## NULL and empty vectors (length 0) This final type of subsetting follows the rules for length coercion with a 0-length vector (i.e. the vector being subset gets coerced to having length 0 if the subsetting vector has length 0) :::: {.columns} ::: {.column width='50%'} ```{r} x = c(1,4,7) ``` ```{r} x[NULL] x[integer()] x[character()] ``` ::: ::: {.column width='50%'} ```{r} y = list(1,4,7) ``` ```{r} y[NULL] y[integer()] y[character()] ``` ::: :::: ## Exercise 1 {.scrollable} ::: {.medium} Below are 100 values, write down how you would create a subset to
accomplish each of the following: * Select every third value starting at position 2 in `x`. * Remove all values with an odd index (e.g. 1, 3, etc.) * Remove every 4th value, but only if it is odd. ::: ```{webr} #| min-lines: 8 x = c(56, 3, 17, 2, 4, 9, 6, 5, 19, 5, 2, 3, 5, 0, 13, 12, 6, 31, 10, 21, 8, 4, 1, 1, 2, 5, 16, 1, 3, 8, 1, 3, 4, 8, 5, 2, 8, 6, 18, 40, 10, 20, 1, 27, 2, 11, 14, 5, 7, 0, 3, 0, 7, 0, 8, 10, 10, 12, 8, 82, 21, 3, 34, 55, 18, 2, 9, 29, 1, 4, 7, 14, 7, 1, 2, 7, 4, 74, 5, 0, 3, 13, 2, 8, 1, 6, 13, 7, 1, 10, 5, 2, 4, 4, 14, 15, 4, 17, 1, 9) ``` ```{r} #| echo: false countdown::countdown(5, top="0") ``` # The other subset operators
`[[` and `$` ## Atomic vectors - [ vs. [[ `[[` subsets like `[` except it can only subset for a *single* value ::: {.small} ```{r, error=TRUE} x = c(a=1,b=4,c=7) ``` ```{r} x[1] ``` ::: . . . ::: {.small} ```{r, error=TRUE} x[[1]] x[["a"]] x[[1:2]] x[[TRUE]] ``` ::: ## Generic Vectors (lists) - [ vs. [[ Subsets a single value, but returns the value - not a list containing that value. Multiple values are interpreted as nested subsetting. ::: {.small} ```{r, error=TRUE} y = list(a=1, b=4, c=7:9) ``` ::: :::: {.columns .small} ::: {.column width='50%'} ```{r, error=TRUE} y[2] ``` ::: ::: {.column width='50%'} ```{r, error=TRUE} str( y[2] ) ``` ::: :::: . . . ::: {.small} ```{r, error=TRUE} y[[2]] y[["b"]] y[[1:2]] y[[2:1]] ``` ::: ## Hadley's Analogy (1) ```{r echo=FALSE, fig.align="center", out.width="37%"} knitr::include_graphics("imgs/list_train1.png") ``` . . . ```{r echo=FALSE, fig.align="center", out.width="37%"} knitr::include_graphics("imgs/list_train2.png") ``` . . . ```{r echo=FALSE, fig.align="center", out.width="37%"} knitr::include_graphics("imgs/list_train3.png") ``` ::: {.aside} From Advanced R - [Chapter 4.3](https://adv-r.hadley.nz/subsetting.html#subset-single) ::: ## Hadley's Analogy (2) ```{r echo=FALSE, fig.align="center", out.width="80%"} knitr::include_graphics("imgs/pepper_subset.png") ``` ## [[ vs. $ `$` is equivalent to `[[` but it only works for name based subsetting of *lists* (it also uses partial matching for names) ::: {.medium} ```{r, error=TRUE} x = c("abc"=1, "def"=5) ``` ```{r, error=TRUE} x$abc ``` ::: . . . ::: {.medium} ```{r} y = list("abc"=1, "def"=5) ``` ```{r} y[["abc"]] y$abc y$d ``` ::: ## A common error Why does the following code not work? ```{r} #| error: True x = list(abc = 1:10, def = 10:1) y = "abc" ``` :::: {.columns} ::: {.column width='50%'} ```{r} x[[y]] ``` ::: ::: {.column width='50%'} ```{r} x$y ``` ::: :::: . . . The expression `x$y` gets interpreted as `x[["y"]]` by R, note the inclusion of the `"`s, this is not the same as the expression `x[[y]]`. # Subsetting Data Frames ## Subsetting rows As data frames have 2 dimensions, we can subset on either the rows or the columns - the subsetting values are separated by a comma. ::: {.small} ```{r} (df = data.frame(x = 1:3, y = c("A","B","C"), z = TRUE)) ``` ::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[1, ] ``` ::: ::: {.column width='50%'} ```{r} str( df[1, ] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[c(1,3), ] ``` ::: ::: {.column width='50%'} ```{r} str( df[c(1,3), ] ) ``` ::: :::: ## Subsetting Columns ::: {.small} ```{r} df ``` ::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[, 1] ``` ::: ::: {.column width='50%'} ```{r} str( df[, 1] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[, 1:2] ``` ::: ::: {.column width='50%'} ```{r} str( df[, 1:2] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[, -3] ``` ::: ::: {.column width='50%'} ```{r} str( df[, -3] ) ``` ::: :::: ## Subsetting both ::: {.small} ```{r} df ``` ::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[1, 1] ``` ::: ::: {.column width='50%'} ```{r} str( df[1, 1] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[1:2, 1:2] ``` ::: ::: {.column width='50%'} ```{r} str( df[1:2, 1:2] ) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} df[-1, 2:3] ``` ::: ::: {.column width='50%'} ```{r} str( df[-1, 2:3] ) ``` ::: :::: ## Preserving vs Simplifying Most of the time, R's `[` subset operator is a *preserving* operator, in that the returned object will always have the same type/class as the object being subset. Confusingly, when used with some classes (e.g. data frame, matrix or array) `[` becomes a *simplifying* operator (does not preserve type) - this behavior is instead controlled by the `drop` argument. ## Drop w/ row subset :::: {.columns .small} ::: {.column width='50%'} ```{r} #| error: True df[1, ] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[1, ]) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} #| error: True df[1, , drop=TRUE] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[1, , drop=TRUE]) ``` ::: :::: ## Drop w/ column subset :::: {.columns .small} ::: {.column width='50%'} ```{r} #| error: True df[, 1] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[, 1]) ``` ::: :::: . . . :::: {.columns .small} ::: {.column width='50%'} ```{r} #| error: True df[, 1, drop=FALSE] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[, 1, drop=FALSE]) ``` ::: :::: ## Exceptions `drop` only works when the resulting value can be represented as a 1d vector (either a list or atomic). :::: {.columns} ::: {.column width='50%'} ```{r} #| error: True df[1:2, 1:2] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[1:2, 1:2]) ``` ::: :::: . . . :::: {.columns} ::: {.column width='50%'} ```{r} #| error: True df[1:2, 1:2, drop=TRUE] ``` ::: ::: {.column width='50%'} ```{r} #| error: True str(df[1:2, 1:2, drop=TRUE]) ``` ::: :::: ## Preserving vs Simplifying Subsets
Type | Simplifying | Preserving :----------------|:-------------------------|:-------------------------------------------------------- Atomic Vector | `x[[1]]` | `x[1]` List | `x[[1]]` | `x[1]` Matrix / Array | `x[[1]]`
`x[1, ]`
`x[, 1]` | `x[1, , drop=FALSE]`
`x[, 1, drop=FALSE]` Factor | `x[1:4, drop=TRUE]` | `x[1:4]`
`x[[1]]` Data frame | `x[, 1]`
`x[[1]]` | `x[, 1, drop=FALSE]`
`x[1]` ```{r} #| echo: False knitr::knit_exit() ``` # Subsetting and assignment ## Subsetting and assignment Subsets can also be used with assignment to update specific values within an object (in-place). ```{r} x = c(1, 4, 7, 9, 10, 15) ``` . . . ```{r} x[2] = 2 x ``` . . . ```{r} x %% 2 != 0 ``` . . . ```{r} x[x %% 2 != 0] = (x[x %% 2 != 0] + 1) / 2 x ``` . . . ```{r} x[c(1,1)] = c(2,3) x ``` ```{r} x = 1:6 ``` ```{r} x[c(2,NA)] = 1 x ``` . . . ```{r} x = 1:6 ``` ```{r} x[c(-1,-2)] = 3 x ``` . . . ```{r} x = 1:6 ``` ```{r} x[c(TRUE,NA)] = 1 x ``` . . . ```{r} x = 1:6 ``` ```{r} x[] = 1:3 x ``` ## Subsets of Subsets ```{r} ( df = data.frame(a = c(5,1,NA,3)) ) ``` ```{r} df$a[df$a == 5] = 0 df ``` . . . ```{r} df[1][df[1] != 3] = -1 df ``` ## Exercise 2 Some data providers choose to encode missing values using values like `-999`. Below is a sample data frame with missing values encoded in this way. ```{r} d = data.frame( patient_id = c(1, 2, 3, 4, 5), age = c(32, 27, 56, 19, 65), bp = c(110, 100, 125, -999, -999), o2 = c(97, 95, -999, -999, 99) ) ``` * *Task 1* - using the subsetting tools we've discussed come up with code that will replace the `-999` values in the `bp` and `o2` column with actual `NA` values. Save this as `d_na`. * *Task 2* - Once you have created `d_na` come up with code that translate it back into the original data frame `d`, i.e. replace the `NA`s with `-999`.