You are reading the work-in-progress second edition of R for Data Science. This chapter is currently currently a dumping ground for ideas, and we don’t recommend reading it. You can find the polished first edition at https://r4ds.had.co.nz.
library(tidyverse) #> ── Attaching packages ─────────────────────────────────────── tidyverse 1.3.1 ── #> ✔ ggplot2 3.3.5 ✔ purrr 0.3.4 #> ✔ tibble 3.1.5 ✔ dplyr 1.0.7 #> ✔ tidyr 1.1.4 ✔ stringr 188.8.131.5200 #> ✔ readr 2.0.2 ✔ forcats 0.5.1 #> ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── #> ✖ dplyr::filter() masks stats::filter() #> ✖ dplyr::lag() masks stats::lag()
One important feature of R that can make comparison tricky is missing values, or
NAs (“not availables”).
NA represents an unknown value so missing values are “contagious”: almost any operation involving an unknown value will also be unknown.
NA > 5 #>  NA 10 == NA #>  NA NA + 10 #>  NA NA / 2 #>  NA
The most confusing result is this one:
NA == NA #>  NA
It’s easiest to understand why this is true with a bit more context:
# Let x be Mary's age. We don't know how old she is. x <- NA # Let y be John's age. We don't know how old he is. y <- NA # Are John and Mary the same age? x == y #>  NA # We don't know!
If you want to determine if a value is missing, use
is.na(x) #>  TRUE
How many flights have a missing
dep_time? What other variables are missing? What might these rows represent?
How could you use
arrange()to sort all missing values to the start? (Hint: use
Come up with another approach that will give you the same output as
not_cancelled %>% count(dest)and
not_cancelled %>% count(tailnum, wt = distance)(without using
Look at the number of cancelled flights per day. Is there a pattern? Is the proportion of cancelled flights related to the average delay?
Changing the representation of a dataset brings up an important subtlety of missing values. Surprisingly, a value can be missing in one of two possible ways:
Explicitly, i.e. flagged with
- Implicitly, i.e. simply not present in the data.
Let’s illustrate this idea with a very simple data set:
There are two missing values in this dataset:
The return for the fourth quarter of 2015 is explicitly missing, because the cell where its value should be instead contains
The return for the first quarter of 2016 is implicitly missing, because it simply does not appear in the dataset.
One way to think about the difference is with this Zen-like koan: An explicit missing value is the presence of an absence; an implicit missing value is the absence of a presence.
The way that a dataset is represented can make implicit values explicit. For example, we can make the implicit missing value explicit by putting years in the columns:
stocks %>% pivot_wider(names_from = year, values_from = return) #> # A tibble: 4 × 3 #> qtr `2015` `2016` #> <dbl> <dbl> <dbl> #> 1 1 1.88 NA #> 2 2 0.59 0.92 #> 3 3 0.35 0.17 #> 4 4 NA 2.66
Because these explicit missing values may not be important in other representations of the data, you can set
values_drop_na = TRUE in
pivot_longer() to turn explicit missing values implicit:
stocks %>% pivot_wider(names_from = year, values_from = return) %>% pivot_longer( cols = c(`2015`, `2016`), names_to = "year", values_to = "return", values_drop_na = TRUE ) #> # A tibble: 6 × 3 #> qtr year return #> <dbl> <chr> <dbl> #> 1 1 2015 1.88 #> 2 2 2015 0.59 #> 3 2 2016 0.92 #> 4 3 2015 0.35 #> 5 3 2016 0.17 #> 6 4 2016 2.66
Another important tool for making missing values explicit in tidy data is
complete() takes a set of columns, and finds all unique combinations.
It then ensures the original dataset contains all those values, filling in explicit
NAs where necessary.
There’s one other important tool that you should know for working with missing values. Sometimes when a data source has primarily been used for data entry, missing values indicate that the previous value should be carried forward:
treatment <- tribble( ~person, ~treatment, ~response, "Derrick Whitmore", 1, 7, NA, 2, 10, NA, 3, 9, "Katherine Burke", 1, 4 )
You can fill in these missing values with
It takes a set of columns where you want missing values to be replaced by the most recent non-missing value (sometimes called last observation carried forward).
filter() only includes rows where the condition is
TRUE; it excludes both
If you want to preserve missing values, ask for them explicitly:
Missing values are always sorted at the end: