18 Strings

You are reading the work-in-progress second edition of R for Data Science. This chapter is currently undergoing heavy restructuring and may be confusing or incomplete. You can find the polished first edition at https://r4ds.had.co.nz.

18.1 Introduction

This chapter introduces you to strings. You’ll learn the basics of how strings work in R and how to create them “by hand”. Big topic so spread over three chapters: here we’ll focus on the basic mechanics, in Chapter 19 we’ll dive into the details of regular expressions the sometimes cryptic language for describing patterns in strings, and we’ll return to strings later in Chapter 37 when we think about them about from a programming perspective (rather than a data analysis perspective). We’ll finish up with a discussion of some of the new challenges that arise when working with non-English strings.

While base R contains functions that allow us to perform pretty much all of the operations described in this chapter, here we’re going to use the stringr package. stringr has been carefully designed to be as consistent as possible so that knowledge gained about one function can be more easily transferred to the next. stringr functions all start with the same str_ prefix. This is particularly useful if you use RStudio, because typing str_ will trigger autocomplete, allowing you to see all stringr’s functions:

18.1.1 Prerequisites

This chapter will focus on the stringr package for string manipulation, which is part of the core tidyverse. We’ll also work with the babynames dataset.

18.2 Creating a string

To begin, let’s discuss the mechanics of creating a string. We’ve created strings in passing earlier in the book, but didn’t discuss the details. First, there are two basic ways to create a string: using either single quotes (') or double quotes ("). Unlike other languages, there is no difference in behaviour. I recommend always using ", unless you want to create a string that contains multiple ".

string1 <- "This is a string"
string2 <- 'If I want to include a "quote" inside a string, I use single quotes'

If you forget to close a quote, you’ll see +, the continuation character:

> "This is a string without a closing quote
+ 
+ 
+ HELP I'M STUCK

If this happen to you, press Escape and try again.

18.2.1 Escapes

To include a literal single or double quote in a string you can use \ to “escape” it:

double_quote <- "\"" # or '"'
single_quote <- '\'' # or "'"

Which means if you want to include a literal backslash, you’ll need to double it up: "\\":

backslash <- "\\"

Beware that the printed representation of a string is not the same as string itself, because the printed representation shows the escapes. To see the raw contents of the string, use str_view():

x <- c(single_quote, double_quote, backslash)
x
#> [1] "'"  "\"" "\\"
str_view(x)
#> '
#> "
#> \

18.2.2 Raw strings

Creating a string with multiple quotes or backslashes gets confusing quickly. For example, lets create a string that contains the contents of the chunk where I define the double_quote and single_quote variables:

tricky <- "double_quote <- \"\\\"\" # or '\"'
single_quote <- '\\'' # or \"'\""
str_view(tricky)
#> double_quote <- "\"" # or '"'
#> single_quote <- '\'' # or "'"

You can instead use a raw string4 to reduce the amount of escaping:

tricky <- r"(double_quote <- "\"" # or '"'
single_quote <- '\'' # or "'"
)"
str_view(tricky)
#> double_quote <- "\"" # or '"'
#> single_quote <- '\'' # or "'"

A raw string starts with r"( and finishes with )". If your string contains )" you can instead use r"[]" or r"{}", and if that’s still not enough, you can insert any number of dashes to make the opening and closing pairs unique, e.g. `r"--()--", `r"---()---",etc.

18.2.3 Other special characters

As well as \", \', and \\ there are a handful of other special characters that may come in handy. The most common are "\n", newline, and "\t", tab, but you can see the complete list in ?'"'.

You’ll also sometimes see strings containing Unicode escapes that start with \u or \U. This is a way of writing non-English characters that works on all systems:

x <- c("\u00b5", "\U0001f604")
x
#> [1] "µ"  "😄"
str_view(x)
#> µ
#> 😄

18.3 Combining strings

Use str_c()5 to join together multiple character vectors into a single vector:

str_c("x", "y")
#> [1] "xy"
str_c("x", "y", "z")
#> [1] "xyz"

str_c() obeys the usual recycling rules:

names <- c("Timothy", "Dewey", "Mable")
str_c("Hi ", names, "!")
#> [1] "Hi Timothy!" "Hi Dewey!"   "Hi Mable!"

And like most other functions in R, missing values are contagious. You can use coalesce() to replace missing values with a value of your choosing:

x <- c("abc", NA)
str_c("|-", x, "-|")
#> [1] "|-abc-|" NA
str_c("|-", coalesce(x, ""), "-|")
#> [1] "|-abc-|" "|--|"

Since str_c() creates a vector, you’ll usually use it with a mutate():

starwars %>% 
  mutate(greeting = str_c("Hi! I'm ", name, "."), .after = name)
#> # A tibble: 87 × 15
#>   name   greeting  height  mass hair_color skin_color eye_color birth_year sex  
#>   <chr>  <chr>      <int> <dbl> <chr>      <chr>      <chr>          <dbl> <chr>
#> 1 Luke … Hi! I'm …    172    77 blond      fair       blue            19   male 
#> 2 C-3PO  Hi! I'm …    167    75 NA         gold       yellow         112   none 
#> 3 R2-D2  Hi! I'm …     96    32 NA         white, bl… red             33   none 
#> 4 Darth… Hi! I'm …    202   136 none       white      yellow          41.9 male 
#> 5 Leia … Hi! I'm …    150    49 brown      light      brown           19   fema…
#> 6 Owen … Hi! I'm …    178   120 brown, gr… light      blue            52   male 
#> # … with 81 more rows, and 6 more variables: gender <chr>, homeworld <chr>,
#> #   species <chr>, films <list>, vehicles <list>, starships <list>

Another powerful way of combining strings is with the glue package. You can either use glue::glue() directly or call it via the str_glue() wrapper that stringr provides for you. Glue works a little differently to the other methods: you give it a single string then within the string use {} to indicate where existing variables should be evaluated:

str_glue("|-{x}-|")
#> |-abc-|
#> |-NA-|

Like str_c(), str_glue() pairs well with mutate():

starwars %>% 
  mutate(
    intro = str_glue("Hi! My is {name} and I'm a {species} from {homeworld}"),
    .keep = "none"
  )
#> # A tibble: 87 × 1
#>   intro                                                 
#>   <glue>                                                
#> 1 Hi! My is Luke Skywalker and I'm a Human from Tatooine
#> 2 Hi! My is C-3PO and I'm a Droid from Tatooine         
#> 3 Hi! My is R2-D2 and I'm a Droid from Naboo            
#> 4 Hi! My is Darth Vader and I'm a Human from Tatooine   
#> 5 Hi! My is Leia Organa and I'm a Human from Alderaan   
#> 6 Hi! My is Owen Lars and I'm a Human from Tatooine     
#> # … with 81 more rows

You can use any valid R code inside of {}, but it’s a good idea to pull complex calculations out into their own variables so you can more easily check your work.

18.4 Length and subsetting

It’s natural to think about the letters that make up an individual string. (But note that the idea of a “letter” isn’t a natural fit to every language, we’ll come back to that in Section 18.14). For example, str_length() tells you the length of a string in characters:

str_length(c("a", "R for data science", NA))
#> [1]  1 18 NA

You could use this with count() to find the distribution of lengths of US babynames, and then with filter() to look at the longest names:

babynames %>%
  count(length = str_length(name), wt = n)
#> # A tibble: 14 × 2
#>   length        n
#>    <int>    <int>
#> 1      2   338150
#> 2      3  8589596
#> 3      4 48506739
#> 4      5 87011607
#> 5      6 90749404
#> 6      7 72120767
#> # … with 8 more rows

babynames %>% 
  filter(str_length(name) == 15) %>% 
  count(name, wt = n, sort = TRUE)
#> # A tibble: 34 × 2
#>   name                n
#>   <chr>           <int>
#> 1 Franciscojavier   123
#> 2 Christopherjohn   118
#> 3 Johnchristopher   118
#> 4 Christopherjame   108
#> 5 Christophermich    52
#> 6 Ryanchristopher    45
#> # … with 28 more rows

You can extract parts of a string using str_sub(string, start, end). The start and end arguments are inclusive, so the length of the returned string will be end - start + 1:

x <- c("Apple", "Banana", "Pear")
str_sub(x, 1, 3)
#> [1] "App" "Ban" "Pea"

You can use negative values to count back from the end of the string: -1 is the last character, -2 is the second to last character, etc.

str_sub(x, -3, -1)
#> [1] "ple" "ana" "ear"

Note that str_sub() won’t fail if the string is too short: it will just return as much as possible:

str_sub("a", 1, 5)
#> [1] "a"

We could use str_sub() with mutate() to find the first and last letter of each name:

babynames %>% 
  mutate(
    first = str_sub(name, 1, 1),
    last = str_sub(name, -1, -1)
  )
#> # A tibble: 1,924,665 × 7
#>    year sex   name          n   prop first last 
#>   <dbl> <chr> <chr>     <int>  <dbl> <chr> <chr>
#> 1  1880 F     Mary       7065 0.0724 M     y    
#> 2  1880 F     Anna       2604 0.0267 A     a    
#> 3  1880 F     Emma       2003 0.0205 E     a    
#> 4  1880 F     Elizabeth  1939 0.0199 E     h    
#> 5  1880 F     Minnie     1746 0.0179 M     e    
#> 6  1880 F     Margaret   1578 0.0162 M     t    
#> # … with 1,924,659 more rows

Sometimes you’ll get a column that’s made up of individual fixed length strings that have been joined together:

df <- tribble(
  ~ sex_year_age,
  "M200115",
  "F201503",
)

You can extract the columns using str_sub():

df %>% mutate(
  sex = str_sub(sex_year_age, 1, 1),
  year = str_sub(sex_year_age, 2, 5),
  age = str_sub(sex_year_age, 6, 7),
)
#> # A tibble: 2 × 4
#>   sex_year_age sex   year  age  
#>   <chr>        <chr> <chr> <chr>
#> 1 M200115      M     2001  15   
#> 2 F201503      F     2015  03

Or use the separate() helper function:

df %>% 
  separate(sex_year_age, c("sex", "year", "age"), c(1, 5))
#> # A tibble: 2 × 3
#>   sex   year  age  
#>   <chr> <chr> <chr>
#> 1 M     2001  15   
#> 2 F     2015  03

Note that you give separate() three columns but only two positions — that’s because you’re telling separate() where to break up the string.

TODO: draw diagram to emphasise that it’s the space between the characters.

Later on, we’ll come back two related problems: the components have varying length and are a separated by a character, or they have an varying number of components and you want to split up into rows, rather than columns.

18.4.1 Exercises

  1. Use str_length() and str_sub() to extract the middle letter from each baby name. What will you do if the string has an even number of characters?

18.5 Long strings

Sometimes the reason you care about the length of a string is because you’re trying to fit it into a label. stringr provides two useful tools for cases where your string is too long:

  • str_trunc(x, 20) ensures that no string is longer than 20 characters, replacing any thing too long with .

  • str_wrap(x, 20) wraps a string introducing new lines so that each line is at most 20 characters (it doesn’t hyphenate, however, so any word longer than 20 characters will make a longer time)

x <- "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat."

str_trunc(x, 30)
#> [1] "Lorem ipsum dolor sit amet,..."
str_view(str_wrap(x, 30))
#> Lorem ipsum dolor sit amet,
#> consectetur adipiscing
#> elit, sed do eiusmod tempor
#> incididunt ut labore et dolore
#> magna aliqua. Ut enim ad
#> minim veniam, quis nostrud
#> exercitation ullamco laboris
#> nisi ut aliquip ex ea commodo
#> consequat.

18.6 String summaries

str_c() combines multiple character vectors into a single character vector; the output is the same length as the input. An related function is str_flatten(): it takes a character vector and returns a single string:

str_flatten(c("x", "y", "z"))
#> [1] "xyz"
str_flatten(c("x", "y", "z"), ", ")
#> [1] "x, y, z"
str_flatten(c("x", "y", "z"), ", ", ", and ")
#> [1] "x, y, and z"

Just like sum() and mean() take a vector of numbers and return a single number, str_flatten() takes a character vector and returns a single string. This makes str_flatten() a summary function for strings, so you’ll often pair it with summarise():

df <- tribble(
  ~ name, ~ fruit,
  "Carmen", "banana",
  "Carmen", "apple",
  "Marvin", "nectarine",
  "Terence", "cantaloupe",
  "Terence", "papaya",
  "Terence", "madarine"
)
df %>%
  group_by(name) %>% 
  summarise(fruits = str_flatten(fruit, ", "))
#> # A tibble: 3 × 2
#>   name    fruits                      
#>   <chr>   <chr>                       
#> 1 Carmen  banana, apple               
#> 2 Marvin  nectarine                   
#> 3 Terence cantaloupe, papaya, madarine

18.7 Detect matches

To determine if a character vector matches a pattern, use str_detect(). It returns a logical vector the same length as the input:

x <- c("apple", "banana", "pear")
str_detect(x, "e")
#> [1]  TRUE FALSE  TRUE

This makes it a logical pairing with filter(). The following example returns all names that contain a lower-case “x”:

babynames %>% filter(str_detect(name, "x"))
#> # A tibble: 16,317 × 5
#>    year sex   name          n      prop
#>   <dbl> <chr> <chr>     <int>     <dbl>
#> 1  1880 F     Roxie        62 0.000635 
#> 2  1880 F     Dixie        15 0.000154 
#> 3  1880 F     Roxanna       9 0.0000922
#> 4  1880 F     Texas         5 0.0000512
#> 5  1880 M     Alexander   211 0.00178  
#> 6  1880 M     Alex        147 0.00124  
#> # … with 16,311 more rows

Remember that when you use a logical vector in a numeric context, FALSE becomes 0 and TRUE becomes 1. That means you can use summarise() with sum() or mean() and str_detect() if you want to answer questions about the prevalence of patterns. For example, the following snippet, gives the proportion of names containing an “x” by year:

babynames %>% 
  group_by(year) %>% 
  summarise(prop_x = mean(str_detect(name, "x"))) %>% 
  ggplot(aes(year, prop_x)) + 
  geom_line()
A timeseries showing the proportion of baby names that contain the letter x. The proportion declines gradually from 8 per 1000 in 1880 to 4 per 1000 in 1980, then increases rapidly to 16 per 1000 in 2019.

(Note that this gives us the proportion of names that contain an x; if you wanted the proportion of babies given a name containing an x, you’d need to perform a weighted mean).

A variation on str_detect() is str_count(): rather than a simple yes or no, it tells you how many matches there are in a string:

str_count(x, "p")
#> [1] 2 0 1

It’s natural to use str_count() with mutate():

babynames %>% 
  distinct(name) %>% 
  mutate(
    vowels = str_count(name, "[aeiou]"),
    consonants = str_count(name, "[^aeiou]")
  )
#> # A tibble: 97,310 × 3
#>   name      vowels consonants
#>   <chr>      <int>      <int>
#> 1 Mary           1          3
#> 2 Anna           1          3
#> 3 Emma           1          3
#> 4 Elizabeth      3          6
#> 5 Minnie         3          3
#> 6 Margaret       3          5
#> # … with 97,304 more rows

18.7.1 Exercises

  1. What word has the highest number of vowels? What word has the highest proportion of vowels? (Hint: what is the denominator?)

18.8 Introduction to regular expressions

Before we can continue on we need to discuss the second argument to str_detect() — the pattern that you want to match. Above, I used a simple string, but the pattern actually a much richer tool called a regular expression. A regular expression uses special characters to match string patterns. For example, . will match any character, so "a." will match any string that contains an a followed by another character:

str_detect(c("a", "ab", "ae", "bd", "ea", "eab"), "a.")
#> [1] FALSE  TRUE  TRUE FALSE FALSE  TRUE

str_view() shows you regular expressions to help understand what’s happening:

str_view(c("a", "ab", "ae", "bd", "ea", "eab"), "a.")
#> a
#> <ab>
#> <ae>
#> bd
#> ea
#> e<ab>

Regular expressions are a powerful and flexible language which we’ll come back to in Chapter 19. Here we’ll use only the most important components of the syntax as you learn the other stringr tools for working with patterns.

There are three useful quantifiers that can be applied to other pattern: ? makes a pattern option (i.e. it matches 0 or 1 times), + lets a pattern repeat (ie. it matches at least once), and * lets a pattern be optional or repeat (i.e. it matches any number of times, including 0).

  • ab? match an “a”, optionally followed by a b

  • ab+ matches an “a”, followed by at least one b

  • ab* matches an “a”, followed by any number of bs

You can use () to control precedence:

  • (ab)? optionally matches “ab”

  • (ab)+ matches one or more “ab” repeats

str_view(c("aba", "ababab", "abbbbbb"), "ab+")
#> <ab>a
#> <ab>abab
#> <abbbbbb>
str_view(c("aba", "ababab", "abbbbbb"), "(ab)+")
#> <ab>a
#> <ababab>
#> <ab>bbbbb

There are various alternatives to . that match a restricted set of characters. One useful operator is the character class: [abcd] match “a”, “b”, “c”, or “d”; [^abcd] matches anything except “a”, “b”, “c”, or “d”.

You can opt-out of the regular expression rules by using fixed:

str_view(c("", "a", "."), fixed("."))
#> 
#> a
#> <.>

Note that both fixed strings and regular expressions are case sensitive by default. You can opt out by setting ignore_case = TRUE.

str_view_all("x  X  xy", "X")
#> x  <X>  xy
str_view_all("x  X  xy", fixed("X", ignore_case = TRUE))
#> <x>  <X>  <x>y
str_view_all("x  X  xy", regex(".Y", ignore_case = TRUE))
#> x  X  <xy>

We’ll come back to case later, because it’s not trivial for many languages.

18.8.1 Exercises

  1. For each of the following challenges, try solving it by using both a single regular expression, and a combination of multiple str_detect() calls.

    1. Find all words that start or end with x.
    2. Find all words that start with a vowel and end with a consonant.
    3. Are there any words that contain at least one of each different vowel?

18.9 Replacing matches

str_replace_all() allow you to replace matches with new strings. The simplest use is to replace a pattern with a fixed string:

x <- c("apple", "pear", "banana")
str_replace_all(x, "[aeiou]", "-")
#> [1] "-ppl-"  "p--r"   "b-n-n-"

With str_replace_all() you can perform multiple replacements by supplying a named vector. The name gives a regular expression to match, and the value gives the replacement.

x <- c("1 house", "1 person has 2 cars", "3 people")
str_replace_all(x, c("1" = "one", "2" = "two", "3" = "three"))
#> [1] "one house"               "one person has two cars"
#> [3] "three people"

str_remove_all() is a short cut for str_replace_all(x, pattern, "") — it removes matching patterns from a string.

Use in mutate()

Using pipe inside mutate. Recommendation to make a function, and think about testing it — don’t need formal tests, but useful to build up a set of positive and negative test cases as you.

18.9.0.1 Exercises

  1. Replace all forward slashes in a string with backslashes.

  2. Implement a simple version of str_to_lower() using str_replace_all().

  3. Switch the first and last letters in words. Which of those strings are still words?

18.10 Extract full matches

If your data is in a tibble, it’s often easier to use tidyr::extract(). It works like str_match() but requires you to name the matches, which are then placed in new columns:

tibble(sentence = sentences) %>% 
  tidyr::extract(
    sentence, c("article", "noun"), "(a|the) ([^ ]+)", 
    remove = FALSE
  )
#> # A tibble: 720 × 3
#>   sentence                                    article noun   
#>   <chr>                                       <chr>   <chr>  
#> 1 The birch canoe slid on the smooth planks.  the     smooth 
#> 2 Glue the sheet to the dark blue background. the     sheet  
#> 3 It's easy to tell the depth of a well.      the     depth  
#> 4 These days a chicken leg is a rare dish.    a       chicken
#> 5 Rice is often served in round bowls.        NA      NA     
#> 6 The juice of lemons makes fine punch.       NA      NA     
#> # … with 714 more rows

18.10.1 Exercises

  1. In the previous example, you might have noticed that the regular expression matched “flickered”, which is not a colour. Modify the regex to fix the problem.
  2. Find all words that come after a “number” like “one”, “two”, “three” etc. Pull out both the number and the word.
  3. Find all contractions. Separate out the pieces before and after the apostrophe.

18.11 Strings -> Columns

18.12 Separate

separate() pulls apart one column into multiple columns, by splitting wherever a separator character appears. Take table3:

table3
#> # A tibble: 6 × 3
#>   country      year rate             
#> * <chr>       <int> <chr>            
#> 1 Afghanistan  1999 745/19987071     
#> 2 Afghanistan  2000 2666/20595360    
#> 3 Brazil       1999 37737/172006362  
#> 4 Brazil       2000 80488/174504898  
#> 5 China        1999 212258/1272915272
#> 6 China        2000 213766/1280428583

The rate column contains both cases and population variables, and we need to split it into two variables. separate() takes the name of the column to separate, and the names of the columns to separate into, as shown in Figure 18.1 and the code below.

table3 %>%
  separate(rate, into = c("cases", "population"))
#> # A tibble: 6 × 4
#>   country      year cases  population
#>   <chr>       <int> <chr>  <chr>     
#> 1 Afghanistan  1999 745    19987071  
#> 2 Afghanistan  2000 2666   20595360  
#> 3 Brazil       1999 37737  172006362 
#> 4 Brazil       2000 80488  174504898 
#> 5 China        1999 212258 1272915272
#> 6 China        2000 213766 1280428583
Two panels, one with a data frame with three columns (country, year, and rate) and the other with a data frame with four columns (country, year, cases, and population). Arrows show how the rate variable is separated into two variables: cases and population.

Figure 18.1: Separating rate into cases and population to make table3 tidy

By default, separate() will split values wherever it sees a non-alphanumeric character (i.e. a character that isn’t a number or letter). For example, in the code above, separate() split the values of rate at the forward slash characters. If you wish to use a specific character to separate a column, you can pass the character to the sep argument of separate(). For example, we could rewrite the code above as:

table3 %>%
  separate(rate, into = c("cases", "population"), sep = "/")

separate_rows()

18.13 Strings -> Rows

starwars %>% 
  select(name, eye_color) %>% 
  filter(str_detect(eye_color, ", ")) %>% 
  separate_rows(eye_color)
#> # A tibble: 4 × 2
#>   name     eye_color
#>   <chr>    <chr>    
#> 1 R4-P17   red      
#> 2 R4-P17   blue     
#> 3 Grievous green    
#> 4 Grievous yellow

18.13.1 Exercises

  1. Split up a string like "apples, pears, and bananas" into individual components.

  2. Why is it better to split up by boundary("word") than " "?

  3. What does splitting with an empty string ("") do? Experiment, and then read the documentation.

18.14 Other writing systems

Unicode is a system for representing the many writing systems used around the world. Fundamental unit is a code point. This usually represents something like a letter or symbol, but might also be formatting like a diacritic mark or a (e.g.) the skin tone of an emoji. Character vs grapheme cluster.

Include some examples from https://gankra.github.io/blah/text-hates-you/.

All stringr functions default to the English locale. This ensures that your code works the same way on every system, avoiding subtle bugs.

Maybe things you think are true, but aren’t list?

18.14.1 Encoding

You will not generally find the base R Encoding() to be useful because it only supports three different encodings (and interpreting what they mean is non-trivial) and it only tells you the encoding that R thinks it is, not what it really is. And typically the problem is that the declaring encoding is wrong.

The tidyverse follows best practices6 of using UTF-8 everywhere, so any string you create with the tidyverse will use UTF-8. It’s still possible to have problems, but they’ll typically arise during data import. Once you’ve diagnosed you have an encoding problem, you should fix it in data import (i.e. by using the encoding argument to readr::locale()).

18.14.2 Length and subsetting

This seems like a straightforward computation if you’re only familiar with English, but things get complex quick when working with other languages.

Four most common are Latin, Chinese, Arabic, and Devangari, which represent three different systems of writing systems:

  • Latin uses an alphabet, where each consonant and vowel gets its own letter.

  • Chinese. Logograms. Half width vs full width. English letters are roughly twice as high as they are wide. Chinese characters are roughly square.

  • Arabic is an abjad, only consonants are written and vowels are optionally as diacritics. Additionally, it’s written from right-to-left, so the first letter is the letter on the far right.

  • Devangari is an abugida where each symbol represents a consonant-vowel pair, , vowel notation secondary.

For instance, ‘ch’ is two letters in English and Latin, but considered to be one letter in Czech and Slovak. — http://utf8everywhere.org

# But
str_split("check", boundary("character", locale = "cs_CZ"))
#> [[1]]
#> [1] "c" "h" "e" "c" "k"

This is a problem even with Latin alphabets because many languages use diacritics, glyphs added to the basic alphabet. This is a problem because Unicode provides two ways of representing characters with accents: many common characters have a special codepoint, but others can be built up from individual components.

x <- c("á", "x́")
str_length(x)
#> [1] 1 2
# str_width(x)
str_sub(x, 1, 1)
#> [1] "á" "x"

# stri_width(c("全形", "ab"))
# 0, 1, or 2
# but this assumes no font substitution
cyrillic_a <- "А"
latin_a <- "A"
cyrillic_a == latin_a
#> [1] FALSE
stringi::stri_escape_unicode(cyrillic_a)
#> [1] "\\u0410"
stringi::stri_escape_unicode(latin_a)
#> [1] "A"

18.14.3 Collation rules

coll(): compare strings using standard collation rules. This is useful for doing case insensitive matching. Note that coll() takes a locale parameter that controls which rules are used for comparing characters. Unfortunately different parts of the world use different rules!B oth fixed() and regex() have ignore_case arguments, but they do not allow you to pick the locale: they always use the default locale. You can see what that is with the following code; more on stringi later.

a1 <- "\u00e1"
a2 <- "a\u0301"
c(a1, a2)
#> [1] "á" "á"
a1 == a2
#> [1] FALSE

str_detect(a1, fixed(a2))
#> [1] FALSE
str_detect(a1, coll(a2))
#> [1] TRUE

The downside of coll() is speed; because the rules for recognising which characters are the same are complicated, coll() is relatively slow compared to regex() and fixed().

18.14.4 Upper and lower case

Relatively few writing systems have upper and lower case: Latin, Greek, and Cyrillic, plus a handful of lessor known languages.

Above I used str_to_lower() to change the text to lower case. You can also use str_to_upper() or str_to_title(). However, changing case is more complicated than it might at first appear because different languages have different rules for changing case. You can pick which set of rules to use by specifying a locale:

# Turkish has two i's: with and without a dot, and it
# has a different rule for capitalising them:
str_to_upper(c("i", "ı"))
#> [1] "I" "I"
str_to_upper(c("i", "ı"), locale = "tr")
#> [1] "İ" "I"

The locale is specified as a ISO 639 language code, which is a two or three letter abbreviation. If you don’t already know the code for your language, Wikipedia has a good list, and you can see which are supported with stringi::stri_locale_list(). If you leave the locale blank, it will use English.

The locale also affects case-insensitive matching, which coll(ignore_case = TRUE) which you can control with coll():

i <- c("Iİiı")

str_view_all(i, coll("i", ignore_case = TRUE))
#> <I>İ<i>ı
str_view_all(i, coll("i", ignore_case = TRUE, locale = "tr"))
#> I<İ><i>ı

You can also do case insensitive matching this fixed(ignore_case = TRUE), but this uses a simple approximation which will not work in all cases.

18.14.5 Sorting

Unicode collation algorithm: https://unicode.org/reports/tr10/

Another important operation that’s affected by the locale is sorting. The base R order() and sort() functions sort strings using the current locale. If you want robust behaviour across different computers, you may want to use str_sort() and str_order() which take an additional locale argument.

Can also control the “strength”, which determines how accents are sorted.

str_sort(c("a", "ch", "c", "h"))
#> [1] "a"  "c"  "ch" "h"
str_sort(c("a", "ch", "c", "h"), locale = "cs_CZ")
#> [1] "a"  "c"  "h"  "ch"

TODO: add connection to arrange()