16  Regular expressions

You are reading the work-in-progress second edition of R for Data Science. This chapter is undergoing heavy restructuring and may be confusing or incomplete. You can find the complete first edition at https://r4ds.had.co.nz.

16.1 Introduction

You learned the basics of regular expressions in Chapter 15, but regular expressions are fairly rich language so it’s worth spending some extra time on the details.

The chapter starts by expanding your knowledge of patterns, to cover six important new topics (escaping, anchoring, character classes, shorthand classes, quantifiers, and alternation). Here we’ll focus mostly on the language itself, not the functions that use it. That means we’ll mostly work with toy character vectors, showing the results with str_view() and str_view_all(). You’ll need to take what you learn here and apply it to data frames with tidyr functions or by combining dplyr and stringr functions. We’ll then take what you’ve learned a show a few useful strategies when creating more complex patterns.

Next we’ll talk about the important concepts of “grouping” and “capturing” which give you new ways to extract variables out of strings using tidyr::separate_group(). Grouping also allows you to use back references which allow you do things like match repeated patterns. We’ll finish by discussing the various “flags” that allow you to tweak the operation of regular expressions

16.1.1 Prerequisites

This chapter will use regular expressions as provided by the stringr package.

It’s worth noting that the regular expressions used by stringr are very slightly different to those of base R. That’s because stringr is built on top of the stringi package, which is in turn built on top of the ICU engine, whereas base R functions (like gsub() and grepl()) use either the TRE engine or the PCRE engine. Fortunately, the basics of regular expressions are so well established that you’ll encounter few variations when working with the patterns you’ll learn in this book (and we’ll point them out where important). You only need to be aware of the difference when you start to rely on advanced features like complex Unicode character ranges or special features that use the (?…) syntax. You can learn more about these advanced features in vignette("regular-expressions", package = "stringr"). Another useful reference is https://www.regular-expressions.info/. It’s not R specific, but it covers the most advanced features and explains how regular expressions work under the hood.

16.1.2 Exercises

  1. Explain why each of these strings don’t match a \: "\", "\\", "\\\".

  2. How would you match the sequence "'\?

  3. What patterns will the regular expression \..\..\.. match? How would you represent it as a string?

16.2 Pattern language

You learned the very basics of the regular expression pattern language in Chapter 15, and now its time to dig into more of the details. First, we’ll start with escaping, which allows you to match characters that the pattern language otherwise treats specially. Next you’ll learn about anchors, which allow you to match the start or end of the string. Then you’ll learn about character classes and their shortcuts, which allow you to match any character from a set. We’ll finish up with quantifiers, which control how many times a pattern can match, and alternation, which allows you to match either this or that.

The terms we use here are the technical names for each component. They’re not always the most evocative of their purpose, but it’s very helpful to know the correct terms if you later want to Google for more details.

We’ll concentrate on showing how these patterns work with str_view() and str_view_all() but remember that you can use them with any of the functions that you learned about in Chapter 15, i.e.:

  • str_detect(x, pattern) returns a logical vector the same length as x, indicating whether each element matches (TRUE) or doesn’t match (FALSE) the pattern.
  • str_count(x, pattern) returns the number of times pattern matches in each element of x.
  • str_replace_all(x, pattern, replacement) replaces every instance of pattern with replacement.

16.2.1 Escaping

In Chapter 15, you’ll learned how to match a literal . by using fixed("."). But what if you want to match a literal . as part of a bigger regular expression? You’ll need to use an escape, which tells the regular expression you want it to match exactly, not use its special behavior. Like strings, regexps use the backslash for escaping, so to match a ., you need the regexp \.. Unfortunately this creates a problem. We use strings to represent regular expressions, and \ is also used as an escape symbol in strings. So, as the following example shows, to create the regular expression \. we need the string "\\.".

# To create the regular expression \., we need to use \\.
dot <- "\\."

# But the expression itself only contains one \
str_view(dot)
#> [1] \.

# And this tells R to look for an explicit .
str_view(c("abc", "a.c", "bef"), "a\\.c")
#> [1] abc
#> [2] <a.c>
#> [3] bef

In this book, we’ll write regular expression as \. and strings that represent the regular expression as "\\.".

If \ is used as an escape character in regular expressions, how do you match a literal \? Well you need to escape it, creating the regular expression \\. To create that regular expression, you need to use a string, which also needs to escape \. That means to match a literal \ you need to write "\\\\" — you need four backslashes to match one!

x <- "a\\b"
str_view(x)
#> [1] a\b
str_view(x, "\\\\")
#> [1] a<\>b

Alternatively, you might find it easier to use the raw strings you learned about in Section 15.2.2). That lets you to avoid one layer of escaping:

str_view(x, r"(\\)")
#> [1] a<\>b

The full set of characters with special meanings that need to be escaped is .^$\|*+?{}[](). In general, look at punctuation characters with suspicion; if your regular expression isn’t matching what you think it should, check if you’ve used any of these characters.

16.2.2 Anchors

By default, regular expressions will match any part of a string. If you want to match at the start of end you need to anchor the regular expression using ^ or $.

  • ^ to match the start of the string.
  • $ to match the end of the string.
x <- c("apple", "banana", "pear")
str_view(x, "a")  # match "a" anywhere
#> [1] <a>pple
#> [2] b<a>nana
#> [3] pe<a>r
str_view(x, "^a") # match "a" at start
#> [1] <a>pple
#> [2] banana
#> [3] pear
str_view(x, "a$") # match "a" at end
#> [1] apple
#> [2] banan<a>
#> [3] pear

To remember which is which, try this mnemonic which Hadley learned from Evan Misshula: if you begin with power (^), you end up with money ($). It’s tempting to put $ at the start, because that’s how we write sums of money, but it’s not what regular expressions want.

To force a regular expression to only match the full string, anchor it with both ^ and $:

x <- c("apple pie", "apple", "apple cake")
str_view(x, "apple")
#> [1] <apple> pie
#> [2] <apple>
#> [3] <apple> cake
str_view(x, "^apple$")
#> [1] apple pie
#> [2] <apple>
#> [3] apple cake

You can also match the boundary between words (i.e. the start or end of a word) with \b. This is not that useful in R code, but it can be handy when searching in RStudio. It’s useful to find the name of a function that’s a component of other functions. For example, if to find all uses of sum(), you can search for \bsum\b to avoid matching summarise, summary, rowsum and so on:

x <- c("summary(x)", "summarise(df)", "rowsum(x)", "sum(x)")
str_view(x, "sum")
#> [1] <sum>mary(x)
#> [2] <sum>marise(df)
#> [3] row<sum>(x)
#> [4] <sum>(x)
str_view(x, "\\bsum\\b")
#> [1] summary(x)
#> [2] summarise(df)
#> [3] rowsum(x)
#> [4] <sum>(x)

When used alone these anchors will produce a zero-width match:

str_view_all("abc", c("$", "^", "\\b"))
#> [1] abc<>
#> [2] <>abc
#> [3] <>abc<>

16.2.3 Character classes

A character class, or character set, allows you to match any character in a set. The basic syntax lists each character you want to match inside of [], so [abc] will match a, b, or c. Inside of [] only -, ^, and \ have special meanings:

  • - defines a range. [a-z]: matches any lower case letter and [0-9] matches any number.
  • ^ takes the inverse of the set. [^abc]: matches anything except a, b, or c.
  • \ escapes special characters, so [\^\-\]]: matches ^, -, or ].
str_view_all("abcd12345-!@#%.", c("[abc]", "[a-z]", "[^a-z0-9]"))
#> [1] <a><b><c>d12345-!@#%.
#> [2] <a><b><c><d>12345-!@#%.
#> [3] abcd12345<-><!><@><#><%><.>

# You need an escape to match characters that are otherwise
# special inside of []
str_view_all("a-b-c", "[a\\-c]")
#> [1] <a><->b<-><c>

Remember that regular expressions are case sensitive so if you want to match any lowercase or uppercase letter, you’d need to write [a-zA-Z0-9].

16.2.4 Shorthand character classes

There are a few character classes that are used so commonly that they get their own shortcut. You’ve already seen ., which matches any character apart from a newline. There are three other particularly useful pairs:

  • \d: matches any digit;
    \D: matches anything that isn’t a digit.
  • \s: matches any whitespace (e.g. space, tab, newline);
    \S: matches anything that isn’t whitespace.
  • \w: matches any “word” character, i.e. letters and numbers;
    \W: matches any “non-word” character.

Remember, to create a regular expression containing \d or \s, you’ll need to escape the \ for the string, so you’ll type "\\d" or "\\s". The following code demonstrates the different shortcuts with a selection of letters, numbers, and punctuation characters.

str_view_all("abcd12345!@#%. ", "\\d+")
#> [1] abcd<12345>!@#%.
str_view_all("abcd12345!@#%. ", "\\D+")
#> [1] <abcd>12345<!@#%. >
str_view_all("abcd12345!@#%. ", "\\w+")
#> [1] <abcd12345>!@#%.
str_view_all("abcd12345!@#%. ", "\\W+")
#> [1] abcd12345<!@#%. >
str_view_all("abcd12345!@#%. ", "\\s+")
#> [1] abcd12345!@#%.< >
str_view_all("abcd12345!@#%. ", "\\S+")
#> [1] <abcd12345!@#%.>

16.2.5 Quantifiers

The quantifiers control how many times a pattern matches. In Chapter 15 you learned about ? (0 or 1 matches), + (1 or more matches), and * (0 or more matches). For example, colou?r will match American or British spelling, \d+ will match one or more digits, and \s? will optionally match a single whitespace.

You can also specify the number of matches precisely:

  • {n}: exactly n
  • {n,}: n or more
  • {n,m}: between n and m

The following code shows how this works for a few simple examples using to \b match the start or end of a word.

x <- " x xx xxx xxxx"
str_view_all(x, "\\bx{2}")
#> [1]  x <xx> <xx>x <xx>xx
str_view_all(x, "\\bx{2,}")
#> [1]  x <xx> <xxx> <xxxx>
str_view_all(x, "\\bx{1,3}")
#> [1]  <x> <xx> <xxx> <xxx>x
str_view_all(x, "\\bx{2,3}")
#> [1]  x <xx> <xxx> <xxx>x

16.2.6 Alternation

You can use alternation to pick between one or more alternative patterns. Here are a few examples:

  • Match apple, pear, or banana: apple|pear|banana.
  • Match three letters or two digits: \w{3}|\d{2}.

16.2.7 Parentheses and operator precedence

What does ab+ match? Does it match “a” followed by one or more “b”s, or does it match “ab” repeated any number of times? What does ^a|b$ match? Does it match the complete string a or the complete string b, or does it match a string starting with a or a string starting with “b”? The answer to these questions is determined by operator precedence, similar to the PEMDAS or BEDMAS rules you might have learned in school for what a + b * c.

You already know that a + b * c is equivalent to a + (b * c) not (a + b) * c because * has high precedence and + has lower precedence: you compute * before +. In regular expressions, quantifiers have high precedence and alternation has low precedence. That means ab+ is equivalent to a(b+), and ^a|b$ is equivalent to (^a)|(b$). Just like with algebra, you can use parentheses to override the usual order (because they have the highest precedence of all).

Technically the escape, character classes, and parentheses are all operators that also have precedence. But these tend to be less likely to cause confusion because they mostly behave how you expect: it’s unlikely that you’d think that \(s|d) would mean (\s)|(\d).

16.2.8 Exercises

  1. How would you match the literal string "$^$"?

  2. Given the corpus of common words in stringr::words, create regular expressions that find all words that:

    1. Start with “y”.
    2. Don’t start with “y”.
    3. End with “x”.
    4. Are exactly three letters long. (Don’t cheat by using str_length()!)
    5. Have seven letters or more.

    Since words is long, you might want to use the match argument to str_view() to show only the matching or non-matching words.

  3. Create regular expressions that match the British or American spellings of the following words: grey/gray, modelling/modeling, summarize/summarise, aluminium/aluminum, defence/defense, analog/analogue, center/centre, sceptic/skeptic, aeroplane/airplane, arse/ass, doughnut/donut.

  4. What strings will $a match?

  5. Create a regular expression that will match telephone numbers as commonly written in your country.

  6. Write the equivalents of ?, +, * in {m,n} form.

  7. Describe in words what these regular expressions match: (read carefully to see if each entry is a regular expression or a string that defines a regular expression.)

    1. ^.*$
    2. "\\{.+\\}"
    3. \d{4}-\d{2}-\d{2}
    4. "\\\\{4}"
  8. Solve the beginner regexp crosswords at https://regexcrossword.com/challenges/beginner.

16.3 Practice

To put these ideas in practice we’ll solve a few semi-authentic problems using the words and sentences datasets built into stringr. words is a list of common English words and sentences is a set of simple sentences originally used for testing voice transmission.

str_view(head(words))
#> [1] a
#> [2] able
#> [3] about
#> [4] absolute
#> [5] accept
#> [6] account
str_view(head(sentences))
#> [1] The birch canoe slid on the smooth planks.
#> [2] Glue the sheet to the dark blue background.
#> [3] It's easy to tell the depth of a well.
#> [4] These days a chicken leg is a rare dish.
#> [5] Rice is often served in round bowls.
#> [6] The juice of lemons makes fine punch.

The following three sections help you practice the components of a pattern by discussing three general techniques: checking you work by creating simple positive and negative controls, combining regular expressions with Boolean algebra, and creating complex patterns using string manipulation.

16.3.1 Check your work

First, let’s find all sentences that start with “The”. Using the ^ anchor alone is not enough:

str_view(sentences, "^The", match = TRUE)
#>  [1] <The> birch canoe slid on the smooth planks.
#>  [4] <The>se days a chicken leg is a rare dish.
#>  [6] <The> juice of lemons makes fine punch.
#>  [7] <The> box was thrown beside the parked truck.
#>  [8] <The> hogs were fed chopped corn and garbage.
#> [11] <The> boy was there when the sun rose.
#> [13] <The> source of the huge river is the clear spring.
#> [18] <The> soft cushion broke the man's fall.
#> [19] <The> salt breeze came across from the sea.
#> [20] <The> girl at the booth sold fifty bonds.
#> [21] <The> small pup gnawed a hole in the sock.
#> [22] <The> fish twisted and turned on the bent hook.
#> [24] <The> swan dive was far short of perfect.
#> [25] <The> beauty of the view stunned the young boy.
#> [28] <The> colt reared and threw the tall rider.
#> [36] <The> wrist was badly strained and hung limp.
#> [37] <The> stray cat gave birth to kittens.
#> [38] <The> young girl gave no clear response.
#> [39] <The> meal was cooked before the bell rang.
#> [42] <The> ship was torn apart on the sharp reef.
#> ... and 264 more

Because it all matches sentences starting with They or Those. We need to make sure that the “e” is the last letter in the word, which we can do by adding adding a word boundary:

str_view(sentences, "^The\\b", match = TRUE)
#>  [1] <The> birch canoe slid on the smooth planks.
#>  [6] <The> juice of lemons makes fine punch.
#>  [7] <The> box was thrown beside the parked truck.
#>  [8] <The> hogs were fed chopped corn and garbage.
#> [11] <The> boy was there when the sun rose.
#> [13] <The> source of the huge river is the clear spring.
#> [18] <The> soft cushion broke the man's fall.
#> [19] <The> salt breeze came across from the sea.
#> [20] <The> girl at the booth sold fifty bonds.
#> [21] <The> small pup gnawed a hole in the sock.
#> [22] <The> fish twisted and turned on the bent hook.
#> [24] <The> swan dive was far short of perfect.
#> [25] <The> beauty of the view stunned the young boy.
#> [28] <The> colt reared and threw the tall rider.
#> [36] <The> wrist was badly strained and hung limp.
#> [37] <The> stray cat gave birth to kittens.
#> [38] <The> young girl gave no clear response.
#> [39] <The> meal was cooked before the bell rang.
#> [42] <The> ship was torn apart on the sharp reef.
#> [44] <The> wide road shimmered in the hot sun.
#> ... and 242 more

What about finding all sentences that begin with a pronoun?

str_view(sentences, "^She|He|It|They\\b", match = TRUE)
#>   [3] <It>'s easy to tell the depth of a well.
#>  [15] <He>lp the woman get back to her feet.
#>  [27] <He>r purse was full of useless trash.
#>  [29] <It> snowed, rained, and hailed the same morning.
#>  [63] <He> ran half way to the hardware store.
#>  [90] <He> lay prone and hardly moved a limb.
#> [116] <He> ordered peach pie with ice cream.
#> [118] <He>mp is a weed found in parts of the tropics.
#> [127] <It> caught its hind paw in a rusty trap.
#> [132] <He> said the same phrase thirty times.
#> [153] <He> broke a new shoelace that day.
#> [159] <She> sewed the torn coat quite neatly.
#> [168] <He> knew the skill of the great young actress.
#> [179] <They> felt gay when the ship arrived in port.
#> [189] <He> carved a head from the round block of marble.
#> [190] <She> has st smart way of wearing clothes.
#> [199] <They> could laugh although they were sad.
#> [209] <He> wrote his last novel there at the inn.
#> [217] <It> is hard to erase blue or red ink.
#> [225] <They> took the axe and the saw to the forest.
#> ... and 43 more

A quick inspection of the results shows that we’re getting some spurious matches. That’s because we’ve forgotten to use parentheses:

str_view(sentences, "^(She|He|It|They)\\b", match = TRUE)
#>   [3] <It>'s easy to tell the depth of a well.
#>  [29] <It> snowed, rained, and hailed the same morning.
#>  [63] <He> ran half way to the hardware store.
#>  [90] <He> lay prone and hardly moved a limb.
#> [116] <He> ordered peach pie with ice cream.
#> [127] <It> caught its hind paw in a rusty trap.
#> [132] <He> said the same phrase thirty times.
#> [153] <He> broke a new shoelace that day.
#> [159] <She> sewed the torn coat quite neatly.
#> [168] <He> knew the skill of the great young actress.
#> [179] <They> felt gay when the ship arrived in port.
#> [189] <He> carved a head from the round block of marble.
#> [190] <She> has st smart way of wearing clothes.
#> [199] <They> could laugh although they were sad.
#> [209] <He> wrote his last novel there at the inn.
#> [217] <It> is hard to erase blue or red ink.
#> [225] <They> took the axe and the saw to the forest.
#> [232] <They> are pushed back each time they attack.
#> [233] <He> broke his ties with groups of former friends.
#> [234] <They> floated on the raft to sun their white backs.
#> ... and 37 more

You might wonder how you might spot such a mistake if it didn’t occur in the first few matches. A good technique is to create a few positive and negative matches and use them to test that you pattern works as expected.

pos <- c("He is a boy", "She had a good time")
neg <- c("Shells come from the sea", "Hadley said 'It's a great day'")

pattern <- "^(She|He|It|They)\\b"
str_detect(pos, pattern)
#> [1] TRUE TRUE
str_detect(neg, pattern)
#> [1] FALSE FALSE

It’s typically much easier to come up with positive examples than negative examples, because it takes some time until you’re good enough with regular expressions to predict where your weaknesses are. Nevertheless they’re still useful; even if you don’t get them correct right away, you can slowly accumulate them as you work on your problem. If you later get more into programming and learn about unit tests, you can then turn these examples into automated tests that ensure you never make the same mistake twice.)

16.3.2 Boolean operations

Imagine we want to find words that only contain consonants. One technique is to create a character class that contains all letters except for the vowels ([^aeiou]), then allow that to match any number of letters ([^aeiou]+), then force it to match the whole string by anchoring to the beginning and the end (^[^aeiou]+$):

str_view(words, "^[^aeiou]+$", match = TRUE)
#> [123] <by>
#> [249] <dry>
#> [328] <fly>
#> [538] <mrs>
#> [895] <try>
#> [952] <why>

But we can make this problem a bit easier by flipping the problem around. Instead of looking for words that contain only consonants, we could look for words that don’t contain any vowels:

words[!str_detect(words, "[aeiou]")]
#> [1] "by"  "dry" "fly" "mrs" "try" "why"

This is a useful technique whenever you’re dealing with logical combinations, particularly those involving “and” or “not”. For example, imagine if you want to find all words that contain “a” and “b”. There’s no “and” operator built in to regular expressions so we have to tackle it by looking for all words that contain an “a” followed by a “b”, or a “b” followed by an “a”:

words[str_detect(words, "a.*b|b.*a")]
#>  [1] "able"      "about"     "absolute"  "available" "baby"      "back"     
#>  [7] "bad"       "bag"       "balance"   "ball"      "bank"      "bar"      
#> [13] "base"      "basis"     "bear"      "beat"      "beauty"    "because"  
#> [19] "black"     "board"     "boat"      "break"     "brilliant" "britain"  
#> [25] "debate"    "husband"   "labour"    "maybe"     "probable"  "table"

It’s simpler to combine the results of two calls to str_detect():

words[str_detect(words, "a") & str_detect(words, "b")]
#>  [1] "able"      "about"     "absolute"  "available" "baby"      "back"     
#>  [7] "bad"       "bag"       "balance"   "ball"      "bank"      "bar"      
#> [13] "base"      "basis"     "bear"      "beat"      "beauty"    "because"  
#> [19] "black"     "board"     "boat"      "break"     "brilliant" "britain"  
#> [25] "debate"    "husband"   "labour"    "maybe"     "probable"  "table"

What if we wanted to see if there was a word that contains all vowels? If we did it with patterns we’d need to generate 5! (120) different patterns:

words[str_detect(words, "a.*e.*i.*o.*u")]
#> character(0)
# ...
words[str_detect(words, "u.*o.*i.*e.*a")]
#> character(0)

It’s much simpler to combine six calls to str_detect():

words[
  str_detect(words, "a") &
  str_detect(words, "e") &
  str_detect(words, "i") &
  str_detect(words, "o") &
  str_detect(words, "u")
]
#> character(0)

In general, if you get stuck trying to create a single regexp that solves your problem, take a step back and think if you could break the problem down into smaller pieces, solving each challenge before moving onto the next one.

16.3.3 Creating a pattern with code

What if we wanted to find all sentences that mention a color? The basic idea is simple: we just combine alternation with word boundaries.

str_view(sentences, "\\b(red|green|blue)\\b", match = TRUE)
#>   [2] Glue the sheet to the dark <blue> background.
#>  [26] Two <blue> fish swam in the tank.
#>  [92] A wisp of cloud hung in the <blue> air.
#> [148] The spot on the blotter was made by <green> ink.
#> [160] The sofa cushion is <red> and of light weight.
#> [174] The sky that morning was clear and bright <blue>.
#> [204] A <blue> crane is a tall wading bird.
#> [217] It is hard to erase <blue> or red ink.
#> [224] The lamp shone with a steady <green> flame.
#> [247] The box is held by a bright <red> snapper.
#> [256] The houses are built of <red> clay bricks.
#> [274] The <red> tape bound the smuggled food.
#> [288] Hedge apples may stain your hands <green>.
#> [302] The plant grew large and <green> in the window.
#> [330] Bathe and relax in the cool <green> grass.
#> [368] The lake sparkled in the <red> hot sun.
#> [372] Mark the spot with a sign painted <red>.
#> [452] The couch cover and hall drapes were <blue>.
#> [491] A man in a <blue> sweater sat at the desk.
#> [551] The small <red> neon lamp went out.
#> ... and 6 more

But it would be tedious to construct this pattern by hand. Wouldn’t it be nice if we could store the colours in a vector?

rgb <- c("red", "green", "blue")

Well, we can! We’d just need to create the pattern from the vector using str_c() and str_flatten():

str_c("\\b(", str_flatten(rgb, "|"), ")\\b")
#> [1] "\\b(red|green|blue)\\b"

We could make this pattern more comprehensive if we had a good list of colors. One place we could start from is the list of built-in colours that R can use for plots:

colors()[1:27]
#>  [1] "white"          "aliceblue"      "antiquewhite"   "antiquewhite1" 
#>  [5] "antiquewhite2"  "antiquewhite3"  "antiquewhite4"  "aquamarine"    
#>  [9] "aquamarine1"    "aquamarine2"    "aquamarine3"    "aquamarine4"   
#> [13] "azure"          "azure1"         "azure2"         "azure3"        
#> [17] "azure4"         "beige"          "bisque"         "bisque1"       
#> [21] "bisque2"        "bisque3"        "bisque4"        "black"         
#> [25] "blanchedalmond" "blue"           "blue1"

But first lets element the numbered variants:

cols <- colors()
cols <- cols[!str_detect(cols, "\\d")]
cols[1:27]
#>  [1] "white"          "aliceblue"      "antiquewhite"   "aquamarine"    
#>  [5] "azure"          "beige"          "bisque"         "black"         
#>  [9] "blanchedalmond" "blue"           "blueviolet"     "brown"         
#> [13] "burlywood"      "cadetblue"      "chartreuse"     "chocolate"     
#> [17] "coral"          "cornflowerblue" "cornsilk"       "cyan"          
#> [21] "darkblue"       "darkcyan"       "darkgoldenrod"  "darkgray"      
#> [25] "darkgreen"      "darkgrey"       "darkkhaki"

Then we can turn this into one giant pattern:

pattern <- str_c("\\b(", str_flatten(cols, "|"), ")\\b")
str_view(sentences, pattern, match = TRUE)
#>   [2] Glue the sheet to the dark <blue> background.
#>  [12] A rod is used to catch <pink> salmon.
#>  [26] Two <blue> fish swam in the tank.
#>  [66] Cars and buses stalled in <snow> drifts.
#>  [92] A wisp of cloud hung in the <blue> air.
#> [112] Leaves turn <brown> and yellow in the fall.
#> [148] The spot on the blotter was made by <green> ink.
#> [149] Mud was spattered on the front of his <white> shirt.
#> [160] The sofa cushion is <red> and of light weight.
#> [167] The office paint was a dull sad <tan>.
#> [174] The sky that morning was clear and bright <blue>.
#> [201] The <brown> house was on fire to the attic.
#> [204] A <blue> crane is a tall wading bird.
#> [217] It is hard to erase <blue> or red ink.
#> [221] A pencil with <black> lead writes best.
#> [224] The lamp shone with a steady <green> flame.
#> [230] Slash the <gold> cloth into fine ribbons.
#> [234] They floated on the raft to sun their <white> backs.
#> [246] The birch looked stark <white> and lonesome.
#> [247] The box is held by a bright <red> snapper.
#> ... and 43 more

In this example cols only contains numbers and letters so you don’t need to worry about metacharacters. But in general, when creating patterns from existing strings it’s good practice to run through str_escape() which will automatically add \ in front of otherwise special characters.

16.3.4 Exercises

  1. Construct patterns to find evidence for and against the rule “i before e except after c”?
  2. colors() contains a number of modifiers like “lightgray” and “darkblue”. How could you automatically identify these modifiers? (Think about how you might detect and removed what is being modified).
  3. Create a regular expression that finds any use of base R dataset. You can get a list of these datasets via a special use of the data() function: data(package = "datasets")$results[, "Item"]. Note that a number of old datasets are individual vectors; these contain the name of the grouping “data frame” in parentheses, so you’ll need to also strip these off.

16.4 Grouping and capturing

Like in algebra, parentheses are an important tool for controlling the order in which pattern operations are applied. But they also have an important additional effect: they create capturing groups that allow you to use to sub-components of the match. There are three main ways you can use them:

  • To match a repeated pattern.
  • To include a matched pattern in the replacement.
  • To extract individual components of the match.

If needed, there’s also a special form of parentheses that only affect operator precedence without creating capturing a group. All of these are these described below.

16.4.1 Matching a repeated pattern

You can refer back to previously matched text inside parentheses by using back reference. Back references are usually numbered: \1 refers to the match contained in the first parenthesis, \2 in the second parenthesis, and so on. For example, the following pattern finds all fruits that have a repeated pair of letters:

str_view(fruit, "(..)\\1", match = TRUE)
#>  [4] b<anan>a
#> [20] <coco>nut
#> [22] <cucu>mber
#> [41] <juju>be
#> [56] <papa>ya
#> [73] s<alal> berry

And this one finds all words that start and end with the same pair of letters:

str_view(words, "^(..).*\\1$", match = TRUE)
#> [152] <church>
#> [217] <decide>
#> [617] <photograph>
#> [699] <require>
#> [739] <sense>

16.4.2 Replacing with the matched pattern

You can also use back references when replacing with str_replace() and str_replace_all(). The following code will switch the order of the second and third words:

sentences |> 
  str_replace("(\\w+) (\\w+) (\\w+)", "\\1 \\3 \\2") |> 
  head(5)
#> [1] "The canoe birch slid on the smooth planks." 
#> [2] "Glue sheet the to the dark blue background."
#> [3] "It's to easy tell the depth of a well."     
#> [4] "These a days chicken leg is a rare dish."   
#> [5] "Rice often is served in round bowls."

You’ll sometimes see people using str_replace() to extract a single match:

pattern <- "^.*the ([^ .,]+).*$"
sentences |> 
  str_subset(pattern) |> 
  str_replace(pattern, "\\1") |> 
  head(10)
#>  [1] "smooth"  "dark"    "depth"   "parked"  "sun"     "clear"   "ball"   
#>  [8] "woman"   "evening" "man's"

But you’re generally better off using str_match() or tidyr::separate_groups(), which you’ll learn about next.

16.4.3 Extracting groups

stringr provides a lower-level function for extract matches called str_match(). But it returns a matrix, so isn’t as easy to work with:

sentences |> 
  str_match("the (\\w+) (\\w+)") |> 
  head()
#>      [,1]                [,2]     [,3]    
#> [1,] "the smooth planks" "smooth" "planks"
#> [2,] "the sheet to"      "sheet"  "to"    
#> [3,] "the depth of"      "depth"  "of"    
#> [4,] NA                  NA       NA      
#> [5,] NA                  NA       NA      
#> [6,] NA                  NA       NA

Instead, we recommend using tidyr’s separate_groups() which creates a column for each capturing group.

16.4.4 Named groups

If you have many groups, referring to them by position can get confusing. It’s possible to give them a name with (?<name>…). You can refer to it with \k<name>.

str_view(words, "^(?<first>.).*\\k<first>$", match = TRUE)
#>  [36] <america>
#>  [49] <area>
#> [209] <dad>
#> [213] <dead>
#> [223] <depend>
#> [258] <educate>
#> [266] <else>
#> [268] <encourage>
#> [270] <engine>
#> [278] <europe>
#> [283] <evidence>
#> [285] <example>
#> [287] <excuse>
#> [288] <exercise>
#> [291] <expense>
#> [292] <experience>
#> [296] <eye>
#> [386] <health>
#> [394] <high>
#> [450] <knock>
#> ... and 16 more

This verbosity is a good fit with comments = TRUE:

pattern <- regex(
  r"(
    ^           # start at the beginning of the string
    (?<first>.) # and match the <first> letter
    .*          # then match any other letters
    \k<first>$  # ensuring the last letter is the same as the <first>
  )", 
  comments = TRUE
)

You can also use named groups as an alternative to the col_names argument to tidyr::separate_groups().

16.4.5 Non-capturing groups

Occasionally, you’ll want to use parentheses without creating matching groups. You can create a non-capturing group with (?:).

x <- c("a gray cat", "a grey dog")
str_match(x, "(gr(e|a)y)")
#>      [,1]   [,2]   [,3]
#> [1,] "gray" "gray" "a" 
#> [2,] "grey" "grey" "e"
str_match(x, "(gr(?:e|a)y)")
#>      [,1]   [,2]  
#> [1,] "gray" "gray"
#> [2,] "grey" "grey"

Typically, however, you’ll find it easier to just ignore that result by setting the col_name to NA:

16.4.6 Exercises

  1. Describe, in words, what these expressions will match:

    1. (.)\1\1
    2. "(.)(.)\\2\\1"
    3. (..)\1
    4. "(.).\\1.\\1"
    5. "(.)(.)(.).*\\3\\2\\1"
  2. Construct regular expressions to match words that:

    1. Who’s first letter is the same as the last letter, and the second letter is the same as the second to last letter.
    2. Contain one letter repeated in at least three places (e.g. “eleven” contains three “e”s.)

16.5 Flags

The are a number of settings, often called flags in other programming languages, that you can use to control some of the details of the regex. In stringr, you can use these by wrapping the pattern in a call to regex():

# The regular call:
str_view(fruit, "nana")
# is shorthand for
str_view(fruit, regex("nana"))

The most useful flag is probably ignore_case = TRUE because it allows characters to match either their uppercase or lowercase forms:

bananas <- c("banana", "Banana", "BANANA")
str_view(bananas, "banana")
#> [1] <banana>
#> [2] Banana
#> [3] BANANA
str_view(bananas, regex("banana", ignore_case = TRUE))
#> [1] <banana>
#> [2] Banana
#> [3] BANANA

If you’re doing a lot of work with multiline strings (i.e. strings that contain \n), multiline and dotall can also be useful. dotall = TRUE allows . to match everything, including \n:

x <- "Line 1\nLine 2\nLine 3"
str_view_all(x, ".L")
#> [1] Line 1
#>     Line 2
#>     Line 3
str_view_all(x, regex(".L", dotall = TRUE))
#> [1] Line 1
#>     Line 2
#>     Line 3

And multiline = TRUE allows ^ and $ to match the start and end of each line rather than the start and end of the complete string:

x <- "Line 1\nLine 2\nLine 3"
str_view_all(x, "^Line")
#> [1] <Line> 1
#>     Line 2
#>     Line 3
str_view_all(x, regex("^Line", multiline = TRUE))
#> [1] <Line> 1
#>     Line 2
#>     Line 3

Finally, if you’re writing a complicated regular expression and you’re worried you might not understand it in the future, comments = TRUE can be extremely useful. It allows you to use comments and whitespace to make complex regular expressions more understandable. Spaces and new lines are ignored, as is everything after #. (Note that we use a raw string here to minimize the number of escapes needed.)

phone <- regex(r"(
  \(?     # optional opening parens
  (\d{3}) # area code
  [)\ -]? # optional closing parens, space, or dash
  (\d{3}) # another three numbers
  [\ -]?  # optional space or dash
  (\d{3}) # three more numbers
  )", comments = TRUE)

str_match("514-791-8141", phone)
#>      [,1]          [,2]  [,3]  [,4] 
#> [1,] "514-791-814" "514" "791" "814"

If you’re using comments and want to match a space, newline, or #, you’ll need to escape it:

str_view("x x #", regex("x #", comments = TRUE))
#> [1] x <x #>
str_view("x x #", regex(r"(x\ \#)", comments = TRUE))
#> [1] x <x #>