Introduction to R, Class 3: Data manipulation with tidyverse
In the last section, we imported a clinical cancer dataset and learned to manipulate it using base R (tools included in every R installation).
In this session, we’ll continue to work with the same dataset, but will introduce a set of tools specifically designed for data science in R. By the end of this session, you should be able to:
- install and load packages
- use tidyverse tools to import data and access rows/columns
- combine commands using pipes
- transform and summarize data
Installing and loading packages
Please ensure RStudio is open with your project directory path (e.g.,
~/Desktop/intro_r) listed at the top of your Console. If you do not
see the path to your project directory, go to
File -> Open Project and
navigate to the location of your project directory. Alternatively, using
your operating system’s file browser, double click on the
Create a new R script called
class3.R, and add
# Introduction to R:
Class 3 as a title.
For this lesson, we’ll be working with a group of R packages called
tidyverse. A package is a group of related functions that help you
accomplish particular tasks.
packages have been designed specifically to support tasks related to
data science, such as data manipulation, filtering, and visualization.
The first thing we need to do is install the software:
# install package install.packages("tidyverse")
A few notes about installing packages:
- You only need to perform this installation once per computer, or when updating R or the package.
- If you see red text output in the Console during this installation, don’t be alarmed: this doens’t necessarily indicate a problem. You are seeing a report of the various pieces of software being downloaded and installed.
- If prompted, you should install all packages (say yes or all), as well as yes to compiling any packages
- When the installation is complete (this may take several minutes),
you’ll see the command prompt (
>) in your Console.
Once you have the software installed, you’ll need to load it:
# load library/package library(tidyverse)
## ── Attaching packages ───────────────────────────────────────────────────────────────── tidyverse 1.3.0 ── ## ✓ ggplot2 3.3.2 ✓ purrr 0.3.4 ## ✓ tibble 3.0.3 ✓ dplyr 1.0.1 ## ✓ tidyr 1.1.1 ✓ stringr 1.4.0 ## ✓ readr 1.3.1 ✓ forcats 0.5.0 ## ── Conflicts ──────────────────────────────────────────────────────────────────── tidyverse_conflicts() ── ## x dplyr::filter() masks stats::filter() ## x dplyr::lag() masks stats::lag()
Loading packages is similar to opening a software application on your computer; it makes a previously installed set of software available for use. A few notes about loading packages:
- You’ll need to load packages every time you open RStudio (or R restarts)
tidyverseloads a collection of packages; these are listed under “Attaching packages”
- There are many other packages included as dependencies. If some of
them did not install successfully, you will receive an error at this
step. For this lesson, you can try
library(dplyr), and ask your instructor for help later.
- The section in the output above referencing “Conflicts” shows you which functions you just loaded have names identical to packages you already have loaded (in base R). This shouldn’t affect the code we write in this lesson, though it’s useful to know the double colon syntax (::) allows you to reference functions in a different package with same name.
You can check to make sure the new package we’ll be using is available
?select in the Console, or by searching for that function
in the help panel. You can also look in the “Packages” tab in the same
panel. If the package (in this case, either
present in the list, it’s installed. If the box next to the package name
is checked, it’s loaded. In this lesson, if you receive an error saying
a function isn’t available or recognized, check to make sure the package
Selecting columns and rows
The first task we’ll undertake with our newly installed
tools is importing our data:
# import data clinical <- read_csv("data/clinical.csv")
## Parsed with column specification: ## cols( ## .default = col_character(), ## age_at_diagnosis = col_double(), ## days_to_death = col_double(), ## days_to_birth = col_double(), ## days_to_last_follow_up = col_double(), ## cigarettes_per_day = col_double(), ## years_smoked = col_double(), ## year_of_birth = col_double(), ## year_of_death = col_double() ## ) ## See spec(...) for full column specifications.
Note that this function looks similar to what we used in the last lesson
read.csv), but the underscore replacing the dot means it’s actually a
different function. In fact, this is the data import function from
tidyverse. The output provided by this function indicates a few key
differences with our import yesterday.
The data import execution includes a description of how each variable
(column) is interpreted. In our data’s case, the numeric data are
col_double and the character data are
col_character (not factors!).
We can explore these differences further:
# inspect object str(clinical)
## tibble [6,832 × 20] (S3: spec_tbl_df/tbl_df/tbl/data.frame) ## $ primary_diagnosis : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ... ## $ tumor_stage : chr [1:6832] "stage ia" "stage ib" "stage ib" "stage ia" ... ## $ age_at_diagnosis : num [1:6832] 24477 26615 28171 27154 29827 ... ## $ vital_status : chr [1:6832] "dead" "dead" "dead" "alive" ... ## $ morphology : chr [1:6832] "8070/3" "8070/3" "8070/3" "8083/3" ... ## $ days_to_death : num [1:6832] 371 136 2304 NA 146 ... ## $ state : chr [1:6832] "live" "live" "live" "live" ... ## $ tissue_or_organ_of_origin : chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ... ## $ days_to_birth : num [1:6832] -24477 -26615 -28171 -27154 -29827 ... ## $ site_of_resection_or_biopsy: chr [1:6832] "C34.1" "C34.1" "C34.3" "C34.1" ... ## $ days_to_last_follow_up : num [1:6832] NA NA 2099 3747 NA ... ## $ cigarettes_per_day : num [1:6832] 10.96 2.19 1.64 1.1 NA ... ## $ years_smoked : num [1:6832] NA NA NA NA NA NA NA NA NA NA ... ## $ gender : chr [1:6832] "male" "male" "female" "male" ... ## $ year_of_birth : num [1:6832] 1936 1931 1927 1930 1923 ... ## $ race : chr [1:6832] "white" "asian" "white" "white" ... ## $ ethnicity : chr [1:6832] "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" "not hispanic or latino" ... ## $ year_of_death : num [1:6832] 2004 2003 NA NA 2004 ... ## $ bcr_patient_barcode : chr [1:6832] "TCGA-18-3406" "TCGA-18-3407" "TCGA-18-3408" "TCGA-18-3409" ... ## $ disease : chr [1:6832] "LUSC" "LUSC" "LUSC" "LUSC" ... ## - attr(*, "spec")= ## .. cols( ## .. primary_diagnosis = col_character(), ## .. tumor_stage = col_character(), ## .. age_at_diagnosis = col_double(), ## .. vital_status = col_character(), ## .. morphology = col_character(), ## .. days_to_death = col_double(), ## .. state = col_character(), ## .. tissue_or_organ_of_origin = col_character(), ## .. days_to_birth = col_double(), ## .. site_of_resection_or_biopsy = col_character(), ## .. days_to_last_follow_up = col_double(), ## .. cigarettes_per_day = col_double(), ## .. years_smoked = col_double(), ## .. gender = col_character(), ## .. year_of_birth = col_double(), ## .. race = col_character(), ## .. ethnicity = col_character(), ## .. year_of_death = col_double(), ## .. bcr_patient_barcode = col_character(), ## .. disease = col_character() ## .. )
You may notice the presence of
tbl and related labels in the classes
for this object.
tbl stands for
tibble, which is a type of data frame
with specific constraints to ensure better data handling.
If you preview the dataset, it will look the same, and we can interact
with the data in the same way. These assumptions about the data mesh
nicely with the other tools in the
Now that our data are imported, we can explore the
for extracting parts of the dataset.
First, we can explore selecting certain columns by name:
# selecting columns with tidyverse (dplyr) sel_columns <- select(clinical, tumor_stage, ethnicity, disease)
The syntax for the
select function is to specify the dataset first,
then the names of each of the columns you would like to retain in the
output object. If we look at the object, we’ll see it has only three
columns but all rows.
You’ll note that the column headers don’t require quotation marks; this
is a shortcut programmed into
As with base R functions, we can also select a range of columns:
# select range of columns sel_columns2 <- select(clinical, tumor_stage:vital_status)
In addition to these approaches, we can also use other helper functions
for selecting columns:
are examples that assist in extracting columns with headers that meet
certain conditions. For example, using
starts_with(tumor) in place of
the column names will give you all columns that start with the word
We can use a separate function to extract rows that meet particular conditions:
# select rows conditionally: keep only lung cancer cases filtered_rows <- filter(clinical, disease == "LUSC")
The syntax here is similar to
select, and the conditional filters can
be applied in similarly to base R functions.
Create a new object from clinical called race_disease that includes only the race, > ethnicity, and disease columns.
Create a new object from race_disease called race_BRCA that includes only BRCA (from
The last challenges used an intermediate object to obtain an object with two subsetting methods applied. It’s common in data science to apply more than one requirement for extracting data. If you want to avoid creating an intermediate object, you could nest one command inside the other:
# same task as challenges, but nested commands race_BRCA2 <- select(filter(clinical, disease == "BRCA"), race, ethnicity, disease)
In this case,
filter(clinical, disease == "BRCA") becomes the input
While this is a common approach, especially in base R, it can be difficult for us as coders to read and interpret the code.
One of the most useful features of
tidyverse is its inclusion of a
programming method called pipes. This approach can be found in many
programming languages, in part because of its utility: a pipe sends the
output from the lefthand side of the symbol as the input for the
righthand side. In R, pipes are represented as
We can use pipes to connect the same two data extraction tasks:
# same task as above, but with pipes piped <- clinical %>% select(race, ethnicity, disease) %>% filter(disease == "BRCA")
The command above starts by naming the object that will result from this
assignment. The dataset is named as the first input. Because executing
the name of an object sends the object contents as output, this means
the second line receives the object as input. The output from the
select line is sent as input to the
filter line. This effectively
demonstrates how pipes can be used to connect multiple commands
Now that we are running code in chunks that span multiple lines, you can see one of the other nice features of RStudio: your cursor can be placed on any line of the multi-line chunk when you execute, and the entire set of code will run together.
These examples also help highlight the importance of style and convention in code formatting. After the first line, the code is indented. While this isn’t necessary for the code to work, it does make it a lot easier to read and understand the code.
Let’s take a look at another example of piped commands:
# extract race, ethinicity, and disease from cases born prior to 1930 piped2 <- clinical %>% filter(year_of_birth < 1930) %>% select(race, ethnicity, disease)
In the code above, we’re applying a mathematical condition to find
specific rows, and the selecting certain columns. Does the order of
commands differ? We can switch the order of the
lines to see:
piped3 <- clinical %>% select(race, ethnicity, disease) %>% filter(year_of_birth < 1930)
The code above should give you an error, because in this case, the order
does matter! The output from the second line does not include the
year_of_birth column, so R is unable to apply the filter in the third
Use pipes to extract the columns
year_of_birthfrom the object
clinicalfor only living patients (vital_status) who have smoked fewer than 1
This lesson so far has mostly shown new ways of accomplishing the same
tasks we learned in the last lesson.
tidyverse includes much more
functionality, however, including the ability to
Common tasks for which
mutate is useful include unit conversions,
transformation, and creating ratios from among existing columns.
We can use this function to convert the
days_to_death column to years:
# convert days to years clinical_years <- clinical %>% mutate(years_to_death = days_to_death / 365)
The actual conversion works by providing a formula (
/ 365) and the name of the new column (
years_to_death). If you
inspect the resulting object, you’ll see
years_to_death added as a new
column at the end of the table.
mutateworks by retaining all previous columns and creating new columns as per the formula specified.
transmute, which drops the existing columns used to calculcate the new columns.
We can use
mutate to perform multiple conversions at once:
# convert days to year and months at same time, and we don't always need to assign to object clinical %>% mutate(years_to_death = days_to_death / 365, months_to_death = days_to_death / 30) %>% glimpse() # preview data output
## Rows: 6,832 ## Columns: 22 ## $ primary_diagnosis <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.… ## $ tumor_stage <chr> "stage ia", "stage ib", "stage ib", "stag… ## $ age_at_diagnosis <dbl> 24477, 26615, 28171, 27154, 29827, 23370,… ## $ vital_status <chr> "dead", "dead", "dead", "alive", "dead", … ## $ morphology <chr> "8070/3", "8070/3", "8070/3", "8083/3", "… ## $ days_to_death <dbl> 371, 136, 2304, NA, 146, NA, 345, 716, 28… ## $ state <chr> "live", "live", "live", "live", "live", "… ## $ tissue_or_organ_of_origin <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.… ## $ days_to_birth <dbl> -24477, -26615, -28171, -27154, -29827, -… ## $ site_of_resection_or_biopsy <chr> "C34.1", "C34.1", "C34.3", "C34.1", "C34.… ## $ days_to_last_follow_up <dbl> NA, NA, 2099, 3747, NA, 3576, NA, NA, 181… ## $ cigarettes_per_day <dbl> 10.9589041, 2.1917808, 1.6438356, 1.09589… ## $ years_smoked <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 2… ## $ gender <chr> "male", "male", "female", "male", "male",… ## $ year_of_birth <dbl> 1936, 1931, 1927, 1930, 1923, 1942, 1953,… ## $ race <chr> "white", "asian", "white", "white", "not … ## $ ethnicity <chr> "not hispanic or latino", "not hispanic o… ## $ year_of_death <dbl> 2004, 2003, NA, NA, 2004, NA, 2005, 2006,… ## $ bcr_patient_barcode <chr> "TCGA-18-3406", "TCGA-18-3407", "TCGA-18-… ## $ disease <chr> "LUSC", "LUSC", "LUSC", "LUSC", "LUSC", "… ## $ years_to_death <dbl> 1.0164384, 0.3726027, 6.3123288, NA, 0.40… ## $ months_to_death <dbl> 12.366667, 4.533333, 76.800000, NA, 4.866…
The code above also features a new function,
glimpse, that can be
useful when developing new piped code. Note that we did not assign the
output above to a new object; we allowed it to be printed to the
Console. Because this is a large dataset, that type of output can be
glimpse allows us to see a preview of the data, including
the two new columns created.
Extract only lung cancer patients (LUSC, from disease) and create a new column called
total_cigrepresenting an estimate of the total number of cigarettes smoked (use columns
clinical dataset includes categorical (character) data. One
example is the
gender column. We can assess the different categories
available using a base R function:
# show categories in gender unique(clinical$gender)
##  "male" "female" NA
tidyverse includes an approach called split-apply-combine that allows
- split data into groups,
- apply a task for each group,
- combine the results back together into a single table.
We can try out this approach by counting the number of each gender in our dataset:
# count number of individuals of each gender clinical %>% group_by(gender) %>% tally()
## # A tibble: 3 x 2 ## gender n ## <chr> <int> ## 1 female 3535 ## 2 male 3258 ## 3 <NA> 39
group_by is not particularly useful by itself, but powerful together
with a second function like
tally. The two columns in the resulting
tibble represent the categories from
group_by and the number of cases
for each gender (n).
An additional function for use with
# summarize average days to death by gender clinical %>% group_by(gender) %>% summarize(mean_days_to_death = mean(days_to_death, na.rm = TRUE))
## `summarise()` ungrouping output (override with `.groups` argument) ## # A tibble: 3 x 2 ## gender mean_days_to_death ## <chr> <dbl> ## 1 female 947. ## 2 male 826. ## 3 <NA> NaN
mutate, we provide
summarize with a formula indicating
how we would like the groups to be handled.
In the command above, we use
na.rm = TRUE to exclude missing data from
the calculation of mean from
days_to_death. We still have
reported in the output table, though, because of the
NA category in
We can apply an additional filter to remove this missing data, prior to grouping:
# remove NA clinical %>% filter(!is.na(gender)) %>% group_by(gender) %>% summarize(mean_days_to_death = mean(days_to_death))
## `summarise()` ungrouping output (override with `.groups` argument) ## # A tibble: 2 x 2 ## gender mean_days_to_death ## <chr> <dbl> ## 1 female NA ## 2 male NA
Create object called smoke_complete from clinical that contains no missing data for cigarettes per day or age at diagnosis.
How do you save resulting table to file? How would you find this answer?
The solution to the challenges above represent the first of two datasets we’ll be using for data visualization in our next class. Make sure you’ve executed this code to save the filtered data file for use next time:
smoke_complete <- clinical %>% filter(!is.na(age_at_diagnosis)) %>% filter(!is.na(cigarettes_per_day)) write_csv(smoke_complete, "data/smoke_complete.csv")
The command above uses
write_csv, which is the
tidyverse method of
saving a csv file. Base R possesses a function,
performs a similar task, but by default includes quotation marks around
cells with character data as well as row names (sequential numbers,
unless otherwise specified).
Create a new object called birth_complete that contains no missing data for year of birth or vital status.
This challenge begins filtering the second of our two datasets for next time. Make sure you include the filter to remove missing data that’s been encoded as “not reported”!
# make sure ALL missing data is removed! birth_complete <- clinical %>% filter(!is.na(year_of_birth)) %>% filter(!is.na(vital_status)) %>% filter(vital_status != "not reported")
Filtering data based on number of cases of each type
We’re going to perform one last manipulation on this second dataset for next time, which will allow us to reduce the total number of cancer types present in this dataset.
First, we’ll need to count how many cases for each cancer type exist in the dataset:
# counting number of records in each cancer cancer_counts <- clinical %>% count(disease) %>% arrange(n)
count function is similar to
tally, but doesn’t need to have
group_by applied first. The
arrange function added at the end sorts
the table using the column specified. Although this isn’t necessary for
the analysis to proceed, it makes it easier for us to interpret the
Next, we’ll identify which cancer types are represented by at least 500 cases in this dataset:
# get names of frequently occurring cancers frequent_cancers <- cancer_counts %>% filter(n >= 500)
We can then use this object to filter based on the number of cases:
# extract data from cancers to keep birth_reduced <- birth_complete %>% filter(disease %in% frequent_cancers$disease)
The new syntax here is
%in%, which allows you to compare each entry in
birth_complete to the disease column in
frequent_cancers (remember that
frequent_cancers$disease means the
disease column from
frequent_cancers). This keeps only cases from
birth_complete dataset that are from cancers that are frequently
Finally, we’ll write the final output to a file:
# save results to file in data/ named birth_reduced write_csv(birth_reduced, "data/birth_reduced.csv")
Extract all tumor stages with more than 200 cases (Hint: also check to see if there are any other missing/ambiguous data!)
In this session, we acquainted ourselves with
tidyverse, and learned
some tools for data filtering and manipulation. We covered examples from
many of the main categories of data manipulation tasks; if you’d like
more information on these functions and others available (including
methods of joining multiple tables together), please check out the
In the next session, we’ll wrap up the course by creating
publication-quality images using
ggplot2, a data visualization package
tidyverse, and the two datasets we filtered in the sections above.
This document is written in R
markdown, which is a method of
formatting text, code, and output to create documents that are sharable
with other people. While this document is intended to serve as a
reference for you to read while typing code into your own script, you
may also be interested in modifying and running code in the original R
markdown file (
class3.Rmd in the GitHub repository).
Materials for all lessons in this course include:
- Class 1: R syntax, assigning objects, using functions
- Class 2: Data types and structures; slicing and subsetting data
- Class 3: Data manipulation with
- Class 4: Data visualization in
Answers to all challenge exercises are available here.
The following exercises all use the same
clinical data from this
How many hispanic or latino individuals in clinical are not also white? What are their races?
Create a new column for clinical called
age_at_death that calculates
this statistic (in years) from
dplyr includes several “helpers” that allows selection of columns
meeting particular criteria (described on the first page of the dplyr
cheatsheet near the top of the right hand column:
Using one of these tools, extract all columns that include “diagnosis”.
How many patients are hispanic or latino patients (column ethnicity), died after the year 2000 (year_of_death), and possess no missing data for cigarettes per day?