From vectors to tables
Atomic vectors are powerful, but they hold one type of thing. Real data is rarely like that. A weather station has a name (text), an altitude (a number), a list of sensors (several strings) and readings from each sensor (vectors of different lengths). A survey has one row per person, and for each person an age, a country, an answer on a scale and a date. R needs containers that can hold different types side by side.
It has two, and they are closely related. A list is a vector whose elements can be anything, including other lists, so it can represent any nested structure. A data frame is a special list whose elements are column vectors of equal length, which makes it a table: each column has one type, each row is one observation. Almost every analysis in R begins by getting data into a data frame, and much of an analyst's working day is spent selecting, filtering, sorting, combining and splitting data frames. This lesson covers the base R tools for all of that; the data analysis lesson then puts them to work on larger data, and shows the tidyverse equivalents. By the end of this one you should be able to look at any list or table in R and say exactly how to get any piece of it out.
Lists
Definition— List
A list is a vector whose elements may be of any type and length, including other lists. It is created with list(), usually with names for its elements, and inspected with str(), which shows its structure compactly.
There are three ways to take things out of a list, and the difference between them is one of the most important details in R:
x["a"](single brackets) returns a list containing the selected elements. It is like taking a smaller box with some of the items still inside it.x[["a"]](double brackets) returns the element itself: the item, out of the box.x$ais shorthand forx[["a"]]when the name is a fixed word.
# A list can hold anything, including other lists.
station <- list(
name = "Mount Kenya East",
altitude = 3100,
sensors = c("rain", "wind", "temperature"),
readings = list(rain = c(0, 2.5, 11), wind = c(14, 9))
)
str(station)
# [ ] returns a smaller list; [[ ]] and $ return the element itself.
station["altitude"]
station[["altitude"]]
station$sensors[2]
station$readings$rain
class(station["sensors"]); class(station[["sensors"]])
# Adding and removing elements.
station$active <- TRUE
station$altitude <- NULL # assigning NULL removes an element
names(station)
# Many R functions return lists. unlist() flattens one into a vector.
unlist(station$readings)List of 4
$ name : chr "Mount Kenya East"
$ altitude: num 3100
$ sensors : chr [1:3] "rain" "wind" "temperature"
$ readings:List of 2
..$ rain: num [1:3] 0 2.5 11
..$ wind: num [1:2] 14 9
$altitude
[1] 3100
[1] 3100
[1] "wind"
[1] 0.0 2.5 11.0
[1] "list"
[1] "character"
[1] "name" "sensors" "readings" "active"
rain1 rain2 rain3 wind1 wind2
0.0 2.5 11.0 14.0 9.0str() is the best way to understand an unfamiliar object: it shows each element's name, type, length and first values, indenting nested lists. station["altitude"] printed with a $altitude header, because it is still a list; station[["altitude"]] printed the number. You can chain extraction, as in station$readings$rain, to reach into nested lists. Assigning NULL to an element removes it, which is one reason NULL and NA must not be confused. And unlist() flattened the nested readings into one named vector.
Lists are everywhere in R even when you do not create them yourself. The results of statistical tests and models are lists (you will pull the p-value out of a t.test() result with $p.value), as are parsed JSON documents and the results of lapply().
Data frames
Definition— Data frame
A data frame is a list of column vectors that all have the same length, with class "data.frame". Each column has a name and a single type; each row is an observation. Because it is a list, df$col and df[["col"]] extract a column as a vector; because it is also a table, df[i, j] indexes rows and columns like a matrix.
The program below builds a small table of farms (the figures are invented for the example) and inspects it.
# A data frame is a list of columns of equal length.
crops <- data.frame(
farm = c("Asante", "Bello", "Chávez", "Dlamini", "Erdem", "Fonua"),
country = c("Ghana", "Nigeria", "Peru", "South Africa", "Turkey", "Tonga"),
crop = c("cocoa", "maize", "potato", "maize", "hazelnut", "taro"),
hectares = c(4.5, 12, 3.2, 20.5, 6, 2.8),
tonnes = c(2.1, 48.0, 61.5, 95.2, 7.3, 33.0)
)
str(crops)
dim(crops); names(crops)
# Columns are vectors: everything from the last lessons applies.
mean(crops$hectares)
crops$tonnes / crops$hectares
# A new column computed from others.
crops$yield <- round(crops$tonnes / crops$hectares, 1)
head(crops, 3)
summary(crops[, c("hectares", "yield")])'data.frame': 6 obs. of 5 variables:
$ farm : chr "Asante" "Bello" "Chávez" "Dlamini" ...
$ country : chr "Ghana" "Nigeria" "Peru" "South Africa" ...
$ crop : chr "cocoa" "maize" "potato" "maize" ...
$ hectares: num 4.5 12 3.2 20.5 6 2.8
$ tonnes : num 2.1 48 61.5 95.2 7.3 33
[1] 6 5
[1] "farm" "country" "crop" "hectares" "tonnes"
[1] 8.166667
[1] 0.4666667 4.0000000 19.2187500 4.6439024 1.2166667 11.7857143
farm country crop hectares tonnes yield
1 Asante Ghana cocoa 4.5 2.1 0.5
2 Bello Nigeria maize 12.0 48.0 4.0
3 Chávez Peru potato 3.2 61.5 19.2
hectares yield
Min. : 2.800 Min. : 0.500
1st Qu.: 3.525 1st Qu.: 1.900
Median : 5.250 Median : 4.300
Mean : 8.167 Mean : 6.883
3rd Qu.:10.500 3rd Qu.:10.000
Max. :20.500 Max. :19.200str() on a data frame lists the columns with their types: three character columns and two numeric ones. Each column is an ordinary vector, so vectorised arithmetic, mean() and everything else from the earlier lessons apply directly. Adding a column is an assignment to a new name, and the new column must have one value per row (or a single value, which is recycled). summary() of a data frame summarises every column; for numeric columns that means the quartiles and mean, for factors the counts per level.
The row names 1, 2, 3 at the left are part of every data frame. They are mostly a nuisance and are best ignored; the tidyverse's version of the data frame, the tibble, drops them and also prints more compactly, showing each column's type.
Selecting, filtering and sorting
Data frames are indexed with two positions, df[rows, columns], and each part takes any of the index kinds you know from vectors: positive positions, negative positions, names, or a logical vector. Leaving a part empty means "all". So df[df$crop == "maize", ] keeps the rows where the condition is true and all columns, which is the base R way to filter. order() on one or more columns gives the row positions that sort the table.
# A data frame is a list of columns of equal length.
crops <- data.frame(
farm = c("Asante", "Bello", "Chávez", "Dlamini", "Erdem", "Fonua"),
country = c("Ghana", "Nigeria", "Peru", "South Africa", "Turkey", "Tonga"),
crop = c("cocoa", "maize", "potato", "maize", "hazelnut", "taro"),
hectares = c(4.5, 12, 3.2, 20.5, 6, 2.8),
tonnes = c(2.1, 48.0, 61.5, 95.2, 7.3, 33.0)
)
crops$yield <- round(crops$tonnes / crops$hectares, 1)
# df[rows, columns]: each part can be positions, names or a logical vector.
crops[2, "crop"]
crops[crops$crop == "maize", c("farm", "country", "hectares")]
crops[crops$hectares > 4 & crops$yield > 2, ] # empty column part: all columns
# subset() says the same with less repetition.
subset(crops, yield > 10, select = c(farm, crop, yield))
# Sorting rows: order() gives the row positions.
crops[order(crops$yield, decreasing = TRUE), c("farm", "yield")]
# The drop = FALSE trap: one column comes back as a vector...
crops[1:2, "farm"]
# ...unless you ask to keep the data frame.
crops[1:2, "farm", drop = FALSE][1] "maize"
farm country hectares
2 Bello Nigeria 12.0
4 Dlamini South Africa 20.5
farm country crop hectares tonnes yield
2 Bello Nigeria maize 12.0 48.0 4.0
4 Dlamini South Africa maize 20.5 95.2 4.6
farm crop yield
3 Chávez potato 19.2
6 Fonua taro 11.8
farm yield
3 Chávez 19.2
6 Fonua 11.8
4 Dlamini 4.6
2 Bello 4.0
5 Erdem 1.2
1 Asante 0.5
[1] "Asante" "Bello"
farm
1 Asante
2 BelloEvery filter keeps the original row names, which shows which rows survived. subset() lets you write column names without repeating crops$, which is convenient interactively; inside functions that you write for others, the explicit df[cond, cols] form is safer, because subset() looks up names in a way that can pick up variables from outside the data frame by mistake.
The last two results are the drop trap. When a selection leaves a single column, [ simplifies the result to a plain vector, so code that expected a data frame (with nrow(), with column names) breaks only when, one day, someone asks for exactly one column. Inside functions, write drop = FALSE whenever the number of columns is not fixed. Tibbles never drop, which is one of the reasons the tidyverse introduced them.
Merging, stacking and splitting
Data rarely arrives in one table. You might have one table of farms and another of crop prices (again invented here), and need to bring them together. merge() joins two data frames on their common columns, exactly like an SQL join: by default it keeps only rows that match (an inner join); all.x = TRUE keeps every row of the first table (a left join), filling the missing columns with NA. rbind() stacks data frames with the same columns, and split() does the reverse of combining, dividing a vector or a data frame into a list with one element per group.
# A data frame is a list of columns of equal length.
crops <- data.frame(
farm = c("Asante", "Bello", "Chávez", "Dlamini", "Erdem", "Fonua"),
country = c("Ghana", "Nigeria", "Peru", "South Africa", "Turkey", "Tonga"),
crop = c("cocoa", "maize", "potato", "maize", "hazelnut", "taro"),
hectares = c(4.5, 12, 3.2, 20.5, 6, 2.8),
tonnes = c(2.1, 48.0, 61.5, 95.2, 7.3, 33.0)
)
# Another table: prices per tonne, by crop.
prices <- data.frame(
crop = c("cocoa", "maize", "potato", "taro", "rice"),
usd_per_tonne = c(3200, 210, 260, 900, 420)
)
# merge() joins two data frames on common columns (like an SQL join).
m <- merge(crops, prices, by = "crop")
m$value <- m$tonnes * m$usd_per_tonne
m[, c("farm", "crop", "tonnes", "value")]
# all.x = TRUE keeps farms whose crop has no price (a left join).
nrow(merge(crops, prices, by = "crop", all.x = TRUE))
# rbind() adds rows; the columns must match by name.
new_farm <- data.frame(farm = "Gurung", country = "Nepal", crop = "rice",
hectares = 1.5, tonnes = 6.3)
crops <- rbind(crops, new_farm)
nrow(crops)
# split() divides a vector (or a data frame) into a list with one element per group.
by_crop <- split(crops$hectares, crops$crop)
by_crop$maize
sapply(by_crop, sum) farm crop tonnes value
1 Asante cocoa 2.1 6720
2 Bello maize 48.0 10080
3 Dlamini maize 95.2 19992
4 Chávez potato 61.5 15990
5 Fonua taro 33.0 29700
[1] 6
[1] 7
[1] 12.0 20.5
cocoa hazelnut maize potato rice taro
4.5 6.0 32.5 3.2 1.5 2.8The inner join kept five of the six farms: hazelnuts have no price, so Erdem's farm dropped out, while the left join kept all six. As with SQL joins, check row counts before and after every merge; a key that appears twice in one table multiplies rows in the result. The merged table is sorted by the key, crop, which is why the rows changed order.
split() followed by a function applied to each group is a pattern you will use constantly: here sapply(by_crop, sum) gives the total area per crop. The next lesson introduces sapply and its relatives properly, and the lesson after that shows shorter ways to compute grouped summaries, such as aggregate() and tapply().
One row per observation
A data frame can hold the same information in different shapes, and choosing the shape is one of the quiet decisions that makes analysis easy or painful. Consider rainfall for three towns over two months (the figures are made up for the example). A spreadsheet might store it wide, one row per town and one column per month:
| town | jan_mm | feb_mm |
|---|---|---|
| Arusha | 64 | 71 |
| Bergen | 190 | 152 |
| Cusco | 155 | 140 |
or long, one row per town and month:
| town | month | mm |
|---|---|---|
| Arusha | jan | 64 |
| Arusha | feb | 71 |
| Bergen | jan | 190 |
| … | … | … |
The wide form is easy for people to read. The long form is easier for computation: "average rainfall per month" is a grouped mean over the mm column, a plot can map month to the x-axis, and adding a third month adds rows, not columns. Statisticians have long called this the difference between a table for presentation and a table for analysis. Hadley Wickham's 2014 paper on tidy data gave the analysis form a name and three rules: each variable is a column, each observation is a row, and each value is a cell. Most of R's modelling and plotting functions, base and tidyverse alike, expect data in this shape.
When data arrives wide, reshape it before analysing. Base R has reshape() and stack(), which are powerful but notoriously hard to remember; the tidyverse's tidyr package has pivot_longer() and pivot_wider(), which most people find much easier, and the next lesson shows both. The habit to build now is simply to ask, of every table you receive: what is one observation here, and is each one a row?
The shape question also explains why a data frame's columns, not its rows, are its vectors: a variable has one type across all observations, while a single observation mixes types, a name, a number and a date. Two small base R helpers make working with a data frame's columns less repetitive. with(crops, sum(tonnes) / sum(hectares)) evaluates an expression with the columns available by name, and within() does the same for an expression that adds or changes columns, returning the modified data frame.
List and data frame pitfalls
Misconception
"x[i] and x[[i]] are the same for lists." Single brackets return a list; double brackets return the element. mean(station["readings"]) fails because it receives a list, while mean(station[["readings"]][["rain"]]) works. When in doubt, ask class() of what you extracted.
Misconception
"$ always finds the column I named, or fails." On lists and data frames, $ does partial matching of names: if a data frame has a column hectares and none starting with hect… other than it, crops$hect quietly returns that column. That is convenient at the console and dangerous in scripts, where a misspelled name may silently match the wrong column, or, if nothing matches, return NULL without an error. df[["name"]] matches exactly.
Misconception
"Growing a data frame row by row in a loop is fine." Each rbind() copies the whole table, so building a large data frame one row at a time is very slow. Build a list of pieces and combine them once at the end with do.call(rbind, pieces), or better, compute whole columns with vectorised operations.
The same ideas in the tidyverse
Many R users today work with data frames through the tidyverse's dplyr package, whose functions are named after verbs and are designed to be chained with a pipe. You will meet the pipe in the next lesson, but it is worth seeing now how directly the base R operations above correspond to dplyr's verbs. The dplyr package is not installed where this course's programs are checked, so this is shown for reading, not run:
library(dplyr)
crops |>
filter(hectares > 4, yield > 2) |> # crops[crops$hectares > 4 & crops$yield > 2, ]
select(farm, crop, yield) |> # crops[, c("farm", "crop", "yield")]
arrange(desc(yield)) # crops[order(-crops$yield), ]
crops |> mutate(yield = tonnes / hectares) # crops$yield <- crops$tonnes / crops$hectares
left_join(crops, prices, by = "crop") # merge(crops, prices, by = "crop", all.x = TRUE)
bind_rows(crops, new_farm) # rbind(crops, new_farm)Inside dplyr verbs you write bare column names, as with subset(), and the verbs always return a tibble, never a vector. Neither style is "correct": base R needs no installation and is stable across decades, while dplyr reads more like a sentence and has excellent documentation. Professionals read both, and teams usually pick one style for a code base and stay consistent. This course teaches base R first because it is what every package, every Stack Overflow answer from the last twenty years and every R installation has in common.
Exercises
1. Lists and data frames
Warm-upComputationalWhat is length(cfg)?
cfg <- list(title = "Survey", waves = c(2024, 2026),
owner = list(name = "Ife", team = "stats"))
2. Lists and data frames
CoreComputationalWith cfg as in the previous exercise, which expression returns the number 2026?
3. Lists and data frames
CoreComputationalWhat does the last line print? Write it with single spaces.
pets <- data.frame(name = c("Kito", "Luna", "Milo", "Nala"),
kind = c("cat", "dog", "cat", "dog"),
age = c(3, 7, 1, 5))
pets[pets$age > 2, "name"]
4. Lists and data frames
CoreComputationalWith the same pets data frame, what does pets[order(pets$age), ][1, "name"] print?
5. Lists and data frames
StretchComputationalHow many rows does the left join have?
a <- data.frame(id = c(1, 2, 3, 4), city = c("Kumasi", "Lyon", "Medan", "Natal"))
b <- data.frame(id = c(2, 4, 4, 5), visits = c(10, 3, 8, 1))
nrow(merge(a, b, by = "id", all.x = TRUE))
6. Lists and data frames
CoreHumanA function receives a data frame df and a character vector cols of column names chosen by the caller, and runs x <- df[, cols] followed by nrow(x). It works in testing but fails when a user passes a single column name. Why, and what is the fix?
7. Lists and data frames
CoreHumanA file has one row per student and columns maths_2025, maths_2026, art_2025, art_2026 holding marks. In tidy form, what should one row be?
Further reading and resources
- An Introduction to R, chapter 6: Lists and data frames · R Core Team
- Advanced R, chapter 4: Subsetting · Hadley WickhamThe precise rules of [, [[ and $ for every kind of object.
- R for Data Science (2e) · Hadley Wickham, Mine Çetinkaya-Rundel and Garrett GrolemundData frames and tibbles from the tidyverse point of view.
- Tidy Data · Hadley Wickham · 2014The paper that named the "one row per observation" principle, in the Journal of Statistical Software.