Most of the work is before the statistics
Analysts often say that most of their time goes on getting data ready: reading it, discovering what is wrong with it, fixing types, filling in or excluding gaps, recoding categories, joining reference tables, and reshaping it so that the question can be asked. The statistics at the end is often a few lines. Doing the preparation carefully, in code rather than by hand in a spreadsheet, is what makes an analysis trustworthy and repeatable: when next month's file arrives, you run the script again.
This lesson works through that process on a small, deliberately messy example: a survey of market prices for rice and beans in three East African cities, recorded in local currencies, with a stray space in one city name and a missing price written as "n/a". (The data and exchange rates are invented for the example.) Then it summarises real built-in data with a pipeline. Along the way it shows how the same steps look in the tidyverse, which many teams use for exactly this kind of work. The goal is not to memorise every function, but to learn the shape of the process: read, inspect, fix, check, summarise, and keep every step in the script so that anyone can repeat it.
Reading data and looking at it
read.csv() reads comma-separated text into a data frame. Usually you give it a file path or a URL; here the text is given directly with text = so that the program is self-contained. Its most useful arguments are na.strings (which values mean "missing"), strip.white (trim spaces around fields), colClasses (declare column types), dec and sep (for files that use a decimal comma and semicolons, common in much of Europe and Latin America; read.csv2() assumes both), and fileEncoding for text that is not UTF-8.
# A small, messy survey of market prices, as it might arrive in a CSV file.
# (Invented data. In real work this would be read.csv("prices.csv").)
csv <- "market,city,date,item,price_local
Kariakoo, Dar es Salaam ,2026-03-02,rice_kg,3200
Kariakoo,Dar es Salaam,2026-03-02,beans_kg,4100
Kariakoo,Dar es Salaam,2026-04-06,rice_kg,3350
Kariakoo,Dar es Salaam,2026-04-06,beans_kg,n/a
Owino,Kampala,2026-03-03,rice_kg,5200
Owino,Kampala,2026-03-03,beans_kg,6000
Owino,Kampala,2026-04-07,rice_kg,5400
Owino,Kampala,2026-04-07,beans_kg,6100
Kimironko,Kigali,2026-03-04,rice_kg,1650
Kimironko,Kigali,2026-03-04,beans_kg,1200
Kimironko,Kigali,2026-04-08,rice_kg,1700
Kimironko,Kigali,2026-04-08,beans_kg,1150"
prices <- read.csv(text = csv)
str(prices)
# Read it again, telling read.csv what a missing value looks like
# and to trim stray spaces around text fields.
prices <- read.csv(text = csv, na.strings = c("", "n/a"), strip.white = TRUE)
str(prices)
colSums(is.na(prices))
unique(prices$city)'data.frame': 12 obs. of 5 variables:
$ market : chr "Kariakoo" "Kariakoo" "Kariakoo" "Kariakoo" ...
$ city : chr " Dar es Salaam " "Dar es Salaam" "Dar es Salaam" "Dar es Salaam" ...
$ date : chr "2026-03-02" "2026-03-02" "2026-04-06" "2026-04-06" ...
$ item : chr "rice_kg" "beans_kg" "rice_kg" "beans_kg" ...
$ price_local: chr "3200" "4100" "3350" "n/a" ...
'data.frame': 12 obs. of 5 variables:
$ market : chr "Kariakoo" "Kariakoo" "Kariakoo" "Kariakoo" ...
$ city : chr "Dar es Salaam" "Dar es Salaam" "Dar es Salaam" "Dar es Salaam" ...
$ date : chr "2026-03-02" "2026-03-02" "2026-04-06" "2026-04-06" ...
$ item : chr "rice_kg" "beans_kg" "rice_kg" "beans_kg" ...
$ price_local: int 3200 4100 3350 NA 5200 6000 5400 6100 1650 1200 ...
market city date item price_local
0 0 0 0 1
[1] "Dar es Salaam" "Kampala" "Kigali"The first str() shows two problems that are invisible in a spreadsheet. The first city reads " Dar es Salaam " with spaces, so it would count as a fourth city in every table. And price_local is character, because one cell says n/a, so mean() would refuse it. Reading again with na.strings and strip.white fixes both at the source: prices become integers with one NA, and the cities are clean. colSums(is.na(prices)) then counts missing values in every column at once, the check that should follow every import.
Other formats have their own readers. readRDS() and saveRDS() store a single R object exactly, types and all. For Excel files, the readxl package; for SPSS, Stata and SAS files, haven; for very large CSV files, data.table::fread() or arrow (which also reads the columnar Parquet format); for databases, the DBI package with a driver such as RSQLite or RPostgres, which lets you send SQL and receive data frames. The tidyverse's readr package provides read_csv(), which is faster than read.csv(), reports the column types it guessed, and returns a tibble.
Where your files are
read.csv("prices.csv") looks for the file in R's working directory, which getwd() reports. Scripts that begin with setwd("C:/Users/amina/Desktop/project") work only on one computer, so professionals avoid them. Instead, they keep each analysis in its own folder (an RStudio or Positron project), open R in that folder, and use paths relative to it, such as "data/raw/prices.csv"; the here package builds such paths reliably from anywhere inside the project. Keep raw data in its own folder, never edited, and write cleaned data and results to separate folders, so that it is always clear what came from outside and what the script produced.
Fixing types, recoding and joining
With the file read, the next steps give each column its proper type and add the information the analysis needs. Dates become Date objects, from which the month is extracted. The item codes become a factor with readable labels in a chosen order. Prices in three currencies are converted to one by joining a small reference table of exchange rates. And a named vector serves as a lookup table from city to country: indexing it by the city column returns the matching country for every row.
# A small, messy survey of market prices, as it might arrive in a CSV file.
# (Invented data. In real work this would be read.csv("prices.csv").)
csv <- "market,city,date,item,price_local
Kariakoo, Dar es Salaam ,2026-03-02,rice_kg,3200
Kariakoo,Dar es Salaam,2026-03-02,beans_kg,4100
Kariakoo,Dar es Salaam,2026-04-06,rice_kg,3350
Kariakoo,Dar es Salaam,2026-04-06,beans_kg,n/a
Owino,Kampala,2026-03-03,rice_kg,5200
Owino,Kampala,2026-03-03,beans_kg,6000
Owino,Kampala,2026-04-07,rice_kg,5400
Owino,Kampala,2026-04-07,beans_kg,6100
Kimironko,Kigali,2026-03-04,rice_kg,1650
Kimironko,Kigali,2026-03-04,beans_kg,1200
Kimironko,Kigali,2026-04-08,rice_kg,1700
Kimironko,Kigali,2026-04-08,beans_kg,1150"
prices <- read.csv(text = csv, na.strings = c("", "n/a"), strip.white = TRUE)
# Proper types: a Date, and a factor with the items in a chosen order.
prices$date <- as.Date(prices$date)
prices$month <- format(prices$date, "%Y-%m")
prices$item <- factor(prices$item, levels = c("rice_kg", "beans_kg"),
labels = c("rice", "beans"))
# A lookup table: currency and an (invented) exchange rate for each city.
rates <- data.frame(city = c("Dar es Salaam", "Kampala", "Kigali"),
currency = c("TZS", "UGX", "RWF"),
per_usd = c(2600, 3700, 1400))
prices <- merge(prices, rates, by = "city")
prices$usd <- round(prices$price_local / prices$per_usd, 2)
# Recoding with a named vector: a common, fast idiom.
country <- c("Dar es Salaam" = "Tanzania", Kampala = "Uganda", Kigali = "Rwanda")
prices$country <- unname(country[prices$city])
head(prices[, c("country", "month", "item", "price_local", "currency", "usd")], 4) country month item price_local currency usd
1 Tanzania 2026-03 rice 3200 TZS 1.23
2 Tanzania 2026-03 beans 4100 TZS 1.58
3 Tanzania 2026-04 rice 3350 TZS 1.29
4 Tanzania 2026-04 beans NA TZS NAThe factor() call used both levels (the codes as they appear in the data, in the order wanted) and labels (what to call them), which renames and orders categories in one step. The named-vector lookup, country[prices$city], is a small idiom worth remembering: it is vectorised, fast, and clearer than a chain of ifelse() calls; unname() drops the names it would otherwise carry. For a lookup with several columns, merge() with a reference table is the tool, as with the exchange rates. Notice that the missing April price for beans stayed NA through the conversion; nothing invented a value for it.
Summarising by group, and reshaping
The core of descriptive analysis is split, apply, combine: split the data into groups, compute a summary for each, and combine the results into a table. Base R has several functions for it, which differ in their input and the shape of their output:
| Function | Typical use | Result |
|---|---|---|
table(a, b) | counts per combination of categories | a contingency table |
prop.table(t) | turn counts into proportions | the same shape |
aggregate(y ~ g1 + g2, data, FUN) | a summary of y per group, with a formula | a data frame |
tapply(y, list(g1, g2), FUN) | the same | a matrix or array |
cut(x, breaks) | turn a number into bands | a factor |
reshape(..., direction = "wide") | long to wide (or back) | a data frame |
# A small, messy survey of market prices, as it might arrive in a CSV file.
# (Invented data. In real work this would be read.csv("prices.csv").)
csv <- "market,city,date,item,price_local
Kariakoo, Dar es Salaam ,2026-03-02,rice_kg,3200
Kariakoo,Dar es Salaam,2026-03-02,beans_kg,4100
Kariakoo,Dar es Salaam,2026-04-06,rice_kg,3350
Kariakoo,Dar es Salaam,2026-04-06,beans_kg,n/a
Owino,Kampala,2026-03-03,rice_kg,5200
Owino,Kampala,2026-03-03,beans_kg,6000
Owino,Kampala,2026-04-07,rice_kg,5400
Owino,Kampala,2026-04-07,beans_kg,6100
Kimironko,Kigali,2026-03-04,rice_kg,1650
Kimironko,Kigali,2026-03-04,beans_kg,1200
Kimironko,Kigali,2026-04-08,rice_kg,1700
Kimironko,Kigali,2026-04-08,beans_kg,1150"
prices <- read.csv(text = csv, na.strings = c("", "n/a"), strip.white = TRUE)
# Proper types: a Date, and a factor with the items in a chosen order.
prices$date <- as.Date(prices$date)
prices$month <- format(prices$date, "%Y-%m")
prices$item <- factor(prices$item, levels = c("rice_kg", "beans_kg"),
labels = c("rice", "beans"))
# A lookup table: currency and an (invented) exchange rate for each city.
rates <- data.frame(city = c("Dar es Salaam", "Kampala", "Kigali"),
currency = c("TZS", "UGX", "RWF"),
per_usd = c(2600, 3700, 1400))
prices <- merge(prices, rates, by = "city")
prices$usd <- round(prices$price_local / prices$per_usd, 2)
# Recoding with a named vector: a common, fast idiom.
country <- c("Dar es Salaam" = "Tanzania", Kampala = "Uganda", Kigali = "Rwanda")
prices$country <- unname(country[prices$city])
# Counting: how many observations per country and item?
table(prices$country, prices$item)
# aggregate() with a formula: mean USD price by item and month.
aggregate(usd ~ item + month, data = prices, FUN = mean)
# tapply() gives the same numbers as a matrix (NA where data is missing).
round(tapply(prices$usd, list(prices$country, prices$item), mean), 2)
# Change from March to April for each country and item.
wide <- reshape(prices[, c("country", "item", "month", "usd")],
idvar = c("country", "item"), timevar = "month",
direction = "wide")
wide$change_pct <- round(100 * (wide$`usd.2026-04` / wide$`usd.2026-03` - 1), 1)
wide[order(wide$country, wide$item), ]
rice beans
Rwanda 2 2
Tanzania 2 2
Uganda 2 2
item month usd
1 rice 2026-03 1.273333
2 beans 2026-03 1.353333
3 rice 2026-04 1.320000
4 beans 2026-04 1.235000
rice beans
Rwanda 1.19 0.84
Tanzania 1.26 NA
Uganda 1.44 1.64
country item usd.2026-03 usd.2026-04 change_pct
9 Rwanda rice 1.18 1.21 2.5
10 Rwanda beans 0.86 0.82 -4.7
1 Tanzania rice 1.23 1.29 4.9
2 Tanzania beans 1.58 NA NA
5 Uganda rice 1.41 1.46 3.5
6 Uganda beans 1.62 1.65 1.9The table confirms the design is balanced: two observations per country and item. Now compare the two summaries of the April bean price. aggregate() with a formula reports 1.235, the mean of Uganda and Rwanda, because its formula interface silently drops rows with missing values (its na.action defaults to na.omit). tapply() keeps the gap visible: Tanzania's bean price is NA. Neither is wrong, but they answer different questions ("average of the prices we have" versus "average price per country"), and a report should say which it uses and how many observations were missing.
reshape() turned the long table, one row per country, item and month, into a wide one with a column per month, so that the change from March to April is a simple calculation per row. Base R's reshape() is powerful but hard to remember; the tidyverse equivalent shown below is easier to read. Going the other way, from wide to long, is just as common, because data entered by people tends to be wide (a column per month or per year) while models and plots want it long; reshape(..., direction = "long") does it in base R, as does stack() for the simplest cases.
A pipeline on real data
The native pipe makes a sequence of base R steps read like a recipe. This program uses the built-in airquality data set: daily ozone, solar radiation, wind and temperature measurements in New York from May to September 1973.
# A base R pipeline on a built-in data set: New York air quality, May to September 1973.
airquality |>
subset(!is.na(Ozone), select = c(Ozone, Temp, Month)) |>
transform(month = month.abb[Month]) |>
with(tapply(Ozone, factor(month, levels = month.abb[5:9]), median))
# cut() turns a number into bands; table() and prop.table() count them.
bands <- cut(airquality$Temp, breaks = c(50, 70, 80, 90, 100),
labels = c("cool", "mild", "warm", "hot"))
table(bands)
round(100 * prop.table(table(bands)), 1)
# quantile() for any percentiles; complete.cases() for rows with no NA.
quantile(airquality$Ozone, probs = c(0.1, 0.5, 0.9), na.rm = TRUE)
sum(complete.cases(airquality))
# Save results: CSV for people and other programs, RDS to keep R types exactly.
out <- tempfile(fileext = ".rds")
saveRDS(bands, out)
identical(readRDS(out), bands)May Jun Jul Aug Sep
18 23 60 52 23
bands
cool mild warm hot
33 52 54 14
bands
cool mild warm hot
21.6 34.0 35.3 9.2
10% 50% 90%
11.0 31.5 87.0
[1] 111
[1] TRUEThe pipeline drops days with no ozone reading, keeps three columns, adds a month name (month.abb is a built-in vector of abbreviated month names), and computes the median ozone per month with tapply() inside with(), which makes the columns available by name. Ozone levels peak in July and August, the hottest months. cut() divided temperatures, in degrees Fahrenheit, into four labelled bands, and prop.table() turned the counts into percentages. Only 111 of the 153 days have no missing value in any column, a reminder that "complete cases" analysis can quietly discard over a quarter of the data. Finally, saveRDS() and readRDS() round-tripped a factor exactly, labels and all, which CSV cannot do. Use RDS for intermediate results that only R will read, and CSV for anything a person or another program needs.
Checks and decisions that are easy to skip
Data preparation goes wrong quietly, so experienced analysts run the same handful of checks on every new data set, in code, before any summary is trusted:
- Keys are unique. If each row should be one market, item and date,
anyDuplicated(prices[, c("market", "item", "date")])should be 0. Duplicate rows double-count, and duplicated keys multiply rows in every latermerge(). - Categories are spelled one way.
table(prices$city, useNA = "ifany")lists every distinct value with its count; a city appearing twice under slightly different spellings, or a stray empty string, shows up at once. - Values are plausible.
summary()shows ranges: a negative price, a temperature of 999 (a common code for "missing" in older data), or a date in the wrong century are found in seconds. - Joins kept what they should. Compare
nrow()before and after everymerge(), and look forNAs in the columns the join added; a city missing from the reference table silently loses its rows in an inner join.
Misconception
"na.rm = TRUE is a safe default." It answers a different question: the average of the values you have. If values are missing for a reason, for instance prices that were not recorded because the item was scarce and expensive, dropping them biases the result. Statisticians distinguish data missing completely at random, where dropping it only costs precision, from data whose missingness is related to the value itself, where it can mislead. Whatever you do, count and report the missing values, and say how they were handled.
Misconception
"Cleaning by hand in the spreadsheet first is quicker." It is, once. But hand edits leave no record of what was changed or why, cannot be repeated when the next file arrives, and cannot be reviewed by a colleague. Keep the raw file untouched, do every correction in the script, and let the script be the record. If a correction needs a judgement call, write the reason in a comment beside it.
The same work in the tidyverse
Much of the R written for data work today uses the tidyverse: readr to read, dplyr to filter, transform, join and summarise, tidyr to reshape, all chained with the pipe and returning tibbles. These packages are not installed where this course's code is checked, so the code below is for reading. It performs the cleaning and the March-to-April comparison from this lesson:
library(readr); library(dplyr); library(tidyr)
prices <- read_csv("prices.csv", na = c("", "n/a")) |> # trims spaces by default
mutate(month = format(date, "%Y-%m"), # date is parsed automatically
item = factor(item, levels = c("rice_kg", "beans_kg"),
labels = c("rice", "beans"))) |>
left_join(rates, by = "city") |>
mutate(usd = round(price_local / per_usd, 2),
country = unname(country_of[city])) # a named lookup vector, as in base R
prices |>
group_by(item, month) |>
summarise(mean_usd = mean(usd, na.rm = TRUE), n = n(), .groups = "drop")
prices |>
select(country, item, month, usd) |>
pivot_wider(names_from = month, values_from = usd) |>
mutate(change_pct = round(100 * (`2026-04` / `2026-03` - 1), 1))Each verb does one thing and its name says what: mutate adds or changes columns, left_join joins, group_by with summarise is split-apply-combine, pivot_wider and pivot_longer reshape. Note that summarise makes missing values explicit: you write na.rm = TRUE yourself, rather than having aggregate() drop them for you.
How do professionals choose? Teams doing a lot of interactive analysis often prefer the tidyverse for its readability and its consistent documentation. Package authors often prefer base R, because every dependency is something their users must install and keep compatible. Groups working with very large data often use data.table, whose compact syntax (dt[price > 1, .(mean_usd = mean(usd)), by = .(item, month)]) is extremely fast. All three are respected choices; what matters is consistency within a project and the habits that apply everywhere: inspect after every step, count rows before and after joins, and make every decision about missing data explicit.
Exercises
1. Reading, cleaning and summarising data
Warm-upComputationalWhat number does the second line print?
df <- read.csv(text = "id,score\n1,8\n2,missing\n3,6", na.strings = "missing")
mean(df$score, na.rm = TRUE)
2. Reading, cleaning and summarising data
CoreComputationalGiven d <- data.frame(g = c("a", "a", "b", "b", "b"), y = c(2, NA, 5, 7, 9)), what does aggregate(y ~ g, data = d, FUN = length) report for group a?
3. Reading, cleaning and summarising data
CoreComputationalHow many of the values fall in the band (10,20]?
x <- c(3, 12, 18, 25, 7, 30)
table(cut(x, breaks = c(0, 10, 20, 30)))
4. Reading, cleaning and summarising data
CoreComputationalWhat does this print? Write it with single spaces.
codes <- c(K = "Kenya", G = "Ghana", P = "Peru")
unname(codes[c("P", "K", "K", "X")])
5. Reading, cleaning and summarising data
CoreHumanAfter merging a table of 1,200 sales with a table of products on product_id, the result has 1,350 rows. What is the most likely cause?
6. Reading, cleaning and summarising data
Warm-upHumanWhich way of reading a data file makes a script easiest for a colleague to run on their own computer?
Further reading and resources
- R Data Import/Export · R Core TeamThe official manual on reading and writing data in every common format.
- R for Data Science (2e): Data import, Data tidying, Data transformation · Hadley Wickham, Mine Çetinkaya-Rundel and Garrett Grolemund
- dplyrThe tidyverse's grammar of data manipulation.
- data.tableA fast alternative to data frames for large data, with its own concise syntax.