Missing values, duplicates and tidying

A gap is a value, not a nothing

A missing reading is stored as NA, which is a value in the vector: it takes a position, it has a type, and it says "not known". It is not zero and not an empty string, and R will never quietly drop it for you.

is.na(values) gives a logical vector with TRUE exactly where the values are missing. From there the tools you already know do the rest: which gives the positions, sum counts them, and !is.na(values) marks the rows that are present.

Working with the gaps

Most summary functions take na.rm = TRUE, which drops the missing values for that one calculation: mean(temps, na.rm = TRUE) averages the readings you have. Without it, one unknown makes the whole answer unknown, which is R refusing to invent a number.

The other route is to drop the rows: readings[!is.na(readings$temp), ] keeps only the days with a reading. That is a decision about your data, not a formality — ask whether the gaps are random before you throw them away.

Duplicates, and what is in one list but not the other

unique(values) removes repeated values, keeping the first of each, and length(unique(values)) is the count of distinct ones. table gives the same information with the counts attached, which is often more useful than either.

setdiff(a, b) answers "in a but not in b", and intersect(a, b) answers "in both". Both are about whole values rather than positions, so they are the right tools for comparing lists of names or codes.

One more thing worth knowing about NULL: it means no value at all, and assigning it to a column removes that column from a frame.

Find the gaps, then tidy the values

is.na(c(1, NA, 3))
which(is.na(c(1, NA, 3)))
sum(is.na(c(1, NA, 3)))

mean(c(1, NA, 3))
mean(c(1, NA, 3), na.rm = TRUE)

unique(c("A1", "B2", "A1"))
length(unique(c("A1", "B2", "A1")))
setdiff(c("A1", "B2"), c("B2"))
intersect(c("A1", "B2"), c("B2", "C3"))

empty <- NULL
length(empty)

Missing values have types

NA is not one object but a family: NA_real_, NA_integer_, NA_character_ and the logical NA. The bare NA is converted to the type the vector needs, which is why c(1, NA) holds a numeric missing while c("a", NA) holds a text one. This matters when you compare or combine vectors: a missing number and the string "NA" are different things, and only one of them is missing.

is.na works on a whole data frame and returns a matrix of flags, so sum(is.na(customers)) counts missing cells while sum(!complete.cases(customers)) counts incomplete rows. anyNA(x) answers the quick question, and na.omit drops every row that has a gap anywhere — a large decision hiding behind a short name.

NA also has a companion for "not a number": NaN, which appears when a calculation has no sensible answer, such as zero divided by zero. is.na(NaN) is TRUE, because NaN is also unknown, but is.nan tells the two apart when the difference matters. Neither one is zero, and neither should be replaced by zero without a reason.

Duplicates, sets and order

duplicated marks the second and later copies of a row, so !duplicated(x) keeps the first of each. On a frame the comparison uses every column by default; pass a column, as in duplicated(customers$id), to deduplicate by key instead. Those two questions — "is this exact row repeated?" and "is this id repeated?" — have different answers, and the customer list shows both.

unique, setdiff and intersect treat their input as a set: each value appears once, and order is not part of the answer. setdiff(a, b) reads "in a but not in b", so swapping the arguments gives a different list. If order matters, match and %in% are the tools; they answer where a value occurs rather than whether it belongs to a set.

Try these next

• Count missing cells and incomplete rows in the customer table, and compare the two numbers.
• Deduplicate by id and by whole row, and explain why the counts differ.
• Compare setdiff(a, b) with setdiff(b, a) on two short vectors.