Clustered Data
A plain-language tutorial: what changes when your observations arrive in groups rather than one at a time, why the sample is smaller than it looks, and why the mistake this causes is invisible to the check most people run.
Statistical methods are written for observations that arrive one at a time and independently. Measurements rarely do. They arrive 20 to a batch, 9 to a cutting tool, 40 to an operator’s shift, 6 to a patient, 100 to a site. The rows in the spreadsheet look identical either way, and that is the whole problem: nothing in the file records that rows 41 through 60 all came off the same fixture on the same afternoon.
3 things change when observations arrive in groups, in increasing order of how much trouble they cause.
- The sample is smaller than the row count says (§ 02).
- Any interval computed as though the rows were independent is too narrow (§ 04).
- Any validation split that divides rows rather than groups puts the same group on both sides, so the evaluation measures an easier task than the one you asked about (§ 03).
The third is the dangerous one, and § 03 is about why it survives the checks people run. § 05 is about the judgement the other 4 sections depend on: deciding what the group is.
What Counts as a Group
The group is whatever the repeated thing is. It is a fact about how the data was produced, not a column in the file.
A group is any set of observations that share something the analysis does not model. Successive parts machined with one insert share an edge and its accumulated wear. Coupons cut from one plate carry the microstructure of that plate. Inspections booked to one technician all carry that technician’s threshold for what counts as an indication, and in clinical data the nesting goes further still: visits inside patients, patients inside sites. The file records none of it.
Observations are clustered when they can be partitioned into groups such that 2 observations from the same group are more alike than 2 from different groups, for reasons the model does not capture. The strength of that resemblance is the intraclass correlation : the share of total variance that lives between groups rather than within them.
At the groups carry nothing and the rows are effectively independent. At every observation in a group is a copy of the others, and a group of 40 tells you exactly what a group of 1 would.
The awkward part is that the grouping is a fact about the world and the file has no column for it. Nothing in a measurement table marks which rows shared a fixture, so a modelling pipeline treats all of them as independent draws because it can’t know otherwise. Whether the data is clustered is a question for the person who knows how it was collected, and somebody has to ask it out loud.
Clustering belongs to the experiment, not to the file. Cleaning the file does not touch it.
What Your Sample Is Actually Worth
With 10 observations per group and ρ = 0.5, 800 rows are worth fewer than 150.
Groups of similar observations carry less information than the same number of independent ones. The discount has a name and a closed form.
For groups of size with intraclass correlation , the design effect is
and the effective sample size is neff = n / deff. Every quantity whose precision depends on sample size, which is all of them, should use neff rather than . 1The design effect comes from survey sampling, where clustered designs are chosen deliberately for cost reasons and the resulting loss of precision is budgeted for in advance. In machine learning the clustering usually arrives by accident and the loss is discovered afterwards, or not at all.
The formula’s shape is worth reading rather than just applying. It says the penalty grows with group size, not with the number of groups, so collecting more observations per group buys progressively less. Double the number of groups from 5 to 10 and the interval narrows in the usual way, by the square root of 2. Double the observations per group from 10 to 20, at , and the design effect climbs from 5.5 to 10.5 almost as fast as the row count climbs: the effective sample barely moves. Fig. 01 draws it.
Observations that arrive in groups do not each carry a full observation's worth of information. The design effect prices the discount: with m observations per group and an intraclass correlation ρ, the effective sample size is n divided by 1 + (m − 1)ρ. Drag ρ and watch what your sample is actually worth.
The cost lands on every number you report. A coverage claim or a defect rate is estimated with a precision set by neff, and quoting the interval from instead reports a confidence nobody earned.
The natural response to a wide interval is to collect more data, and the cheapest way to collect more data is usually to take more measurements from the units you’ve already got. At high that is close to collecting nothing. The expensive option, more groups, is the one that actually narrows the interval.
Assumption. Groups are roughly equal in size and share one . Unequal groups need a weighted version, and the intuition survives: the effective sample is governed by the number of groups far more than by the rows inside them.
The Split That Hides Its Own Failure
Divide rows at random and the same group lands on both sides. The evaluation that follows reports near-nominal performance with intervals that are too sharp, and passes the check almost everyone runs.
Splitting rows at random answers a different question than the one you asked. Any held-out evaluation asks: how will this perform on data it hasn’t seen? A row split answers how it will perform on new rows from groups it already knows. When groups carry signal, those are not the same question, and the second is much easier.
The leak rarely arrives through a column called group_id, because that column would be dropped. It arrives through an ordinary covariate that happens to be constant within a group, the plate a coupon came from or the machine a part ran on. Those are legitimate features. They are also, in combination, close to an identifier, and a flexible model will use them as one when the rest of the group is sitting in the training set.
The simulation below has known truth, built from 4 pieces: a within-group index, one group-level covariate, a per-group offset, and noise. Both constructions are scored on the same unseen groups using the same predictions, so the only thing that differs between them is where the calibration rows came from.
24 groups of 10, 200 repetitions per point, nominal 90%. As the intraclass correlation rises, splitting by group holds its promise and splitting by row drifts away from it. Both lines are scored on groups neither model has seen, using identical predictions: the only thing that differs is where the calibration rows came from.
At the 2 lines agree. That is the control, and it is the only point on the sweep where the row split is honest. As rises the group-level split holds near its promise while the row-level split falls away from it, reaching 52.8% coverage on a 90% claim at .
Now the widths. The row-split interval barely widens across the whole sweep, 3.86 to 4.22, while the honest interval more than doubles, 3.97 to 10.11. The tell is narrowness, and narrowness reads as good news. The contaminated calibration set reports small errors because it is scoring rows whose groups the model has already met, so the interval built from it stays narrow while the world it will be used in gets harder.
The standard defence is to verify performance on the held-out data and confirm it matches the promise. Against this failure that defence is worthless, because the held-out data is contaminated in the same way. The row-split construction passes its own validation. It has to: the calibration rows and the evaluation rows have the same relationship to the training set.
Detecting it requires a split the check itself cannot supply. You have to know the grouping, and impose it.
Assumption. That the grouping used for the split is the real one. Splitting by a grouping finer than the truth, sites when the real unit is regions, leaks exactly as before.
Intervals: Resample Groups, Not Rows
The bootstrap fixes itself if you resample the thing that actually repeats.
Every standard interval formula is too narrow, for the reason in § 02: they take at face value. There are 2 usable repairs, and they differ in what they need from you.
The first is to compute the design effect and substitute neff, which requires estimating . The second needs no estimate at all.
To put an interval on any statistic computed from clustered data: draw groups at random with replacement from the you have, take all the rows belonging to each drawn group, compute the statistic on the assembled sample, and repeat a few thousand times. The 2.5th and 97.5th percentiles of the resulting values are a 95% interval.
The whole modification is which object gets resampled. Resampling rows treats each row as an independent draw and reproduces the too-narrow answer. Resampling groups treats each group as the independent draw, which is what it is.
The cluster bootstrap is attractive in practice because it requires no model of the dependence. Whatever happens to be, whatever shape the within-group correlation takes, resampling groups carries it along automatically. What it needs instead is groups: with 8 or 10 of them the intervals are honest and wide, and with 4 they are barely worth computing.
The number of groups, not the number of rows, is the budget you are actually spending.
Assumption. Groups are exchangeable with one another. The bootstrap resamples them as interchangeable units, so a design with 1 obviously atypical group, a tool run 3 times longer than the rest, will produce intervals whose width depends visibly on whether that group was drawn.
Choosing the Unit, and What This Cannot Fix
Everything above depends on one judgement no procedure can make for you.
No procedure picks the grouping unit, and no sweep will find it for you. It is a claim about how the data was produced, made by whoever knows the process, and the person running the analysis is usually not that person. 4 questions settle it.
- What physical thing was reused across rows? 6 candidates cover most cases: a tool, a plate, a fixture, a person, a machine, a shift.
- If 2 rows share it, would you expect them to be alike for reasons the model does not know about?
- At what level will the eventual decision be made? If the answer is per batch, evaluating per row answers a question nobody asked.
- Is there a coarser grouping above the obvious one? Tools sit in machines, and machines sit in sites. Pick the coarsest level at which resemblance is still plausible.
Question 4 is where most of the remaining errors live. Split by tool when the real shared cause was the material batch behind 10 tools, and that batch sits on both sides of the split, exactly as it would have if you had never grouped at all.
3 things this framework does not fix.
- It does not recover information. Everything here makes the reported uncertainty match the actual uncertainty. That means intervals get wider and estimates get less impressive. Nothing above will make a small study bigger.
- It does not handle overlapping groupings. When rows are grouped by operator and by machine and neither nests inside the other, a single split cannot honour both. That is a crossed-effects problem and it needs different machinery.
- It does not detect the grouping for you. No diagnostic reads a table and reports that rows 41 to 60 shared a fixture. If nobody says so, the analysis proceeds as though nobody needed to.
The third has a consequence worth stating plainly. A clustered dataset analysed by someone who was not told it was clustered will produce tighter intervals and better-looking numbers than an honest analysis of the same data. Compare 2 sets of results and the sharper ones belong to whoever asked fewer questions about where the rows came from. You should now be able to name the group and price what it costs you. Before you quote an interval, ask what the rows shared.