Collecting data

Print the chapter

Every lesson of the chapter, ready for paper. Exercise sets are lettered A, B, C… and answered at the end.

To save a PDF, choose “Save as PDF” in the print dialog.

1. Populations, samples, parameters and statistics

Warm-up

Three quick questions from earlier lessons.

Discover

A poll of 1,000 voters finds 52% support for a candidate. Is 52% the support among all voters?

  1. Not exactly: it estimates it
  2. Yes

Key idea

We study a sample to learn about a population.

The population is the whole group of interest; a sample is the part we actually observe. A parameter is a number describing the population (usually unknown); a statistic is a number computed from the sample, used to estimate the parameter.

Worked example

A factory tests 200 of today's 10,000 bulbs; 3% are faulty.

Population: the 10,000 bulbs
Everything produced today.
Sample: the 200 bulbs
Those tested.
3% is a statistic estimating the faulty rate today
Computed from the sample.

Your turn, guided

Your turn: how many faulty bulbs were found in the sample?

66
3% of 200.

Practice ★

  1. A researcher measures the sleep of 376 randomly chosen teenagers and finds a mean of 7.1 hours. What is 7.1?
    1. A statistic: it describes the sample
    2. A parameter: it describes all teenagers
    3. The population
  1. To learn how much US college students spend on books, a team surveys 671 students at 20 colleges. What is the population?
    1. All US college students
    2. The 671 students surveyed
    3. The 20 colleges

Watch out!

Practice ★★

  1. Census records show that 52% of all households in a town own a car. What is 52%?
    1. A parameter: it describes the whole population
    2. A statistic
    3. A sample proportion

Exit ticket

Three questions to check you have got it.

  1. A researcher measures the sleep of 376 randomly chosen teenagers and finds a mean of 7.1 hours. What is 7.1?
    1. A statistic: it describes the sample
    2. A parameter: it describes all teenagers
    3. The population
  2. A researcher measures the sleep of 288 randomly chosen teenagers and finds a mean of 7.1 hours. What is 7.1?
    1. A statistic: it describes the sample
    2. A parameter: it describes all teenagers
    3. The population
  3. A researcher measures the sleep of 121 randomly chosen teenagers and finds a mean of 7.1 hours. What is 7.1?
    1. A statistic: it describes the sample
    2. A parameter: it describes all teenagers
    3. The population

Summary card

Population and sample

  • Population → parameter (usually unknown).
  • Sample → statistic (an estimate).

Watch out: a number from the sample is a statistic.

2. Sampling methods: simple random, stratified, cluster, systematic

Warm-up

Three quick questions from earlier lessons.

Discover

A school has 600 juniors and 400 seniors. You want a sample of 50 that reflects both grades. How many juniors?

  1. 30
  2. 25

Key idea

Random sampling methods.

Simple random sample (SRS): every group of nn individuals is equally likely. Stratified: split into similar groups (strata) and take an SRS from each. Cluster: split into groups (clusters), choose some clusters at random, take everyone in them. Systematic: a random start, then every kkth individual.

Worked example

A list of 2,400 members; a systematic sample of 80.

Step k=30k = 30
2,400 ÷ 80.
Start at, say, member 17
Random start among the first 30.
17, 47, 77, … up to 80 members
Then add 30 each time.

Your turn, guided

Your turn: 900 members, systematic sample of 45. The step.

k=20k = 20
900 ÷ 45.

Practice ★

  1. A school has 600 juniors and 300 seniors. A stratified sample of 90 students is taken in proportion to grade. How many juniors are in the sample?
  2. A school has 500 juniors and 600 seniors. A stratified sample of 110 students is taken in proportion to grade. How many juniors are in the sample?
  1. A systematic sample of 30 customers is taken from a list of 480: a random start, then every kkth name. What is kk?

Watch out!

Practice ★★

  1. A district has 60 classrooms. Five are chosen at random and every student in them is surveyed. Which sampling method is this?
    1. Cluster sampling
    2. Stratified sampling
    3. A simple random sample of students
  1. Boss problem. A town has 300 houses in the north and 600 in the south. A stratified sample of 180 houses is planned, in proportion to each area.

Exit ticket

Three questions to check you have got it.

  1. A school has 600 juniors and 300 seniors. A stratified sample of 90 students is taken in proportion to grade. How many juniors are in the sample?
  2. A school has 500 juniors and 600 seniors. A stratified sample of 110 students is taken in proportion to grade. How many juniors are in the sample?
  3. A school has 900 juniors and 800 seniors. A stratified sample of 170 students is taken in proportion to grade. How many juniors are in the sample?

Summary card

Sampling methods

SRS · stratified (some from every group) · cluster (all from some groups) · systematic (every kkth).

Watch out: proportional allocation in stratified samples.

3. Bias in sampling

Warm-up

Three quick questions from earlier lessons.

Discover

A TV show asks viewers to text in their opinion; 50,000 do. Is that a better sample than 1,000 randomly chosen adults?

  1. No, the random 1,000 is better
  2. Yes, it is 50 times bigger

Key idea

Bias is a systematic error from how data are collected.

Voluntary response: people choose to respond. Convenience sampling: easy-to-reach individuals. Undercoverage: part of the population cannot be chosen. Nonresponse: chosen people do not answer. Response bias: wording or setting distorts answers. A larger sample does not reduce bias.

Worked example

A survey by phone at 2 p.m. on weekdays.

Working adults are rarely reached
Who is home?
Undercoverage
They cannot be in the sample.
The bias does not shrink with more calls
The estimate leans towards people at home.

Your turn, guided

Your turn: of 1,200 people chosen, 300 answered. The response rate as a fraction:

14\tfrac{1}{4}
300 ÷ 1,200: a high risk of nonresponse bias.

Practice ★

  1. A news website asks readers to vote online; 4824 vote and 80% favor a new law. What is the main problem?
    1. Voluntary response bias: people with strong opinions choose to respond
    2. The sample is too small
    3. None: 4824 responses are representative
  1. A survey of 115 adults is done by calling landline numbers. What kind of bias is most likely?
    1. Undercoverage: people without landlines cannot be chosen
    2. Response bias from the wording
    3. None, if the numbers are chosen at random

Watch out!

Practice ★★

  1. A biased survey of 238 people is repeated with 10 times as many people, chosen the same way. What happens to the bias?
    1. It stays: a larger sample reduces random variability, not bias
    2. It disappears
    3. It becomes 10 times smaller
  2. A biased survey of 402 people is repeated with 10 times as many people, chosen the same way. What happens to the bias?
    1. It stays: a larger sample reduces random variability, not bias
    2. It disappears
    3. It becomes 10 times smaller

Exit ticket

Three questions to check you have got it.

  1. A news website asks readers to vote online; 4824 vote and 80% favor a new law. What is the main problem?
    1. Voluntary response bias: people with strong opinions choose to respond
    2. The sample is too small
    3. None: 4824 responses are representative
  2. A news website asks readers to vote online; 1714 vote and 80% favor a new law. What is the main problem?
    1. Voluntary response bias: people with strong opinions choose to respond
    2. The sample is too small
    3. None: 1714 responses are representative
  3. A news website asks readers to vote online; 3716 vote and 80% favor a new law. What is the main problem?
    1. Voluntary response bias: people with strong opinions choose to respond
    2. The sample is too small
    3. None: 3716 responses are representative

Summary card

Bias

Voluntary response, convenience, undercoverage, nonresponse, response bias.

Watch out: a bigger biased sample is still biased.

4. Observational studies, experiments and confounding

Warm-up

Three quick questions from earlier lessons.

Discover

Cities with more firefighters at a fire have more damage. Do firefighters cause damage?

  1. No, bigger fires explain both
  2. Yes, the data show it

Key idea

Only experiments can show cause and effect.

In an observational study we record what happens without intervening; in an experiment we impose treatments. A confounding variable is related to both the explanatory and the response variable, so its effect cannot be separated. Observational studies show associations, not causation.

Worked example

Students who take music lessons have higher grades.

Observational study
Nobody assigned the lessons.
Possible confounder: family income
Affects both lessons and grades.
Conclusion: association only
To test cause, assign lessons at random.

Your turn, guided

Your turn: 80 volunteers are split at random, 40 per group. How many get the treatment?

4040
Half of 80.

Practice ★

  1. Researchers randomly assign 66 volunteers to drink coffee or water, then test their reaction times. What kind of study is it?
    1. An experiment: the researchers assign the treatment
    2. An observational study
    3. A sample survey
  1. Children with bigger shoes read better (16% higher scores). What is the most likely explanation?
    1. A confounding variable: age affects both shoe size and reading
    2. Bigger feet improve reading
    3. It must be pure chance

Watch out!

Practice ★★

  1. An observational study of 3416 adults finds that those who eat breakfast weigh less. Does this show that eating breakfast causes lower weight?
    1. No: other differences between the groups could explain it
    2. Yes, because the sample is large
    3. Yes, if the association is strong
  2. An observational study of 5700 adults finds that those who eat breakfast weigh less. Does this show that eating breakfast causes lower weight?
    1. No: other differences between the groups could explain it
    2. Yes, because the sample is large
    3. Yes, if the association is strong

Exit ticket

Three questions to check you have got it.

  1. Researchers randomly assign 66 volunteers to drink coffee or water, then test their reaction times. What kind of study is it?
    1. An experiment: the researchers assign the treatment
    2. An observational study
    3. A sample survey
  2. Researchers randomly assign 166 volunteers to drink coffee or water, then test their reaction times. What kind of study is it?
    1. An experiment: the researchers assign the treatment
    2. An observational study
    3. A sample survey
  3. Researchers randomly assign 46 volunteers to drink coffee or water, then test their reaction times. What kind of study is it?
    1. An experiment: the researchers assign the treatment
    2. An observational study
    3. A sample survey

Summary card

Studies

  • Observational: record. Experiment: assign.
  • Confounding hides cause in observational data.

Watch out: association is not causation.

5. Designing experiments: randomization, replication, blocking

Warm-up

Three quick questions from earlier lessons.

Discover

To compare two fertilizers, you put fertilizer A on the sunny half of a field and B on the shady half. A grows better. What is the flaw?

  1. Sunlight, not the fertilizer, could explain the difference
  2. There is no flaw

Key idea

Principles of experimental design.

Comparison of treatments (often with a control or placebo group); random assignment of units to treatments; replication on enough units; blocking groups similar units first and randomizes within each block. With several factors, the treatments are all combinations of their levels.

Worked example

Test 2 doses × 3 diets on 60 rats.

Treatments: 2×3=62 \times 3 = 6
Each combination.
10 rats per treatment
Replication.
Shuffle the rats into the 6 groups
Random assignment.

Your turn, guided

Your turn: 3 temperatures × 4 times. Number of treatments.

1212
3 × 4.

Practice ★

  1. Why are 196 subjects assigned to the treatments at random?
    1. To balance other variables between the groups, so differences can be blamed on the treatment
    2. So the results apply to the whole population
    3. To give every subject the same chance of being in the study
  1. An experiment tests 2 fertilizer types at 5 watering levels, with every combination used. How many treatments are there?
  2. An experiment tests 3 fertilizer types at 4 watering levels, with every combination used. How many treatments are there?

Watch out!

Practice ★★

  1. A test of two diets on 115 adults first splits them into men and women, then randomizes diets within each group. What is this design?
    1. A randomized block design, blocking by sex
    2. A stratified random sample
    3. A completely randomized design

Exit ticket

Three questions to check you have got it.

  1. An experiment tests 2 fertilizer types at 5 watering levels, with every combination used. How many treatments are there?
  2. An experiment tests 3 fertilizer types at 4 watering levels, with every combination used. How many treatments are there?
  3. An experiment tests 4 fertilizer types at 4 watering levels, with every combination used. How many treatments are there?

Summary card

Experimental design

Compare · randomize · replicate · block. Treatments = combinations of factor levels.

Watch out: blocks (experiments) vs strata (samples).

6. Scope of inference: generalization and causation

Warm-up

Three quick questions from earlier lessons.

Discover

A randomized experiment on 50 volunteer athletes shows a supplement improves sprint times. Can you say it helps all adults?

  1. No, only for these volunteers
  2. Yes, it was a randomized experiment

Key idea

Two kinds of randomness, two kinds of conclusions.

Random sampling → results can be generalized to the population. Random assignment → differences can be attributed to the treatment (cause and effect). Both → causal conclusions about the population. Blinding (single or double) and placebos keep expectations from affecting the results.

Worked example

Four studies.

Volunteers, no assignment: association in these people only
Neither.
Random sample, no assignment: association in the population
Sampling only.
Volunteers, random assignment: cause, in these subjects
Assignment only.

Your turn, guided

Your turn: how many of the four combinations (sampling yes/no × assignment yes/no) allow a causal conclusion about the whole population?

11
Only both random sampling and random assignment.

Practice ★

  1. A random sample of 493 adults in a city shows that those who walk to work report less stress. No treatment was assigned. What can be concluded?
    1. The association can be generalized to adults in the city, but not cause and effect
    2. Walking causes less stress for adults in the city
    3. Nothing beyond the 493 people sampled
  1. 147 volunteers are randomly assigned to a new or old teaching method; the new method scores higher. What can be concluded?
    1. The method caused the difference for these volunteers; generalizing beyond them is risky
    2. The new method works better for all students
    3. Nothing, because there was no random sample

Watch out!

Practice ★★

  1. In a drug trial with 64 patients, neither the patients nor the doctors who assess them know who receives the drug. What is this called?
    1. A double-blind experiment
    2. A placebo
    3. Blocking

Exit ticket

Three questions to check you have got it.

  1. A random sample of 493 adults in a city shows that those who walk to work report less stress. No treatment was assigned. What can be concluded?
    1. The association can be generalized to adults in the city, but not cause and effect
    2. Walking causes less stress for adults in the city
    3. Nothing beyond the 493 people sampled
  2. A random sample of 530 adults in a city shows that those who walk to work report less stress. No treatment was assigned. What can be concluded?
    1. The association can be generalized to adults in the city, but not cause and effect
    2. Walking causes less stress for adults in the city
    3. Nothing beyond the 530 people sampled
  3. A random sample of 712 adults in a city shows that those who walk to work report less stress. No treatment was assigned. What can be concluded?
    1. The association can be generalized to adults in the city, but not cause and effect
    2. Walking causes less stress for adults in the city
    3. Nothing beyond the 712 people sampled

Summary card

Scope of inference

  • Random sampling → generalize.
  • Random assignment → cause.

Watch out: you need both for a causal claim about a population.

Answers: Collecting data

1. Populations, samples, parameters and statistics

A

  1. A statistic: it describes the sample

B

  1. All US college students

C

  1. A parameter: it describes the whole population

D

  1. A statistic: it describes the sample
  2. A statistic: it describes the sample
  3. A statistic: it describes the sample

2. Sampling methods: simple random, stratified, cluster, systematic

A

  1. 6060
  2. 5050

B

  1. 1616

C

  1. Cluster sampling

D

  1. 120120

E

  1. 6060
  2. 5050
  3. 9090

3. Bias in sampling

A

  1. Voluntary response bias: people with strong opinions choose to respond

B

  1. Undercoverage: people without landlines cannot be chosen

C

  1. It stays: a larger sample reduces random variability, not bias
  2. It stays: a larger sample reduces random variability, not bias

D

  1. Voluntary response bias: people with strong opinions choose to respond
  2. Voluntary response bias: people with strong opinions choose to respond
  3. Voluntary response bias: people with strong opinions choose to respond

4. Observational studies, experiments and confounding

A

  1. An experiment: the researchers assign the treatment

B

  1. A confounding variable: age affects both shoe size and reading

C

  1. No: other differences between the groups could explain it
  2. No: other differences between the groups could explain it

D

  1. An experiment: the researchers assign the treatment
  2. An experiment: the researchers assign the treatment
  3. An experiment: the researchers assign the treatment

5. Designing experiments: randomization, replication, blocking

A

  1. To balance other variables between the groups, so differences can be blamed on the treatment

B

  1. 1010
  2. 1212

C

  1. A randomized block design, blocking by sex

D

  1. 1010
  2. 1212
  3. 1616

6. Scope of inference: generalization and causation

A

  1. The association can be generalized to adults in the city, but not cause and effect

B

  1. The method caused the difference for these volunteers; generalizing beyond them is risky

C

  1. A double-blind experiment

D

  1. The association can be generalized to adults in the city, but not cause and effect
  2. The association can be generalized to adults in the city, but not cause and effect
  3. The association can be generalized to adults in the city, but not cause and effect