Sampling Bias
Sampling bias is a systematic distortion in a study’s results that arises because the people (or cases) in the data differ from the population the study claims to describe, in ways that were not accounted for, through how the sample was selected or who ended up taking part. Unlike random sampling error, sampling bias does not shrink as the sample gets larger.
For a dissertation, the practical question is not whether your sample is perfect. No sample is. The question is:
Which groups could be missing from my data, why are they missing, and does their absence change the conclusions I want to draw?
This page explains where sampling bias enters a study, how it differs from sampling error, how each sampling method is exposed to it, and how to detect, reduce and report it in a methodology chapter.
On this page:
- Sampling bias explained simply
- What sampling bias is
- Sampling bias vs sampling error
- Where sampling bias enters a study
- Main types of sampling bias
- Sampling bias by sampling method
- How to detect sampling bias
- How to reduce sampling bias
- Sampling bias in qualitative research
- Application example
- Advantages and limitations of correcting for sampling bias
- Common mistakes
- Sampling bias in the age of AI and digital research
- Dissertation example
- Exam tip
Sampling Bias Explained Simply
Imagine you want to know how satisfied employees in a large retail company are with their working hours. You post a survey link on the company’s LinkedIn page and receive 300 responses.
The problem is who saw the link and who chose to answer. Office staff who follow the company page are more likely to see it than shop-floor staff working shifts. Employees with strong opinions, positive or negative, are more likely to respond than those who feel neutral. Your 300 responses may describe the people who were reachable and motivated, not the company’s workforce.
Collecting 3,000 responses through the same link would not solve this. You would get a more precise estimate of the wrong group.
What Is Sampling Bias?
Sampling bias is a systematic error in which estimates from a sample differ predictably from the true population values, because some members of the target population were more or less likely to end up in the data in ways that were not known or not accounted for, and that relate to the variables being studied.
Unequal chances of inclusion are not, by themselves, bias. A disproportionate stratified sample deliberately over-samples small groups, and weighting in the analysis restores their population shares; the resulting estimates can be perfectly defensible. The problem arises when differences in who ends up in the data are unknown or not adjusted for, and are related to what you are measuring. If left-handed people are slightly under-represented in a survey about pension savings, the distortion is unlikely to matter. If people on low incomes are under-represented in the same survey and nothing is done about it, it almost certainly does.
Survey methodologists treat sampling bias as part of a wider family of errors, known as total survey error, that separates errors of representation (who is in the data) from errors of measurement (whether their answers are accurate) (Groves et al., 2009). Sampling bias belongs to the first family. This distinction matters, because problems such as recall bias or social desirability bias are measurement problems, not sampling problems, and they are solved differently.
Sampling Bias vs Sampling Error
Sampling error is the random difference between a sample estimate and the true population value that occurs simply because a sample, not the whole population, was studied. Sampling bias is a systematic difference caused by how the sample was obtained. Sampling error shrinks as sample size grows; sampling bias does not.
| Feature | Sampling error | Sampling bias |
|---|---|---|
| Nature | Random: estimates scatter around the true value | Systematic: estimates are pushed in one direction |
| Effect of a larger sample | Becomes smaller | Stays the same, or becomes more convincing while still wrong |
| Can it be quantified from the sample itself? | Yes, through standard errors and confidence intervals (probability samples) | No; it needs outside information such as population benchmarks |
| Main remedy | Larger sample, efficient design such as stratification | Better sampling frame, random selection, higher and more even participation, weighting |
The best-known illustration is the Literary Digest poll of the 1936 US presidential election. The magazine mailed about 10 million ballots, received nearly 2.4 million back, and predicted a clear win for Alf Landon. Franklin Roosevelt won with about 61% of the popular vote (Lusinchi, 2012). A sample of 2.4 million was no protection against bias.
The usual textbook explanation is that the Digest sampled from telephone directories and car registration lists, which over-represented wealthier Republican voters. Later analyses challenged this. Squire (1988) found that car and telephone owners still favoured Roosevelt and concluded that the failure came from both the initial sample and nonresponse. Lusinchi (2012) placed most of the weight on nonresponse: Landon supporters returned their ballots at much higher rates than Roosevelt supporters. The lesson for a dissertation is that bias can enter after the sample is drawn, through who chooses to respond, not only through the list you sample from.
Where Sampling Bias Enters a Study
Sampling bias can enter at four points between the population you want to describe and the data you finally analyse. Each point is a potential source of distortion, not an inevitable loss: a well-designed study may lose very few people at some stages, and ordinary random selection of a sample is not a defect. Each gap has its own causes and remedies.
Figure: Where sampling bias can enter a study. An illustrative classification and the author’s synthesis, building on the total survey error framework (Groves et al., 2009).
| Gap | What happens | Typical cause | Question to ask |
|---|---|---|---|
| 1. Coverage gap | The sampling frame (the list or channel you sample from) does not contain the whole target population | Outdated lists, online-only recruitment, organisational access limited to some sites | Who in my population could never have been selected? |
| 2. Selection gap | Some members of the frame have a higher chance of being chosen, in ways that are unknown or not adjusted for in the analysis | Convenience selection, self-selection, interviewer choice, gatekeeper choice | Did I, someone else or the participants themselves decide who took part, in ways I cannot account for? |
| 3. Nonresponse gap | Some selected people do not take part, and they differ from those who do | Low interest, lack of time, distrust, language, survey length | Are the people who declined likely to answer differently? |
| 4. Attrition and exclusion gap | Participants drop out later, or cases are removed during data cleaning | Dropout in multi-stage studies, excluding incomplete responses, survivorship | Who did I lose after data collection began, and why? |
Thinking in terms of these four gaps is more useful in a methodology chapter than listing named biases, because it tells you where to look for evidence and what you can realistically claim. A study can have a small coverage gap but a large nonresponse gap, or the reverse.
The classification is illustrative rather than strict. Survivorship bias, for example, often enters at the frame stage: a study of business success that samples only from a list of currently trading firms has a coverage problem before anyone is contacted.
Main Types of Sampling Bias
The named types of sampling bias found in textbooks are specific forms of the four gaps described above. Undercoverage is a coverage problem; self-selection and convenience bias are selection problems; nonresponse bias is a response problem; exclusion bias is an attrition problem; and survivorship bias can arise either at the coverage stage (a frame that contains only survivors) or through attrition.
| Type | Definition | Gap | Example |
|---|---|---|---|
| Undercoverage bias | Part of the population is missing from the sampling frame | Coverage | An online survey of older adults that excludes those without internet access |
| Self-selection (volunteer) bias | Participants decide for themselves whether to take part, and volunteers differ from non-volunteers | Selection | An open web link about workplace stress that attracts mainly highly stressed employees |
| Convenience bias | The sample consists of people who were easiest to reach | Selection | Surveying classmates about the career intentions of all business students |
| Gatekeeper bias | A manager or organisation decides who is invited | Selection | A manager nominates “suitable” employees for interviews about leadership |
| Nonresponse bias | Those who do not respond differ systematically from those who do | Nonresponse | Customers with complaints are more likely to return a satisfaction survey |
| Survivorship bias | Only cases that “survived” a selection process are observed | Coverage, or attrition and exclusion | Studying success factors only among firms that are still trading |
| Exclusion bias | Cases are removed from the sample in a way related to the outcome | Attrition and exclusion | Deleting all incomplete questionnaires when busier respondents were more likely to stop early |
A research sample can also be biased at the level of whole populations. Henrich, Heine and Norenzayan (2010) showed that much behavioural science draws its participants from Western, educated, industrialised, rich and democratic (WEIRD) societies, often university students, and then generalises findings to humans in general. The same reasoning applies to a dissertation that samples students at one university and draws conclusions about “young consumers”.
Sampling Bias by Sampling Method
Every sampling method has a typical route by which bias can enter. Probability methods give every member of the frame a known chance of selection, which removes selection bias in principle, but they remain exposed to coverage and nonresponse bias. Non-probability methods do not provide known selection probabilities, so their implications depend on the purpose of the study: they are a serious limitation when the aim is to estimate population values, and much less so when the aim is to explore, explain or study information-rich cases.
The table also distinguishes bias from instability. Some design weaknesses mainly make estimates less precise rather than systematically wrong, and they are marked as such.
| Sampling method | Main route to bias | How to detect it | How to reduce it |
|---|---|---|---|
| Simple random sampling | Incomplete sampling frame; nonresponse after selection | Compare frame with population records; compare respondents with the frame | Update the frame; send reminders; weight by known characteristics |
| Systematic sampling | Periodicity: a cyclical pattern in the list coincides with the sampling interval. With a random start this mainly makes estimates unstable (high variance); with a non-random start it can also bias them | Inspect how the list is ordered before sampling | Randomise the list order or choose an interval that does not match any cycle |
| Stratified sampling | Wrong or outdated stratum sizes; nonresponse differing by stratum | Compare achieved stratum sizes with planned ones | Use current population figures; weight strata to their population shares |
| Cluster sampling | Clusters chosen for access rather than at random (bias). Using only a few randomly selected clusters mainly reduces precision rather than causing bias | Compare selected clusters with all clusters on known features | Select clusters randomly; increase the number of clusters |
| Convenience sampling | Selection of accessible people who differ from the wider population | Compare sample profile with population statistics | Recruit across several settings; narrow the claimed population to match the sample |
| Purposive sampling | Researcher’s judgement favours certain cases | Record selection criteria and who met them but was not chosen | State explicit criteria in advance; seek disconfirming or contrasting cases |
| Snowball sampling | Referral chains stay within one social network | Map referral chains and starting points | Start several independent chains with different seeds |
| Quota sampling | Quotas match on a few characteristics, but selection within quotas is not random | Compare on characteristics not used for quotas | Use quotas on variables related to the outcome; vary recruitment locations and times |
| Open online survey link | Undercoverage of people offline plus self-selection of volunteers | Compare with population benchmarks; check where responses came from | Recruit through several channels; weight; restrict claims (Bethlehem, 2010) |
The comparison shows why probability sampling is preferred when a study aims to estimate population values, and why non-probability sampling requires more careful limits on what is claimed (Saunders, Lewis and Thornhill, 2023; Etikan, Musa and Alkassim, 2016).
How to Detect Sampling Bias
Sampling bias cannot be measured from the sample alone, because the people who are missing left no data. Detection therefore depends on comparing your sample with outside information about the population, or with information about the people who were selected but did not take part.
Four practical checks are available to most dissertation students:
- Benchmark comparison. Compare the age, gender, role, region or other known characteristics of your respondents with official statistics, organisational records or published population data. Large differences on variables related to your research question are a warning sign.
- Response rate by group. Where you know who was invited, calculate the response rate separately for each group (for example, each department or site). Uneven response is more informative than the overall rate.
- Early versus late respondents. Compare answers from people who responded quickly with those who responded only after reminders. Armstrong and Overton (1977) proposed that late respondents tend to resemble nonrespondents, so differences between early and late waves may suggest the direction of nonresponse bias. The check rests on that assumption, which may not hold, and finding no significant difference does not show that bias is absent.
- Frame audit. Before collecting data, list who your sampling frame or recruitment channel cannot reach. This is often the simplest way to identify coverage bias.
A high response rate is reassuring but not proof of an unbiased sample. In a meta-analysis of 59 studies, Groves and Peytcheva (2008) found very little correlation between nonresponse rates and the nonresponse bias actually measured. What matters is whether nonresponse is related to the variables you study.
How to Reduce Sampling Bias
Sampling bias is best reduced at the design stage, before data are collected, because statistical corrections afterwards can only adjust for characteristics you have measured. The most effective steps are improving the sampling frame, using random selection where possible, and making participation easier for groups who are least likely to respond.
| Stage | Action | Gap addressed |
|---|---|---|
| Design | Define the target population precisely and choose a frame that covers it | Coverage |
| Design | Use random selection or stratification where the research question requires population estimates | Selection |
| Design | Recruit through several channels rather than one | Coverage, selection |
| Data collection | Send reminders, keep the questionnaire short, offer more than one mode (online and paper, or a phone option) | Nonresponse |
| Data collection | Monitor response by group and target follow-up at under-represented groups | Nonresponse |
| Analysis | Weight the data to known population proportions (post-stratification) | Coverage, nonresponse |
| Analysis | Handle incomplete responses with a stated rule rather than deleting them by default | Attrition and exclusion |
| Reporting | Narrow the population to which findings are generalised so that it matches the achieved sample | All |
Weighting deserves caution. It corrects imbalances only on the variables used to build the weights. If volunteers in an online survey differ from non-volunteers in ways that are not captured by age, gender or region, weighting by those variables will not remove the bias (Bethlehem, 2010). Lohr (2021) provides a fuller treatment of weighting and nonresponse adjustment for readers who need it.
Sampling Bias in Qualitative Research
In qualitative research, sampling bias is usually discussed in terms of whose perspectives are missing rather than whether the sample statistically represents a population. Qualitative studies rarely aim for statistical generalisation, so a small purposive sample is not biased simply because it is small or non-random.
The relevant risk is different. If a qualitative study of employee experiences of remote working interviews only participants nominated by HR, the findings may systematically omit dissatisfied employees. The study can still be valuable, but its account of “employee experiences” is partial in a predictable direction.
In a qualitative methodology chapter, address this by stating your selection criteria, explaining how participants were recruited and by whom, describing who was approached but declined where known, and discussing whose perspectives may be missing. This supports the transferability of your findings rather than claiming representativeness.
Application of Sampling Bias: an Example
Hypothetical example. A hospital wants to evaluate patients’ experience of a new process for discharging patients to their homes. The target population is therefore all patients discharged home during the study period. A researcher hands a questionnaire to patients on the day of discharge over three months and achieves a 70% response rate.
Applying the four gaps reveals several risks. The coverage gap appears if the questionnaire is handed out only on weekdays, so patients discharged at weekends, when staffing and procedures may differ, can never be selected. The nonresponse gap may exclude patients who were too unwell or too rushed to complete the form at discharge. The exclusion gap appears if forms with missing pages are discarded, because patients with poorer literacy or vision may be more likely to leave pages blank.
A stronger design would use the hospital’s discharge records for all patients discharged home as the sampling frame, offer a phone follow-up after discharge, compare respondents with all patients discharged home on age, ward and length of stay, and apply a stated rule to incomplete forms. Patients transferred to other facilities fall outside this target population and would need a separate study if their experience matters.
Advantages and Limitations of Correcting for Sampling Bias
Addressing sampling bias explicitly strengthens a dissertation in a way examiners notice. It shows that the researcher understands the difference between collecting data and collecting evidence, and it protects conclusions from the most common criticism of student research, namely that the sample does not support the generalisation. Checks such as benchmark comparison and wave analysis are cheap and can usually be completed with data the student already holds.
The limitations are real, however. Detection methods only reveal differences on characteristics that can be observed, so a sample can match the population on age and gender while still differing on attitudes. Weighting can increase the variance of estimates and can give a false sense of correction. For many dissertations, especially those using convenience samples, the honest conclusion is that bias cannot be removed, only described, and that the claims must be narrowed accordingly.
Common Mistakes When Dealing With Sampling Bias
- Treating a large sample as unbiased. Size reduces random error, not systematic error.
- Calling a convenience sample “random”. Asking whoever is available is not random sampling, even if no one was deliberately chosen.
- Relying on the response rate alone. A high response rate does not guarantee that respondents resemble nonrespondents.
- Confusing sampling bias with measurement bias. Recall bias and social desirability bias concern the accuracy of answers, not who answered.
- Generalising beyond the sampling frame. Findings from one organisation, region or university describe that setting unless there is evidence that it resembles others. This is a question of external validity.
- Listing bias as a limitation without consequences. A limitation section should say how the bias could affect the results and, where it can be judged, in which direction; if the direction is unknown, say so.
Sampling Bias in the Age of AI and Digital Research
Digital data collection does not remove sampling bias; it changes where the bias comes from. Online surveys, panels and social media make it easy to collect large samples quickly, but a large online sample still describes only the people who can be reached online and who choose to take part.
Web surveys are exposed to undercoverage (people without internet access or outside the recruitment channel cannot be selected) and to self-selection (Bethlehem, 2010). Opt-in panels are non-probability samples, so inference from them depends on modelling assumptions that need to be stated and, where possible, tested (Baker et al., 2013). Online surveys also face a newer threat: automated bots and fraudulent respondents who complete questionnaires to claim incentives, which distorts the data in ways that sample size cannot repair (Storozuk et al., 2020).
Social media data raises the same issue in another form. Platforms attract particular demographic groups, their algorithms shape what is visible, and some accounts are automated or commercial, so findings may describe the platform rather than the population (Ruths and Pfeffer, 2014).
A newer question is whether large language models can stand in for survey respondents. Bisbee et al. (2024) compared synthetic responses with national survey data and found that the average answers could look plausible while the variation was too narrow, regression results often differed from those based on human respondents, and the outputs changed with prompt wording, model version and the date of data collection. For a dissertation, AI-generated respondents are therefore better treated as a topic of study or a pilot aid than as a substitute for a sample.
In a methodology chapter, state the recruitment channel or platform, how respondent quality and possible bots were checked, who was likely to be excluded, and how this limits generalisation.
Dissertation Example
The following paragraph shows how sampling bias can be discussed in a methodology chapter. The study is hypothetical.
“This study used a convenience sample of 214 undergraduate business students recruited through course announcements at one UK university. Because participation was voluntary and limited to one institution, the sample is exposed to coverage bias (students at other institutions and in other disciplines could not be selected) and to self-selection bias (students with stronger interest in entrepreneurship may have been more willing to participate). To assess the extent of these risks, the sample profile was compared with the business school’s published statistics for its undergraduate students by gender, age group and year of study; the sample over-represented final-year students (41% compared with 29%). Responses received before and after the reminder were also compared, following Armstrong and Overton (1977), and showed no significant difference in entrepreneurial intention scores. Because this comparison assumes that late respondents resemble nonrespondents, it does not rule out nonresponse bias. The findings are therefore interpreted as describing the participating students; they may not represent all business undergraduates at the university, and they are not generalised to UK students as a whole.”
This wording names the specific biases, shows what was done to check them, states the limits of those checks and narrows the conclusion to fit the evidence.
Exam Tip
If you are asked to evaluate a study’s sample, do not stop at “the sample may be biased”. Identify which gap the bias entered through (coverage, selection, nonresponse or attrition), state which way it could push the results (or that the direction is unknown), and say what the researcher did or could have done to check it. Mentioning that a larger sample would not fix systematic bias, with the 1936 Literary Digest poll as an example, is a quick way to show you understand the difference between sampling bias and sampling error.
Could your sample limit the conclusions your dissertation can draw?
Dudovskiy Research Assistant can review your topic, population and planned sampling method to identify where bias is likely to enter and how to address it in your methodology chapter.
John Dudovskiy
References
Armstrong, J.S. and Overton, T.S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 14(3), 396–402.
Baker, R., Brick, J.M., Bates, N.A., Battaglia, M., Couper, M.P., Dever, J.A., Gile, K.J. and Tourangeau, R. (2013). Report of the AAPOR Task Force on Non-Probability Sampling. American Association for Public Opinion Research.
Bethlehem, J. (2010). Selection bias in web surveys. International Statistical Review, 78(2), 161–188.
Bisbee, J., Clinton, J.D., Dorff, C., Kenkel, B. and Larson, J.M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis. https://doi.org/10.1017/pan.2024.5
Etikan, I., Musa, S.A. and Alkassim, R.S. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/10.11648/j.ajtas.20160501.11
Groves, R.M., Fowler, F.J., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2009). Survey Methodology. 2nd ed. Hoboken, NJ: Wiley.
Groves, R.M. and Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opinion Quarterly, 72(2), 167–189. https://doi.org/10.1093/poq/nfn011
Henrich, J., Heine, S.J. and Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.
Lohr, S.L. (2021). Sampling: Design and Analysis. 3rd ed. Boca Raton, FL: CRC Press.
Lusinchi, D. (2012). “President” Landon and the 1936 Literary Digest poll: were automobile and telephone owners to blame? Social Science History, 36(1), 23–54. https://doi.org/10.1017/S014555320001035X
Ruths, D. and Pfeffer, J. (2014). Social media for large studies of behavior. Science, 346(6213), 1063–1064.
Saunders, M., Lewis, P. and Thornhill, A. (2023). Research Methods for Business Students. 9th ed. Harlow: Pearson.
Squire, P. (1988). Why the 1936 Literary Digest poll failed. Public Opinion Quarterly, 52(1), 125–133.
Storozuk, A., Ashley, M., Delage, V. and Maloney, E.A. (2020). Got bots? Practical recommendations to protect online survey data from bot attacks. The Quantitative Methods for Psychology, 16(5), 472–481.
Last reviewed: November 2026.