In the present day, as large-scale face-to-face interviews become increasingly difficult to implement, convenience samples obtained through online platforms have become a popular data collection method in survey research, particularly when working in conjunction with the rise of experimental methods.

These samples are typically provided to researchers through the vast membership databases of the survey arms of major e-commerce companies such as Mturk and YouGov. Compared to traditional survey methods, this method has the advantage of speed. Although it is not a traditional probability sample, researchers can use quotas to make the sample more representative of the population they are studying.

While some researchers still have reservations about this type of sample and argue that participants may exhibit specific characteristics that could potentially interfere with survey results (e.g., exclusion of non-internet users), recent experimental literature has confirmed that these concerns are not as worrisome as critics suggest (Coppock, 2017; Coppock et al., 2018). These participants may even be professional respondents who are familiar with online survey procedures and aware of the answers researchers are looking for, leading to what is commonly referred to as "demand effects."

In fact, if the researchers aim to delineate a particular population profile, convenience samples may not be appropriate. However, if the researchers' goal is to test specific theories or causal claims through experiments, convenience samples do have their advantages. The key point is that as long as treatment assignment is as-if random, it naturally guarantees internal validity to obtain the average treatment effect (ATE). In this regard, there is no major problem with the use of convenience samples, as these effects manifest themselves objectively.

Another often-raised question pertains to the external validity of drawing inference from online-recruited convenience samples, which should be considered from several perspectives. Firstly, if the operationalization of the treatment deviates too far from reality, this is not a sample issue, but rather a question of whether the research problem is suitable for experiment design.

Secondly, if there are significant differences in the ATE between the sample and the population, such heterogeneity can be addressed through existing analytical methods, such as mediation analysis, whose finding can inform if sampling weights should be allocated to certain segments of the population shall heterogeneity in ATE existing among the population or, alternatively, buttress the validity of the inference drawn from online-recruited convenience samples if the heterogeneity of the ATE isn’t significant (Hartman et al., 2015). Overall, the benefits brought by convenience samples to the wider discipline of social science are clear, and their disadvantages can be appropriately addressed.