Common Problems in Service-Learning Research

There are a number of problems that are frequently seen in research on service-learning, civic involvement, and community engagement. These are summarized in Table 3 and the discussion below.

• Small sample sizes: Small sample size limits the reliability of the data, making it difficult to have confidence in the results and their implications. Examples: drawing conclusions about all service-learning students by interviewing eight seniors about one class; limiting a study to one section of one course in one semester; conducting a single case study with limited data, and then not conducting cross-case analysis to increase understanding and generalizability. Generally, effects and relationships that are found for larger samples permit more confidence in generalizing from the results to other groups.
• Correlation ≠ Causation: Researchers sometimes conduct a correlational study but draw inappropriate causal (cause and effect) conclusions. For example, correlating hours of service at a site to attitudes about diversity, and then concluding, "serving more hours at a homeless shelter caused students to have more open attitudes about diversity." Without additional evidence or basis for making this causal statement, a more appropriate Service-Learning Research Primer 16 inference would be something like, "students who served more hours at a homeless shelter had higher scores on the diversity scale." Correlations and causal statements are discussed further in the section on Non-Experimental Designs.
• Self-selection bias (non-random assignment): This is one of the most common problems seen in service-learning research. In most colleges and universities, (and often in secondary education) service-learning courses are not required for graduation; in addition, service-learning may be optional in a course. Thus, students select to be in those courses or choose those options. When conducting research, this self-selection of participants into experiences creates the problem of non-random assignment of students to a service-learning group versus a non-service-learning group and confounds the researcher's ability to determine why the students were different at the end of the experience. (See further discussion under Quasi-Experimental Designs.)
• Social desirability bias: This represents a common problem in the measurement of knowledge, skills, attitudes, and behaviors related to service-learning and civic engagement. The difficulty surfaces when the behaviors and attitudes that the research wants to measure are "socially desirable" (e.g., civic-mindedness, social responsibility) and students are inclined to make themselves look good when they present responses. Researchers sometimes try to counteract this bias by including neutral or negativelyworded items in a survey or interview protocol or writing items in ways that control for the bias.
• "Creaming the crop": This problem occurs in research involving only students who are interested in or involved in service-learning, community service, or volunteering. The problem occurs when the investigator over-interprets or over-generalizes the results to draw conclusions about a larger group of students (e.g., all students in freshmen writing, all college students).
• Lack of controls or comparison groups: Many quantitative and qualitative studies do not include adequate control or comparison groups that contrast one intervention (e.g., service-learning) with other interventions (e.g., research paper) in ways that would permit appropriate conclusions. (See more detailed discussion under Experimental and Quasi- Experimental Designs.)
• Lack of generalizability (external validity): In quantitative research, poor research design or sampling issues lead to results that cannot be generalized or applied to other situations or populations. In either qualitative or quantitative research, the nature of some studies limits the usefulness of the conclusions for other contexts. For example, research that consists of a program description may be useful for answering local questions or problems, but might not add significantly to the broader knowledge base of servicelearning research. To address this problem we recommend conducting cross-case or comparative analysis to increase understanding and generalizability by searching for themes and patterns across several cases (Patton, 2002). Limitations of generalizability can apply to many aspects of the research (e.g., sampling, nature of the intervention, Service-Learning Research Primer 17 context-specific elements, and measurement procedures). Generalizability is enhanced when the sample of respondents is heterogeneous (e.g., age, type or discipline of the service-learning course, type of institution). Limiting the generalization to reflect the restrictions of the sample and the study increases confidence in the research conclusions (Bringle, Phillips & Hudson, 2004).
• Not connecting to theory or knowledge base: Research on service-learning too seldom is cumulative across studies in meaningful ways. Rather, the field has been accumulating isolated evaluations of specific courses that have limited implications to other courses and broader practice. More research needs to have interventions and outcomes linked in systematic ways to theory. When this is done, there will be a basis of comparing and contrasting results and better understanding why outcomes were obtained or not obtained.
• Lack of common definition of terms: One difficulty in comparing results is that there may be no common agreement on definitions (e.g., service-learning, community service, volunteering, and reflection). For example, some researchers limit their studies to servicelearning experiences that occur in credit-bearing courses; others include co-curricular service in their definitions. This disparity leads to conclusions that may not be compatible. Lack of clarity and specification of conceptual and procedural aspects of the research can severely limit the value of information collected.
• Measures are mostly of the self-report type: Most service-learning research that involves student measures utilizes tools that are based on self-report (e.g., students self-report that they learned a great deal about diversity in a service-learning class). Although self-report instruments can be useful, they also have limitations (see Bringle et al., 2004; Steinke & Buresh, 2002), including that they may be influenced by social desirability response sets, they may not correspond to behavior, they may not accurately reflect processes that determined outcomes, and they may be affected by inaccurate or biased memories A few studies have utilized other types of tools such as behavioral ratings by an external observer and coding of student products (e.g., Osborne, Hammerich, & Hensley, 1998; Ash et al., 2005.)