This toolkit defines opportunistic experiments as randomized controlled trials of a policy or program an organization already plans to implement; showcases their advantages over more traditional research experiments; explains their potential value in evaluating human services programs; and details the steps needed to conduct this type of experiment.
This toolkit is a how-to guide for conducting opportunistic experiments (OEs), a type of Randomized Controlled Trial (RCT) that studies the effects of an initiative, program change, or policy action that an agency or program plans or intends to implement. The focus is on how to conduct OEs in the policy area of family self-sufficiency and stability. The resource is designed to help researchers, policymakers, and agency administrators recognize and conduct OEs. The resource explains that OEs can be more efficient for agency administrators to implement compared to more traditional research studies. In contrast to OEs, traditional, large-scale RCTs often examine an intervention or policy action that is developed and implemented specifically for a research study. OEs often use existing administrative data to measure outcomes for existing program participants. This can make them less costly and burdensome than other RCTs, which typically must collect new data. OEs initiated by state and local agencies can facilitate the recruitment of study sites, the cooperation of program staff, and the collection of low-cost administrative records data to measure outcomes. By embedding research in planned changes and using existing administrative data, agencies can seize an opportunity to generate rigorous evidence on the effectiveness of program changes. As a result, OEs can help to inform ongoing program improvements in a timely manner.
In the first part (Module 1) of the toolkit, the authors describe the use of OEs, explore their key features, and highlight several case studies. In the second part (Module 2), the authors discuss key steps and issues to consider when recognizing an opportunity for an experiment and determining the feasibility of conducting it. In the third part (Module 3), the authors outline and explain the critical steps in conducting an OE. This toolkit was produced as part of the Advancing Welfare and Family Self-Sufficiency Research Project (Project AWESOME), funded by the Administration of Children and Families’ (ACF) Office of Planning, Research and Evaluation (OPRE) within the U.S. Department of Health and Human Services.Full Publication Title: Advancing Evidence-Based Decision Making: A Toolkit on Recognizing and Conducting Opportunistic Experiments in the Family Self-Sufficiency and Stability Policy Area
Major Findings & Recommendations
Findings from each of the three modules include the following:
- The first part of this toolkit shows that OEs can be embedded within an agency’s ongoing operations in a manner that provides strong and timely evidence of effectiveness. Agencies can seize opportunities to conduct experiments in cases in which the intervention being studied will take place with or without experimentation, where clients or sites can be randomly assigned with minimal disruption to agencies and their staff, where strong partnerships exist between the research team and agency administrators and officials, and where there is fairly easy access to data on key outcomes.
- In Module 2, the authors describe three key steps and related issues to consider when determining both the potential for designing and the feasibility of conducting an OE. The first step is to look for opportunities when program or policy decisions may be made randomly. Many situations in which a decision or choice is made are potentially opportunities to use random assignment. Once an opportunity for random assignment is identified, the next step is to consider the research and program questions of the state or agency and whether the potential evaluation aligns with these questions. The final step is to assess the feasibility of the evaluation.
- In Module 3, the authors provide an overview of five key steps that researchers can follow to conduct an OE. All experiments, whether considered opportunistic or not, include these steps: (1) identify sites; (2) encourage participation in the study; (3) conduct random assignment and monitor compliance; (4) collect data; and (5) analyze data and report findings. The authors focus on aspects of these steps that may be somewhat different in OEs than in traditional RCTs.
Overall, the authors emphasize that at every level of the human services system, leaders need to know which programs, policies, and practices are most effective as they allocate scarce resources. Relatively low-cost and quick turnaround opportunistic experiments can be embedded within an agency’s ongoing operations in a manner designed to provide strong and timely evidence of effectiveness. Such studies allow agencies to learn about an intervention’s effectiveness and make evidence–based decisions to support families as they work toward self-sufficiency and stability.