Developing Conceptual Foundations for Randomized Controlled Trials
Developing Conceptual Foundations for Randomized Controlled Trials
Abstract and Keywords
Chapter 4 reviews the central role of conceptual frameworks in RCTs. The purpose of the conceptual framework in an RCT is to provide a system of ideas for understanding how an intervention is believed to lead to the outcomes. The conceptual framework defines the potential effectiveness of the intervention in terms of activities that are thought to produce change, in what context, and toward what outcome. Theory provides guidance in shaping hypotheses and formulating research questions. Theories may help define mediator and/or moderator effects among concepts, and can enrich the contribution of RCT research to social science. Overall, the rigor and strength of any empirical research is based on the quality of the conceptual framework and its applicability to the service setting.
Service providers are skeptical about the impact of research on community-based psychosocial interventions and social work practice. Some differentiate the “real world” from a mythical world of ivory-tower theories that are removed from practical action. However, without theory, most research would be merely a cacophony of numbers, Greek letters, and funny charts with X’s and O’s (see Chapter 5). Theory provides the foundation for the conceptualization of methods for randomized controlled trials (RCTs) in the “real world.” Its constructs enable the researcher to interpret findings and generalize these finding to other service settings.
A conceptual framework for an RCT is a system of ideas for understanding how an intervention is believed to lead to its outcomes. The conceptual framework defines the potential effectiveness of the intervention in terms of activities that are thought to produce change, in what context, and toward what outcome. Through the judicious application of theory, RCT conceptualization and research design operationalizes the (p.81) effectiveness of an intervention within its service context. A well-developed conceptual framework shapes the basic RCT design, provides an understanding of the pipeline of clients that feed into an intervention, grounds the sample and sampling strategy, and offers direction as to how the interventions are implemented, as well as determining what is measured and how the study is analyzed.
Conceptual work is iterative. Throughout the design process, researchers revisit and revise decisions made earlier based on conceptual, logistical, or political stumbling blocks that may be encountered later. A strong conceptualization will provide a plumb line, a guide for decision-making that will strengthen the overall research design.
The Role of Theory in Randomized Controlled Trials
RCTs are grounded in theory-driven deductive hypotheses. The challenges faced by clients and social workers’ are presumed to be real phenomena that can be measured, quantified, and studied over time to produce generalizable results. A quantitative approach presumes that a researcher is an objective observer, even while the work is motivated by the researcher’s values, grounded in social work ethics. These assumptions provide the structural underpinnings of the intervention.
Therefore, the most effective theories for RCTs are those that support explanatory models of process and outcome. These theoretical models should have the following characteristics:
• A grounding in empirical, quantitatively-driven social science
• Theoretical and empirical support that provides direction for operationalizing both process and outcome(s)
• A conceptual framework that delineates the role of the intervention in affecting change
• An empirical base that justifies change over time, as well as the expected timeframe for specific levels of change
• Proposed mediators that allow for operationalizing mechanisms of change in outcomes that are associated with the intervention and precede the outcome
• Proposed moderators that are associated with the service context and/or service population
A discussion of the full range of theories that can be applied to RCTs employing psychosocial interventions is beyond the scope of this text. The scientific literature available for this work is virtually limitless. For illustrative purposes, the following enumerates selected theoretical traditions commonly utilized in designing conceptual frameworks in RCTs for psychosocial interventions:
• Health Beliefs (Rosenstock, Strecher, & Becker, 1988)
• Transtheoretical Model of Change (DiClemente & Hughes, 1990; Pollio, Spitznagel, North, Thompson, & Foster, 2000; Sutton, 2001)
In some instances, it may prove problematic to identify an adaptable and relevant theory in justifying a proposed intervention leading to targeted outcomes. An integration of a number of smaller theories, along with prior empirical research to provide a rationale for linking the intervention to specific outcomes may be utilized. For example, in designing the RCT of Assertive Community Treatment (ACT) for clients leaving jail, researchers could not apply a specific theory that justified the study outcomes. Instead, a clinical case for integration and coordination of services and resources, along with a number of research studies employing this intervention, provided the framework for justifying the outcomes. Similarly, in RCTs of consumers and nonconsumer case managers, research on the working/professional alliance supplied the conceptual grounding. Furthermore, supportive relationships assist in developing positive adjustments to life stressors. In addition, other relevant literature demonstrates that through helping others, individuals help themselves, or what is called the “helper therapy principle” (Solomon, 2004). Consequently, expecting that clients of consumer case managers will have more positive psychosocial and service outcomes than those of their nonconsumer counterparts was justified by prior research studies, as well by a variety of small theories (Schmidt, Gill, Solomon, & Pratt, 2008). In circumstances in which there is not a single, theoretical tradition shaping an intervention, a clear conceptual framework that links the intervention to each of the specified outcomes is necessary to provide sufficient guidance for the intervention. The RCT may serve to build new theory for future interventions. In summary, a clear theoretical and empirical foundation must be established to justify the anticipated causal relationship between the intervention and each hypothesized outcome.
Mediators and Moderators
Theoretical models that include mediators and moderators serve two important functions for RCT research. First, these conceptualizations (p.84) allow for explanations of what was changed (mediators) and for whom (moderators) in delivering an intervention. The mediator allows for testing mechanisms for change in specified outcomes. Moderators, on the other hand, enable one to test factors that may interact with the intervention in such a way that the interaction of the moderator variable with the intervention has different effects (or strengths of effect) on outcomes. An RCT with a framework that includes mediator and/or moderator effects has the potential to increase the theoretical understanding of the process of change in outcomes.
Specifically, mediators are variables that are hypothesized to help make change happen (Baron & Kenny, 1986). They are a conceptual link in the middle of a cause-and-effect argument. Thus, in intervention studies, the mediator concept measured needs to be one that occurs after the intervention, but before the outcome. Some may think of the intervention itself as a mediator between baseline and outcome. Although this may be statistically accurate, considering the intervention as the mediator does not conceptually provide the most substantial contribution. In an RCT, the hypothesis is not that baseline status caused the outcome—this is likely already supported by the preponderance of evidence in almost any field—but rather that an intervention causes a change in outcome(s) greater than what is otherwise expected without the intervention. The mechanism of change is an interaction of both the intervention and the immediate change that is expected to induce the proposed outcomes. For example, in cognitive behavioral therapy, the homework and skill-building used in therapy (intervention) acts to reframe thinking about a stressful problem (mediator), which leads to reduced anxiety or depression (outcome). Therefore, to understand if the intervention works as intended, both the effect of the intervention on outcomes and the mediating role of the intervention’s impact on change in thinking must be examined.
The benefits of a theoretical grounding for a psychosocial intervention are exemplified in a recent article by Herbst and colleagues (2007). They assert that research in human immunodeficiency virus (HIV) prevention for gay men had moved to a point at which specification of mediators is to be expected in any proposed intervention study, in order to adequately enhance the effectiveness literature in this area. Herbst and colleagues (p.85)
Figure 4.2 proposes a mediation model for an RCT of Critical Time Intervention (CTI) for men with mental illness leaving prison in New Jersey. In this study, CTI is thought to be associated with a number of positive changes in men returning to the community—such as stable housing, fewer symptoms, and increased social functioning. Using an (p.86) individual-level social capital conceptualization, Van Der Gaag and Snijders (2003) propose that an intervention such as CTI functions as a means to increase resources associated with social connections (people who can help obtain jobs, lead to pro-social activities, solve problems, etc.), and that the outcomes of CTI are likely to be mediated through this effect. Therefore, a measure of
Furthermore, the model includes an exploratory set of questions that operationalize poverty as a moderating factor. This moderating factor is also based on a social capital formulation, as the social and economic resources that are accessible to individuals are likely shaped by the social and economic resources available in the neighborhood where an individual resides. Therefore, the moderating influence of the neighborhood is proposed to impact the outcomes of CTI. Conclusions based on the theoretical framework can thus be drawn at both the neighborhood level and the individual level. Such analyses will contribute to our knowledge of how CTI works in community re-entry in general. It will also enhance (p.87) the theoretical literature on how the construct of social capital can increase our understanding of re-entry for people leaving prison with mental illness in particular.
Research Question and Control Condition
The essence of the research question for an RCT is the response to the query: “Compared to what?” Integral to operationalizing the experimental intervention is operationalizing what it is not. Chapter 6 details the process of implementing these conditions. Chapter 5 will discuss how to incorporate them into the research design. The following discussion examines the research question and conceptualization as it relates to the control condition.
A beginning point is to understand what the usual course of care or service may be for the population under study. If the population is a group that is expected to be receiving some treatment or service for the condition or circumstances, then the most relevant control condition is that accepted standard of care. Thus, the comparison for the experimental intervention is made more rigorous by a comparison to what would usually be provided (i.e., treatment as usual; TAU). In some cases, the experimental intervention is sufficiently novel, or addresses a problem that is not within the usual realm of care, such that a no-treatment control can be justified. Examples might be an educational group intervention, creative arts therapies, or any service for a group that traditionally receives little formal treatment or service. Therefore, the relevant comparison is no special intervention as the usual is not receiving care.
However, in other cases, the researcher may be compelled to set up a control condition service that is considered benign in its impact. These are often supportive interventions that are not expected to have a deep or lasting impact on outcome measures. The reason for these control interventions/conditions is that the very fact of paying attention to those in the experimental condition may have an impact on them, independent of the nature of the specific intervention. Therefore, in some RCTs, a control intervention is offered to control for the attention or placebo effects. Placebo effects (p.88) refer to the effect of attention to the participant on the participant’s outcome. For example, supportive talk interventions without detailed protocols or general skill-building activities are known to have low to no impact on health and social welfare outcomes (Crepaz et al., 2006; Herbst et al., 2007; McFarlane, Dixon, Lukens, & Lucksted, 2003). Such models of intervention can therefore be considered relatively benign in terms of substantive outcomes for most populations of interest to social workers. Consequently, a control group intervention that may be a supportive discussion group, operating for an equivalent duration and intensity to the experimental intervention, can provide a design control for an experimental intervention that is expected to yield greater effects. In RCTs in which interventions are time-limited (e.g., with educational groups or time-limited therapy), the control condition could be a wait list control. In these instances, the control group is assigned to wait a period of time and is then offered the experimental intervention if they desire. However, with particular populations, some type of intervention may need to be offered to keep participants involved in the research.
In deciding on a control condition, ask: What question is being answered by the comparison? (see research question examples below). Is this the relevant question? Answering these questions thoroughly is worth extra consideration up front, as there are few chances for a “do-over” once data collection has begun. For example, in Solomon and Draine’s (1995) study of consumer case management, two case management teams provided services to clients of a public mental health system. One team was composed of case managers who identified as consumers, and the control team was a team typical of the case management teams (community treatment teams, or CTTs) operating in the system at the time and consisting mostly of bachelor’s-level mental health workers. The study hypotheses were that outcomes would be essentially the same between the teams, which would support the belief that consumer-run teams could do the core work of case management, as opposed to being limited to supportive, adjunctive roles. As hypothesized, results demonstrated similar outcomes between teams (Solomon & Draine, 1995). However, as the study was presented and reviewed, some pointed out that the teams could have been equally ineffective. Because there was no (p.89) control group without a case management team, this possibility could not be ruled out. The study has since been replicated with the same measures and data points by other investigators—and with a third condition (a no case management control group). This subsequent study essentially showed the two case management conditions as being equally effective when compared to the control condition, with some differences in how the teams operated and only a few positive outcomes for the consumer team clients compared to the nonconsumer team clients (Clarke et al., 2000; Herinckx, Kinney, Clarke, & Paulson, 1997; Paulson et al., 1999).
In a pilot study of a problem-solving educational intervention for older adults with depressive symptoms who were in home care, with a comparison of standard acute home health care for their medical problems with depression education materials and referral for antidepression medication, Gellis and colleagues (2007) noted that standard care alone was not an appropriate control condition. Although the comparison employed limits on the generalizability of study findings, it is a stronger comparison for testing the effectiveness of the experimental intervention. (p.90) Given that the Problem-Solving Therapy-Home Care includes many of the nonspecific therapeutic factors typical of psychotherapies, it was expected that there would be an improvement in depression.
A testable hypothesis poses a relationship between two concepts that are operationalized as the independent and dependent variables. In an RCT, the intervention (versus control) is the independent variable. The outcome is the dependent variable. Incorporating a brief description of the population, the hypothesis for an RCT may be: “Among children who are hospitalized, those who receive play therapy while hospitalized are more likely to be discharged earlier than those who receive only unstructured play time while hospitalized.” The independent variable is operationalized as an intervention of play therapy versus a control condition of unstructured play time. The outcome is a shorter hospitalization, under the assumption that this represents a positive outcome, all things being equal due to randomization. As noted in the previous discussion about research questions, the control condition shapes the question as much as the experimental intervention, and thus also shapes the hypothesis.
The hypothesis encapsulates the conceptualization of the RCT in an empirically testable statement of a relationship of the intervention to specific theorized outcomes. The direction of the relationship is explicit. Ultimately, the test of this hypothesis is only an approximation of the abstract concepts in the study, as each variable (length of hospitalization, play therapy versus unstructured play) is only a close approximation of the abstract phenomena of interest (play therapy, health status). However, given these limitations, empirically supported hypotheses can provide strong evidence for the effectiveness of an intervention as play therapy or any other theoretical relationship of an intervention to specific outcomes.
Because community-based psychosocial interventions incorporate the social environment as a key element of the intervention, the context (p.91) for any psychosocial intervention is complex. Researchers often seek to control for potential confounds (i.e., alternative explanations for variables) that may affect the results of their study. Age, gender, and racial identity are only the initial confounds frequently assessed. Depending on the field of interest, any number of risk factors, protective factors, or background characteristics may be critical confounds. These confounding variables need to be reflected in the hypotheses, with qualifying clauses, such as “controlling for length and severity of illness, insurance status, and age, children who are hospitalized” (adding to the hypothesis stated previously). One must have a full conceptualization of the intervention and outcome(s) reflected in the hypotheses, with all abstract concepts in the hypotheses clearly operationalized with appropriate measures and complete tests of the hypotheses delineated in advance. In this way, a well-crafted hypothesis synthesizes all the various elements of the proposed RCT. Thus, whereas the best hypotheses are often elegantly simple, the stated hypothesis includes both complexity and logic.
(p.92) If mediators and/or moderators have been theoretically established, hypotheses need to reflect these relationships. A specific logic exists to testing these relationships in terms of hypotheses that build on one another, and which then support mediating or moderating relationships among variables. For example, in the randomized trial of CTI, the testing of the mediating relationship illustrated in Figure 4.2 consists of testing three hypotheses:
1. First, it is hypothesized that CTI is associated with a greater growth in individual-level social capital than the control condition. This relationship is represented by the arrow from CTI to the “Mediating Effect” of an increase in resources from community connections.
2. Second, it is hypothesized that CTI is associated with stronger positive outcomes than the control condition. This relationship is represented by the arrow from CTI directly to outcomes.
3. If both these hypotheses are supported, then it is further hypothesized that when the mediator from hypothesis one is included as a control variable in the model that measures the effect from CTI to the outcome, the direct effect of CTI on the outcomes will be reduced or eliminated, with the effect of increased social capital (the arrow from the mediating effect to outcome) significantly explaining the outcomes.
Thus, this interactive system of all three hypotheses is tested to determine support for a mediating relationship (Baron & Kenny, 1986). The hypotheses and the statistical analysis plan for the CTI RCT study reflect this logic, and every outcome in the model corresponds to data collected in the study and a potential test of a mediated effect of CTI on each of the outcomes.
Strategies are available for testing a moderating relationship as well. In the CTI model, an exploratory question concerns the extent to which community-level characteristics may moderate the effect of CTI. For illustrative purposes, setting aside issues relating to the geographically situated nature of these concepts, let us assume that one can measure (p.93) for each CTI study participant a “poverty” variable based on their living situation upon release. Following from the Baron and Kenny method (Baron & Kenny, 1986), this relationship may be more directly portrayed as a moderating effect, as shown in Figure 4.3. To determine a moderating relationship involves testing the direct effects of both CTI and poverty on outcome. If there is an effect for CTI on outcome, and poverty on outcome, the effect of CTI on outcome may be linked to the poverty level of the individual participant, and thus warrants a test of the interaction as well.
Moderating relationships can be complicated by the nature of the effect, as the relationship between the moderating variable and the intervention may be curvilinear, rather than linear. For example, there may be a level of poverty at which CTI is most effective; where those impacted by a greater or lesser degree of poverty have less benefit from CTI services. Further reading of any reliable text on testing such relationships is needed for more detail. For our purposes, the take-home message is that to pose these relationships requires clear, conceptual reasoning grounded in a theoretical framework. Otherwise, the analysis phase can dissolve into the proverbial “fishing expedition,” in which a researcher is searching for statistically significant relationships, may find one by chance (and/or error), and thus feel compelled to dream up an ad hoc theory to explain the significant findings. What is far more likely to contribute to the psychosocial and social work intervention evidence base is a clear
Many studies fail to explore fully the study hypotheses so as to test the change process relationships. Hypotheses based on the proposed relationships between intervention and outcome provide the guide to what concepts will be measured, and what outcomes and processes (change concepts, moderators, and mediators) are expected to be included in statistical models. Building a hypothesis may not be as simple as it seems. Like other aspects of RCT design, the process is iterative. As a preliminary set of hypotheses are conceptualized, the investigator assesses the extent to which they are testable, using measurable concepts and statistically testable interrelationships. In this iterative process, measurement strategies can be reformulated, conceptualizations refined, and even a rethinking of basic elements, such as the experimental and control conditions, may occur. Only after the conceptual work is completed, where hypotheses summarize precisely the question to be explored, does the research being designed have sufficient guidance for developing the detailed mechanics of an RCT.
In reviewing the role of theory, this chapter does not delve into the myriad of theoretical frameworks. Rather, it reviews how theory drives a defensible conceptualization of an RCT design. Therefore, a conceptual framework becomes a structure on which to hang focused research questions and hypotheses. The research question is shaped in interaction with the service context and also in reviewing the relevant research literature. After deriving a clearly defined and measurable research question, a testable hypothesis is drawn. This hypothesis (or hypotheses) should incorporate all elements of the research design (as reviewed in the Chapter 5).
Mediating and moderating frameworks are tools in conceptualizing community-based psychosocial interventions, including those delivered by social workers. Using these tools, researchers can operationalize and (p.95) test mechanisms for change and differential effects for the intervention based on theory, client characteristics, or varied organizational and community contexts. These enhanced frameworks enable further development of theory grounded-in community-based service settings, as well as test the effectiveness of the psychosocial intervention.
For Further Reading
Frazier, P., Tix, A., & Barron, K. (2004). Testing moderator and mediator effects in counseling psychology. Journal of Counseling Psychology, 51, 115–133.
Reynolds, P. (2007). A primer in theory construction. Boston: Pearson Allyn and Bacon.
Becker, H. (1998). Tricks of the trade: How to think about research while you are doing it. Chicago: University of Chicago Press.