Advertisement

Moving From Research to Large-Scale Change in Child Health Care

  • David A. Bergman
    Correspondence
    Address correspondence to David A. Bergman, MD, 770 Welch Rd, Suite 100, Palo Alto, California 94304.
    Affiliations
    Department of Pediatrics, Stanford University School of Medicine, Palo Alto, Calif (Dr Bergman), and Institute for Health Research, Kaiser Permanente Colorado, Denver, Colo (Dr Beck)
    Search for articles by this author
  • Arne Beck
    Affiliations
    Department of Pediatrics, Stanford University School of Medicine, Palo Alto, Calif (Dr Bergman), and Institute for Health Research, Kaiser Permanente Colorado, Denver, Colo (Dr Beck)
    Search for articles by this author

      Abstract

      There is a large and persistent failure to achieve widespread dissemination of evidence-based practices in child health care. Too often studies demonstrating evidence for effective child health care practices are not brought to scale and across different settings and populations. This failure is not due to a lack of knowledge, but rather a failure to bring to bear proven methods in dissemination, diffusion, and implementation (DD&I) science that target the translation of evidence-based medicine to everyday practice. DD&I science offers a framework and a set of tools to identify innovations that are likely to be implemented, and provides methods to better understand the capabilities and preferences of individuals and organizations and the social networks within these organizations that help facilitate widespread adoption. Successful DD&I is dependent on making the intervention context sensitive without losing fidelity to the core components of the intervention. The achievement of these goals calls for new research methods such as pragmatic research trials that combine hypothesis testing with quality improvement, participatory research that engages the target community at the beginning of research design, and other quasi-experimental designs. With the advent of health care reform, it will be extremely important to ensure that the ensuing large demonstration projects that are designed to increase integrated care and better control costs can be rapidly brought to scale across different practices settings, and health plans and will be able to achieve effectiveness in diverse populations.

      Keywords

      The past 50 years have witnessed remarkable strides in the health care of children, yet compelling problems persist. Minority children experience large disparities in health care outcomes,
      National Health Care Disparities Report.
      and there is excessive variability in the diagnostic and therapeutic services children receive.
      • Mangione-Smith R.
      • DeCristofaro A.H.
      • Setodji C.M.
      • et al.
      The quality of ambulatory care delivered to children in the United States.
      Many of these problems stem from a profound gap between what we know in medicine and what we do in practice. A review of any major pediatric journal or scientific meeting shows a plethora of innovative ideas and practices. Further examination however shows that many of the ideas and practices have not been widely implemented and their full impact realized. A recent Institute of Medicine study has shown that the slow translation of scientific discoveries to improved practices accounts for many of the deficiencies in health care.
      • Sung N.S.
      • Crowley Jr., W.F.
      • Genel M.
      • et al.
      Central challenges facing the national clinical research enterprise.
      • Contopoulos-Ioannidis D.G.
      • Ntzani E.
      • Ioannidis J.P.
      Translation of highly promising basic science research into clinical applications.
      • Lenfant C.
      Clinical research to clinical practice—lost in translation?[Shattuck lecture].
      Children have also suffered because of this problem. Research has documented the slow and uneven spread of evidence-based practices across numerous conditions ranging from asthma
      • Finkelstein J.A.
      • Lozano P.
      • Farber H.J.
      • et al.
      Underuse of controller medications among Medicaid-insured children with asthma.
      to developmental screening.
      • Bethell C.
      • Reuland C.H.
      • Halfon N.
      • Schor E.L.
      Measuring the quality of preventive and developmental services for young children: national estimates and patterns of clinicians’ performance.
      Past quality improvement initiatives have attempted to bridge this gap but with only modest success.
      • Bero L.A.
      • Grilli R.
      • Grimshaw J.M.
      • et al.
      Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group.
      • Ranji S.R.
      • Shetty K.
      • Posley K.A.
      • et al.
      • Bravata D.M.
      • Sundaram V.
      • Lewis R.
      • et al.
      • Grimshaw J.
      • Eccles M.
      • Tetroe J.
      Implementing clinical guidelines: current evidence and future implications.
      The traditional sequence for discovery and translation of new scientific knowledge has been described by Woolf and others.
      • Woolf S.H.
      The meaning of translational research and why it matters.
      • Zerhouni E.A.
      Translational and clinical science—time for a new vision.
      Szilagyi has extended this description to community-based practice and population health (Figure 1).
      • Szilagyi P.G.
      Translational research and pediatrics.
      The primary emphasis in this discovery process has been on the first phase of translational research (T1), the development of basic knowledge, and on the second phase (T2), the translation of this knowledge into evidence-based guidelines. Unfortunately, few resources have been directed at the widespread dissemination and implementation of evidence-based guidelines to community-based practice and across diverse populations (T3 and T4 translational research). This has resulted in too few instances where T1 and T2 research results have been brought to scale and achieved widespread population impact.
      • Glasgow R.E.
      • Lichtenstein E.
      • Marcus A.C.
      Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition.
      The slow translation of research into practice has given rise to programs designed to accelerate this process.
      • Zerhouni E.
      Clinical research at the crossroads: The NIH roadmap.

      CTSA Clinical and Translational Science Awards. 2003. Available at: http://www.ctsaweb.org. Accessed October 5, 2010.

      Again, most of this effort has involved T1 and T2 translational research with current research expenditures of $8.3 billion (91%) on T1 and only $787 million (9%) on T2 research.
      • Moses 3rd, H.
      • Dorsey E.R.
      • Matheson D.H.
      • Thier S.O.
      Financial anatomy of biomedical research.
      Failure to achieve widespread use of evidence-based practices can lead to unnecessary mortality and morbidity, increase disparities in the delivery of child health care, waste resources, and adversely impact the sustainability of important intervention in our communities.
      • Halterman J.S.
      • Aligne C.A.
      • Auinger P.
      • et al.
      Inadequate therapy for asthma among children in the United States.
      • Miller M.R.
      • Gergen P.
      • Honour M.
      • et al.
      Burden of illness for children and where we stand in measuring the quality of this health care.
      • Broder K.R.
      • Cohn A.C.
      • Schwartz B.
      • et al.
      Adolescent immunizations and other clinical preventive services: a needle and a hook?.
      The failure to spread effective interventions is not due to a lack of knowledge. There is considerable research in the dissemination, diffusion, and implementation (DD&I) science that can be brought to bear on this problem. In this article, we will first discuss the theoretical underpinnings of DD&I and then review the process of DD&I, beginning with exploration and adoption and ending with implementation. We will then discuss methods for DD&I and evaluation of DD&I ending with implication for researchers and clinicians.

      Definitions

      In this article, dissemination refers to how evidence-based practices are best communicated to potential adopters and implementers. Diffusion follows dissemination and is a process where potential adopters seek advice from their colleagues about adopting a new intervention.
      • Dearing J.
      • Kruetner M.
      Designing for diffusion of cancer communication innovations.
      Adoption, or agreement to try a new practice, requires both dissemination and diffusion of the new intervention. Implementation is then defined as what happens after adoption; it is the “how” of putting evidence-based medicine into practice. Last, sustainability refers to the continued use of evidence-based practices and associated activities to achieve desired outcomes. Unfortunately, there is still no clear-cut consensus on these definitions, and some of the references we cite may use different definitions.

      Theoretical Underpinnings of DD&I

      Modern-day DD&I theories have their origin in the iconic work of Ryan and Gross, who documented the spread of hybrid seed corn in rural Iowa in the 1930s.
      • Ryan B.
      • Gross N.C.
      The diffusion of hybrid seed corn in two Iowa communities.
      Their work demonstrated the well-known S-shaped diffusion curve where initial uptake of a new innovation is slow and then after a period of time increases rapidly, only to taper off toward the end of the adoption process. Building on this work, Everett Rogers developed his theory on the diffusion of innovation.
      • Rogers E.
      Diffusion of Innovation.
      Rogers’s work demonstrated that the rate of change depended on the spread of knowledge and experience with the innovation through different groups of adopters. He identified 4 categories of adopters on the basis of their personal characteristics. These included early adopters who are innovation seeking, constantly scanning the environment for new innovations. This group does not feel constrained by their colleagues and profession but as a consequence may be viewed with some skepticism by other members of the profession. The next group is the early majority. This group looks to the early adopters for ideas but waits to see evidence of beneficial outcomes before adopting the intervention. Next is the late majority, which views new ideas with some skepticism, waiting until the idea has been tried and vetted by the early majority. Rogers called the final group the laggards. This group is characterized by a strong resistance to adoption of new ideas. He posited that diffusion of innovation starts with early adopters and spreads through the late majority with many in the laggard group failing to adopt the intervention. Rogers established that the diffusion of an intervention within a group is a longitudinal social process of advice seeking and advice giving about the intervention where adoption moves through different groups of adopters. This work has been recently popularized by Malcom Gladwell in his book The Tipping Point.
      • Gladwell M.
      The Tipping Point: How Little Things Can Make a Big Difference.
      Rogers’s theory of the diffusion has important implications for the spread of new knowledge. This is particularly true for the field of evidence-based medicine. The term evidence-based medicine was first publicized in 1992 with the publication the seminal article “Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine” by the Evidence-Based Medicine Working Group.
      Evidence-Based Medicine Working Group
      Evidence-based medicine: a new approach to teaching the practice of medicine.
      This approach proposed that empirical research be explicitly incorporated into clinical decision making. The publication of the evidence-based medicine manifesto has led to the emergence of multiple evidence-based centers and journals.
      • Cochrane A.
      Effectiveness and Efficiency: Random Reflections of Health Services.

      Evidence-based practice centers: synthesizing scientific evidence to improve quality and effectiveness in health care. Evidence Based Practice Centers. Available at: http://www.ahrq.gov.laneproxy.stanford.edu/clinic/epc/epcenters.htm. Accessed November 29, 2010.

      Although strong links exist between evidence-based medicine and dissemination and implementation science,
      • Lomas J.
      The in-between world of knowledge brokering.
      • Lomas J.
      Words without action? The production, dissemination, and impact of consensus recommendations.
      • Eddy D.M.
      Evidence-based medicine: a unified approach.
      the evidence-based medicine approach assumes that evidence-based practices will naturally spread among practitioners made aware of the scientific evidence. This runs counter to DD&I science, which proposes that adoption is an active process that can be facilitated through attention to the stages of the process, the nature of the innovation, the individual, professional values and beliefs, and social networks.

      Natural History of DD&I

      DD&I can be best understood through an appreciation of the process, starting with exploration and adoption of new knowledge and ending with the sustainability of the practice in a changing environment. We will briefly review the natural history of the DD&I as outlined by Fixsen and colleagues.
      • Fixsen D.N.
      • Naoom S.F.
      • Blase K.A.
      • et al.
      Implementation Research: A Synthesis of the Literature.

      Exploration and Adoption

      DD&I begins with exploration and adoption of a new practice. Here proponents of the new practice actively engage in discussion and educational activities with the potential adopters to assess whether or not the new practice will be clinically relevant to clinicians and will improve the well-being of the families under their care. This concept of engagement of the target population has been operationalized in community-based participatory research. Here key stakeholders who will be recipients of the intervention are involved in the research design and the development of the DD&I plan.
      • Israel B.A.
      • Schulz A.J.
      • Parker E.A.
      • et al.
      Review of community-based research: assessing partnership approaches to improve public health.
      • Minkler M.
      Community-based research partnerships: challenges and opportunities.
      • Van Olphen J.
      • Schulz A.
      • Israel B.
      • et al.
      Religious involvement, social support, and health among African-American women on the east side of Detroit.
      One example of community-based participatory research is the Healthy Public Housing Initiative. This project involved Boston public agencies for housing and health, 3 universities, and several community housing groups. A steering committee composed of community-based representatives and organizations, health services providers, the city health department and academic researchers met regularly to conceive, design, implement, and analyze a program to reduce environmental hazards for children with asthma living in public housing. The results of the research demonstrated a high degree of acceptance of the program and a significant increase in asthma-related quality of life for the target children.
      • Levy J.I.
      • Brugge D.
      • Peters J.L.
      • et al.
      A community based participatory research study of multifaceted in-home environmental interventions for pediatric asthmatics in public housing.
      Pediatric practitioners have been leaders in the engagement of families in the health care of children. There is a need, however, for child health researchers to more directly involve both providers and families in the design and implementation of T3 and T4 research.
      Rogers and others have described several important criteria for successful adoption of an intervention.
      • Rogers E.
      Diffusion of Innovation.
      Perhaps the most important criteria are the relative advantage and potential excess burden of the intervention to the pediatrician and family. If both groups do not perceive a strong advantage for themselves, there is little likelihood for successful adoption. Successful adoption also depends on how compatible the new practice is with current practice, and professional culture and mission. Compatibility can be enhanced through multiple iterative trials that refine the intervention to meet the needs of the practice, with results that are readily observable to other pediatricians and families.
      The relative complexity of the intervention also relates directly to the adoption of a new practice. Edwards has proposed a simple typology to describe levels of complexity

      Edwards N. Scaling-up health innovations and interventions in public health: a brief review of the current state-of-the-science. Commissioned for the inaugural Conference to Advance the State of the Science and Practice on Scale-up and Spread of Effective Health Programs, Washington, DC, July 6–8, 2010.

      : the least complex interventions are discrete events that involve a single act or process, such as the Back to Sleep program to prevent sudden infant death syndrome. More complex interventions are multicomponent and involve multiple linked interventions—for example, an asthma care path with appropriate diagnostic tests, assessment, treatment, and patient education. The most complex are interventions that involve a wholesale change in the way we conceive of a problem and its solution—for example, a campaign against obesity where change occurs in not only health care practice, but also in the way we grow and sell food and how we change the built environment. Research has identified other criteria, including the strength of the evidence base, the flexibility or adaptability of the intervention to meet the needs of different groups, the reversibility of the intervention, and the cost of the intervention.
      • Greenhalgh T.
      • Robert G.
      • Macfarlane F.
      • et al.
      Diffusion of innovations in service organizations: systematic review and recommendations.

      Before Implementation

      After the decision to adopt a new practice, there is a need to anticipate and plan for the potential impact of the new practice on the individual, the clinical practice, and the organization.
      • Ferlie E.B.
      • Shortell S.M.
      Improving the quality of health care in the United Kingdom and the United States: a framework for change.
      At the level of the individual, there is a need to create a pull for the new intervention among potential adopters.
      • Della Penna R.
      • Martel H.
      • Neuwirth E.B.
      • et al.
      Rapid spread of complex change: a case study in inpatient palliative care.
      Too often, efforts to disseminate new knowledge have involved push strategies where the new knowledge is presented to individuals with the expectation of uptake and use.
      • Rabin B.A.
      • Brownson R.C.
      • Kerner J.F.
      • et al.
      Methodologic challenges in disseminating evidence-based interventions to promote physical activity.
      • Davidson C.
      Knowledge translation: implications for evaluation.
      Such push strategies are much more effective when combined with a pull strategy driven by greater understanding of factors important to the individual.
      • Green L.W.
      • Orleans C.T.
      • Ottoson J.M.
      • et al.
      Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control.
      For example, the early publication of evidence-based practice guidelines by the American Academy of Pediatrics (AAP) led to considerable pushback by pediatricians, who criticized these efforts as cookbook medicine and another unfunded mandate from the AAP. A greater understanding of the important factors, such as the impact of the guideline on cost and practice efficiency, were important to ensuring more widespread adoption.

      Bergman DA. Personal report. American Academy of Pediatrics Committee on Quality Improvement.

      More extensive qualitative research can help to delineate the pull factors that may have helped to facilitate more rapid adoption of these guidelines.
      • Dearing J.W.
      • Rogers E.M.
      • Meyer G.
      • et al.
      Social marketing and diffusion-based strategies for communicating with unique populations: HIV prevention in San Francisco.
      Successful DD&I also requires planning at the group or practice level. Within any large group or practice, 3% to 5% of the members are key individuals to whom others turn for advice about adoption of a new practice or innovation.
      • Green L.W.
      • Gottlieb N.H.
      • Parecel G.S.
      Diffusion theory extended and applied.
      These individuals may be opinion leaders who influence group thinking through their role as advice givers, or they may be change agents who model the intervention for other group members. Social network analysis is a method by which a group or practice can be analyzed to identify these key individuals.
      • Freeman L.
      The Development of Social Network Analysis.
      • Luke D.A.
      • Harris J.K.
      Network analysis in public health: history, methods, and applications.
      • Hawe P.
      • Webster C.
      • Shiell A.
      A glossary of terms for navigating the field of social network analysis.
      By means of this methodology, it is possible to assess the connectivity of individuals in the network to other members in the network, and the social power of an individual on the basis of their connectedness within the network. Figure 2 shows a social network map for a pediatric clinical practice. The network was constructed from a survey of the clinic providers and administrative staff designed to assess who an individual sought out for advice about practice change and preventive care.
      • Beck A.
      • Bergman D.A.
      • Rahm A.K.
      • et al.
      Using implementation and dissemination concepts to spread 21st-century well-child care at a health maintenance organization.
      Four opinion leaders were identified from this group: 021 (a developmental specialist), 019 and 008 (both pediatricians), and 025 (a medical assistant). Importantly, the identified opinion leaders in this network were not necessarily the formal or positional leaders, yet their engagement in the process was essential for effective DD&I of the intervention.
      Figure thumbnail gr2
      Figure 2Social network analysis of pediatric clinic as part of a project to redesign.
      Perhaps the most important area of preimplementation planning for successful DD&I is at the level of the organization. Successful organizations will have a track record for innovation
      • Bevan H.R.G.
      • Bate P.
      • Maher L.
      • et al.
      Using a design approach to assist large-scale organizational change: "10 high impact changes" to improve the National Health Service in England.
      • Bradley E.H.
      • Webster T.R.
      • Baker D.
      • et al.
      Translating research into practice: speeding the adoption of innovative health care programs.
      and will have the capability to translate research into practice.
      • Bradley E.H.
      • Webster T.R.
      • Baker D.
      • et al.
      Translating research into practice: speeding the adoption of innovative health care programs.
      • Gardner K.L.
      • Dowden M.
      • Togni S.
      • Bailie R.
      Understanding uptake of continuous quality improvement in Indigenous primary health care: lessons from a multi-site case study of the Audit and Best Practice for Chronic Disease project.
      Recently, tools for assessing organizational readiness to adopt, implement, and sustain interventions have been developed and tested for health information technology, and these tools may have application to other realms.
      • Wen K.Y.
      • Gustafson D.H.
      • Hawkins R.P.
      • et al.
      Developing and validating a model to predict the success of an IHCS implementation: the Readiness for Implementation Model.
      In large-scale DD&I efforts involving multiple organizations, many of the participating organizations may lack the necessary change capacity.
      • Bate P.
      • Robert G.
      • Bevan H.
      The next phase of healthcare improvement: what can we learn from social movements?.
      These large-scale efforts often require nodal or anchor organizations to provide central support for such key functions as recruitment, coaching, program education, and technical support.
      • Pastor M.
      • Ortiz R.
      Making Change: How Social Movements Work and How to Support Them.
      This model has been successfully deployed in the Institute for Healthcare Improvement 5 Million Lives campaign, which created supporting nodal organizations on a regional level,
      • McCannon J.
      Update: 5 Million Lives campaign. Interview by Dan Pastorius.
      • McCannon C.J.
      • Hackbarth A.D.
      • Griffin F.A.
      Miles to go: an introduction to the 5 Million Lives campaign.
      • McCannon C.J.
      • Perla R.J.
      Learning networks for sustainable, large-scale improvement.
      and in the AAP Chapter Alliance for Quality Improvement program, where state chapters assume an anchor function in disseminating evidence-based asthma guidelines.

      Chapter Alliance for Quality Improvement. 2010. Available at: http://www.aap.org/member/chapters/caqi/about.html. Accessed November 23, 2010.

      The development of a DD&I plan is the final outcome of the preimplementation phase. A good plan begins with a goal easily understandable to everyone that provides a reference point for all future actions. The Institute for Healthcare Improvement’s 5 Million Lives initiative is an example of an overall aim that is both quantifiable and dramatic in its impact.
      • McCannon J.
      Update: 5 Million Lives campaign. Interview by Dan Pastorius.
      In Jonkoping County, Sweden, the phrase “a good life in an attractive county” is used to create a vision of a patient-centered (vs disease-centered) system.
      • Ovretveit J.
      • Staines A.
      Sustained improvement? Findings from an independent case study of the Jonkoping quality program.
      Research has shown that there are core components to a DD&I plan that are essential for success.
      • Fixsen D.N.
      • Naoom S.F.
      • Blase K.A.
      • et al.
      Implementation Research: A Synthesis of the Literature.
      The plan should include selection criteria for providers and families that assess qualifications and experience that may not be teachable in training sessions. For example, implementing an evidence-based practice such as developmental screening requires a workforce that understands the value of screening and has the communication skills to successfully interpret the screening results for families. Another important component of a DD&I plan is the designation of a consultant/coach, to help facilitate implementation of the new practice. For example, the McColl Institute and Qualis Health Care, in their national initiative to “Transform Safety-Net Clinics into Medical Homes,” use practice coaches to help safety-net clinics implement a new medical home model. Here coaches are used to assess and transform care processes and to provide technical support and expert knowledge.

      Commonwealth Fund. Tranforming safety net clinics to medical homes. October 14, 2010. Available at: http://www.commonwealthfund.org/Multimedia-Center.aspx. Accessed November 30, 2010.

      Last, the DD&I plan needs to assure adequate funding and staff support. Too often DD&I efforts fail because of a lack of consideration of needed funds for activities such as provider and organizational time devoted to the project, the development of needed social networks, and start-up costs in new intervention sites.

      Initial Implementation and Pilot Studies

      Although some interventions are implemented at scale at one point in time (the so-called big bang approach to implementation), most implementation begins with a series of iterative cycles where the intervention is tried in small groups with the results studied, the lessons learned, and the intervention improved before the next intervention cycle.
      • Langley G.
      The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.
      Who participates in these initial cycles and how the cycles are conducted are critical for the success of the DD&I effort. Inclusion only of participants who are early adopters or who are enthusiastic about the new practice will decrease the likelihood that the results of the pilot study will translate to the larger target population.
      • Fraser S.
      Undressing the Elephant: Why Good Practice Doesn’t Spread in Health Care.
      Naysayers serve the important function of continually questioning the value of the intervention and keeping open alternative solutions. Pilot studies also need to be conducted with a minimum of special resources that may not be available during more widespread implementation and planning for sustainability, decreasing the likelihood of a successful transition to full-scale implementation.

      Full-Scale Implementation

      Many DD&I projects fail during the transition from pilot studies to full-scale implementation. This failure stems from the common assumption that once success has been demonstrated in the pilot study, a tipping point will be reached, and spread throughout the organization will magically occur. Although there may be a tipping point for discrete interventions such as measurement of body mass index in children, this is much less likely for the multicomponent interventions that are common in health care. In fact, research has demonstrated that a solid evidence base and early success in the pilot phase are not strong predictors of success in bringing a project to full-scale implementation.
      • Van de Ven A.H.
      • Polley D.E.
      • Garud R.
      • et al.
      The Innovation Journey.
      In full-scale implementation, the target population increases in size and becomes more heterogeneous in character. Unlike the single group addressed during the pilot study, the target population becomes composed of different groups with distinct needs. Rogers’s schema of different adopter groups is one way to segment a population according to perceived need and willingness to change. Segmentation can also occur according to practice type, available resources for change, and the nature of the target patient population. The goal is to recognize these different groups and to tailor the intervention to meet the specific needs of each group. For example, implementing an early intervention program for all children within a state will require tailoring the screening process to realities of the different types of practices, providers, and patient populations. The Assuring Better Childhood Health and Development program consulted with office staff and community partners in their region to better define the needs of specific target groups. The developmental screening process was then tailored to the existing screening process in offices and the needs of the community partners.
      • King T.M.
      • Tandon S.D.
      • Macias M.M.
      • et al.
      Implementing developmental screening and referrals: lessons learned from a national project.
      • Klein S.
      • McCarthy D.
      North Carolina’s ABCD program: using community care networks to improve the delivery of childhood developmental screening and referral to early intervention services.
      Full-scale implementation is a dynamic process demanding a constant reevaluation of an intervention in light of changing systems, patient populations, and external environments. In the words of one expert, “Continuous adoption requires continuous improvement.”
      • Fraser S.
      Undressing the Elephant: Why Good Practice Doesn’t Spread in Health Care.
      Research has demonstrated that interventions that adapt in response to different segments in the target group are more successful in their implementation efforts.
      • Greenhalgh T.
      • Robert G.
      • Macfarlane F.
      • et al.
      Diffusion of innovations in service organizations: systematic review and recommendations.
      This process of continuous adoption to meet the needs of different groups can, however, create a tension between the desire to replicate the positive effects of the intervention across dissimilar settings and conditions, and the need to maintain fidelity to the evidence-based practices. Reconciliation of the needs to be both flexible while maintaining fidelity to the evidence is best accomplished through a clear articulation of essential and nonessential components of the intervention. Greenhalgh and colleagues refer to the nonessential components of an intervention as the “soft periphery,” or those aspects of the intervention that can be modified at the edges without adversely impacting its fidelity to core components of the intervention.
      • Greenhalgh T.
      • Robert G.
      • Macfarlane F.
      • et al.
      Diffusion of innovations in service organizations: systematic review and recommendations.
      For example, in an asthma guideline, a core component may be the provision of an asthma action plan by the pediatrician, while needed asthma education could take place in the pediatric office or the school, or could even be Web-based.

      Sustainability

      In many ways, sustainability resembles full-scale implementation in its demand for continuous improvement. Inevitably, there are turnovers in key stakeholders, changes in funding streams, new health policies and legislation, and new political alliances and partnerships, all of which must be accommodated to maintain fidelity to the core components of the program and ensure the desired outcomes. This requires constant surveillance of the environment and regular re-assessment of the both process and outcome measures.

      Method or Process of DD&I

      Most successful DD&I efforts occur within the context of an improvement model. Several models have been widely used for DD&I work. These include the Improvement Model, Lean Systems or Six Sigma, the Chronic Care Model, and the Breakthrough Series Collaborative.
      • Langley G.
      The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.
      • de Koning H.
      • Verver J.P.
      • van den Heuvel J.
      • et al.
      Lean Six Sigma in healthcare.
      • Bodenheimer T.
      • Wagner E.H.
      • Grumbach K.
      Improving primary care for patients with chronic illness.
      • Bodenheimer T.
      • Wagner E.H.
      • Grumbach K.
      Improving primary care for patients with chronic illness: the chronic care model, part 2.
      • Toncich G.
      • Cameron P.
      • Virtue E.
      • et al.
      Institute for Health Care Improvement Collaborative Trial to improve process times in an Australian emergency department.
      • Besserman E.
      • Brennan M.
      • Brown 3rd, P.A.
      • et al.
      Multidisciplinary achievement: the collaborative approach to rapid cycle ICU and hospital change.
      There is no single correct model for DD&I. For example, the implementation of a discrete intervention such as a bundle of changes designed to decrease catheter-associated bloodstream infections in a newborn intensive care unit may be best accomplished by using the improvement model, while a multicomponent intervention such as an obesity prevention program may be best accomplished through collaborative teams in a breakthrough series or lean systems analysis. The key is to select the most applicable model or specific components of a model to the problem at hand, rather than let the model drive the activities. The most appropriate choice of model may depend on the whether the intervention is occurring at the level of the individual, practice, or organization. The improvement model may work best at the practice level, while the campaign model may be most effective in the context of a group or network of organizations. Last, it may be necessary during the DD&I process to use a mixed model of different methods for spread.
      It is sometimes difficult to distinguish between DD&I and quality improvement in achieving the widespread adoption and use of new practices. The best distinction may be one of scale. Quality improvement efforts have more traditionally focused on a group of individuals or an organization as the target of change, whereas DD&I efforts have at their core the spread of new practice and innovation across different organizations and diverse target populations. Ultimately DD&I science will provide an important set of concepts and tools to facilitate meaningful quality improvement work, and improvement science will continue to inform model development and process change in DD&I.
      DD&I science has also fostered the development of new research methodologies that attempt to study clinical interventions in real-world settings. Pragmatic or practical research trials
      • Tunis S.R.
      • Stryer D.B.
      • Clancy C.M.
      Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
      • Godwin M.
      • Ruhland L.
      • Casson I.
      • et al.
      Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity.
      • Helms P.J.
      “Real world” pragmatic clinical trials: what are they and what do they tell us?.
      • Glasgow R.E.
      • Magid D.J.
      • Beck A.
      • et al.
      Practical clinical trials for translating research to practice: design and measurement recommendations.
      are designed to answer the question of whether a given intervention works in usual settings to determine how the intervention can be best adapted to increase adoption without losing the core components that are responsible for the desired outcomes. There is an explicit recognition that there will not be strict compliance with an intervention and that it is important to study this variation and its impact on outcomes. Pragmatic research trials often incorporate quasi-experimental rather than randomized designs to evaluate practice improvements or natural experiments. For example, multiple baselines across settings design may stagger or delay the intervention for a segment of the population (Figure 3). This design allows the researcher to learn from the early phases of the process and incorporate this information into a new version of the intervention, which is then tested with the next group in the study population. The Depression Improvement Across Minnesota Offering a New Direction (DIAMOND) is an example of a pragmatic research trial in a large number of primary care practices in Minnesota. Evidence-based practices and reimbursement change for depression care were implemented in 5 staggered sequences over 3 years. This approach allowed for more time to train staff in multiple sites while allowing time to revise the approach on the basis of experience from the prior implementation sequence. Each site served as its own pre- and post-control group. The staggered implementation and repeated measures allowed for the replication of the intervention across multiple settings and over the time course of the intervention.
      • Solberg L.I.
      • Glasgow R.E.
      • Unützer J.
      • et al.
      Partnership research: a practical trial design for evaluation of a natural experiment to improve depression care.
      It also is important to note that many of these pragmatic trials are by necessity cluster trials with randomization at the level of the clinic rather than the individual. However, there are excellent methods for the design and analysis of these trials.
      • Donner A.
      • Klar N.
      Design and Analysis of Cluster Randomization Trials in Health Research.

      Evaluation of DD&I

      Pawson and Tilley best described the objective in DD&I evaluation: “To determine what works for whom in what circumstances.”
      • Pawson R.
      • Tilley N.
      Realistic Evaluation.
      DD&I evaluation emphasizes outcomes in heterogeneous groups of patients and providers and in different settings. It also focuses more on process assessment and qualitative or mixed research methods in an attempt to establish the link between good outcomes, degree of fidelity to the intervention, and differences in the target population and the implementation site. Several models and frameworks have been proposed to evaluate DD&I. Some of these, such as the Precede-Proceed Planning Model, can be applied prospectively during the research design phase to help facilitate spread,
      • Thorpe K.E.
      • Zwarenstein M.
      • Oxman A.D.
      • et al.
      A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
      • Damschroder L.J.
      • Aron D.C.
      • Keith R.E.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
      while others, such as the RE-AIM model developed by Glasgow and colleagues,
      • Glasgow R.E.
      • Vogt T.M.
      • Boles S.M.
      Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
      can be used prospectively in research design and/or for subsequent evaluation of the success of the DD&I effort in the target population. In RE-AIM, “R” stands for reach or how effective the intervention was in reaching the target population. The “E” stands for effectiveness or how effective the intervention was in achieving the desired outcomes. The “A” stands for adoption or what percentage of the target population successfully adopts the intervention. The “I” stands for implementation or what percentage of the time the intervention was implemented with a high degree of fidelity in the target population. This also includes measurement of implementation and start-up costs. The “M” stands for maintenance or how successful the intervention was in maintaining the desired outcomes over an extended period of time. Each of these outcome variables is associated with a specific metric. For example, Reach is measured by the percentage of target audience that agrees to participate, and representativeness of those participants. Ultimately DD&I measurement systems such as RE-AIM can be used to facilitate the design of research that can be more readily spread and to evaluate programs with attention paid to external validity, context, and setting.

      Implications for Researchers

      DD&I research is still in its infancy; many important questions remain unanswered. At the most basic level, there is a need for more detailed study of the DD&I process to learn not only whether or not a program was successfully spread, but also what program characteristics led to the success or failure of the process. This research does not necessarily need to follow formal research guidelines. It may take place within the context of activities such as quality improvement and lean systems redesign. Future research is needed to address the interaction between the intervention and the setting in which it occurs and the impact this contextual information has on the sustainability of the intervention. For example, we will need to understand how potential changes in payment reform contemplated under the Accountable Care Act will change the spread and sustainability of evidenced-based practices in pediatrics. Another important area for future research will be the refinement of a typology or classification of interventions. As previously noted, a good match between the DD&I method/model and the type of intervention is an important predictor of success. Another important area for future research is how knowledge of and intervention with existing social networks can best facilitate the transmission of new ideas. Too often innovators at the periphery of these networks do not communicate with individuals at the center of the network who can act as champions for these ideas.
      New DD&I research will also call for a broader variety of quasiexperimental research designs, including the multiple baselines across settings, time series and regression discontinuity designs (often used in quality improvement and program evaluation efforts),
      • Imbens G.W.
      • Lemieux T.
      Regression discontinuity designs: a guide to practice.
      • Speroff T.
      • O’Connor G.T.
      Study designs for PDSA quality improvement research.
      comparative case studies, comparative effectiveness studies, simulation modeling, and plausibility and adequacy designs.
      • Victora C.G.
      • Habicht J.P.
      • Bryce J.
      Evidence-based public health: moving beyond randomized trials.
      Future DD&I research will require additional funding. As noted above, there is a considerable imbalance of research dollars toward T1 and T2 translational research. Recently the National Institute of Health has begun to address this issue with specific program announcements for DD&I research.

      Dissemination and implementation research in health (R01). Available at: http://grants.nih.gov/grants/guide/pa-files/PAR-10-038.html. Accessed May 22, 2011.

      Implications for Clinicians

      One of the most difficult challenges for the clinician is the translation of new knowledge into practice. DD&I research has shown that new interventions developed in a rigorous research environment cannot be easily taken off the shelf and implemented in the pediatric office. Clinicians have the daunting task of understanding how the intervention will impact their staff and patients as well as what changes in work flow will be required. Once an effective intervention is implemented in practice, there remains a need to measure the degree of adoption and adherence to the intervention and how this changes over time. It will also be important to document clinical outcomes after implementation to ensure that the desired outcomes are still achieved. The resources required to accomplish this are often limited or unavailable in the small to medium pediatric practice. The difficulty in articulating a strong business case to improve quality through the spread of effective interventions makes the task of allocating the needed resources for these efforts even more difficult.
      • Homer C.
      Exploring the business case for improving the quality of health care for children.
      One possibility is to make resources available through professional organizations such as the AAP that have a vested interest in the DD&I of evidence-based practices. New organizational structures such as accountable care organizations may also become a mechanism to provide needed resources to primary care pediatricians.

      Conclusions

      There is a compelling gap in the widespread dissemination of evidence-based practices in child health with a failure to achieve scale up and spread of important improvements in practice. Too often clinical research has not appreciated the exigencies of practice and patient populations that facilitate or impede widespread adoption and implementation. DD&I offers an important framework and set of tools to help close this gap by identifying innovations that are most likely to implemented and spread, and organizations that are able to adopt the proposed changes. It also provides methods to leverage existing social networks to enhance the diffusion of new ideas and to ensure that an intervention is context sensitive without losing fidelity to core components of the scientific evidence. DD&I research has also highlighted new ways of conducting research that will facilitate spread. These include participatory research, pragmatic or practical trials, and the use of various quasiexperimental designs.
      This is a particularly propitious time for the DD&I field. With the advent of health care reform, we are likely to see large demonstration projects designed to more efficiently provide integrated care and address important morbidities. The ultimate success of these projects will have to go beyond proven effectiveness. Their impact on child health care will depend on the ability to spread these effective practices across different settings and target populations, and their ultimate success will depend on our ability to better understand and use the principles and tools of DD&I.

      References

      1. National Health Care Disparities Report.
        Agency for Health Quality and Research, Washington, DC2006
        • Mangione-Smith R.
        • DeCristofaro A.H.
        • Setodji C.M.
        • et al.
        The quality of ambulatory care delivered to children in the United States.
        N Engl J Med. 2007; 357: 1515-1523
        • Sung N.S.
        • Crowley Jr., W.F.
        • Genel M.
        • et al.
        Central challenges facing the national clinical research enterprise.
        JAMA. 2003; 289: 1278-1287
        • Contopoulos-Ioannidis D.G.
        • Ntzani E.
        • Ioannidis J.P.
        Translation of highly promising basic science research into clinical applications.
        Am J Med. 2003; 114: 477-484
        • Lenfant C.
        Clinical research to clinical practice—lost in translation?[Shattuck lecture].
        N Engl J Med. 2003; 349: 868-874
        • Finkelstein J.A.
        • Lozano P.
        • Farber H.J.
        • et al.
        Underuse of controller medications among Medicaid-insured children with asthma.
        Arch Pediatr Adolesc Med. 2002; 156: 562-567
        • Bethell C.
        • Reuland C.H.
        • Halfon N.
        • Schor E.L.
        Measuring the quality of preventive and developmental services for young children: national estimates and patterns of clinicians’ performance.
        Pediatrics. 2004; 113: 1973-1983
        • Bero L.A.
        • Grilli R.
        • Grimshaw J.M.
        • et al.
        Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group.
        BMJ. 1998; 317: 465-468
        • Ranji S.R.
        • Shetty K.
        • Posley K.A.
        • et al.
        Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Vol. 6, Prevention of Healthcare-Associated Infections. Agency for Healthcare Research and Quality, Rockville, Md2007
        • Bravata D.M.
        • Sundaram V.
        • Lewis R.
        • et al.
        Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Vol. 5, Asthma Care. Agency for Healthcare Research and Quality, Rockville, Md2007
        • Grimshaw J.
        • Eccles M.
        • Tetroe J.
        Implementing clinical guidelines: current evidence and future implications.
        J Contin Educ Health Prof. 2004; 24: S31-S37
        • Woolf S.H.
        The meaning of translational research and why it matters.
        JAMA. 2008; 299: 211-213
        • Zerhouni E.A.
        Translational and clinical science—time for a new vision.
        N Engl J Med. 2005; 353: 1621-1623
        • Szilagyi P.G.
        Translational research and pediatrics.
        Acad Pediatr. 2009; 9: 71-80
        • Glasgow R.E.
        • Lichtenstein E.
        • Marcus A.C.
        Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition.
        Am J Public Health. 2003; 93: 1261-1267
        • Zerhouni E.
        Clinical research at the crossroads: The NIH roadmap.
        Science. 2003; 302: 63-72
      2. CTSA Clinical and Translational Science Awards. 2003. Available at: http://www.ctsaweb.org. Accessed October 5, 2010.

        • Moses 3rd, H.
        • Dorsey E.R.
        • Matheson D.H.
        • Thier S.O.
        Financial anatomy of biomedical research.
        JAMA. 2005; 294: 1333-1342
        • Halterman J.S.
        • Aligne C.A.
        • Auinger P.
        • et al.
        Inadequate therapy for asthma among children in the United States.
        Pediatrics. 2000; 105: 272-276
        • Miller M.R.
        • Gergen P.
        • Honour M.
        • et al.
        Burden of illness for children and where we stand in measuring the quality of this health care.
        Ambul Pediatr. 2005; 5: 268-278
        • Broder K.R.
        • Cohn A.C.
        • Schwartz B.
        • et al.
        Adolescent immunizations and other clinical preventive services: a needle and a hook?.
        Pediatrics. 2008; 121: S25-S34
        • Dearing J.
        • Kruetner M.
        Designing for diffusion of cancer communication innovations.
        Parent Educ Couns. 2010; 81: S100-S110
        • Ryan B.
        • Gross N.C.
        The diffusion of hybrid seed corn in two Iowa communities.
        Rural Sociol. 1943; 8: 15-24
        • Rogers E.
        Diffusion of Innovation.
        Free Press, New York, NY2003
        • Gladwell M.
        The Tipping Point: How Little Things Can Make a Big Difference.
        Little, Brown, New York, NY2002
        • Evidence-Based Medicine Working Group
        Evidence-based medicine: a new approach to teaching the practice of medicine.
        JAMA. 1992; 268: 2420-2425
        • Cochrane A.
        Effectiveness and Efficiency: Random Reflections of Health Services.
        Nuffield Provincial Hospitals Trust, London, England1971
      3. Evidence-based practice centers: synthesizing scientific evidence to improve quality and effectiveness in health care. Evidence Based Practice Centers. Available at: http://www.ahrq.gov.laneproxy.stanford.edu/clinic/epc/epcenters.htm. Accessed November 29, 2010.

        • Lomas J.
        The in-between world of knowledge brokering.
        BMJ. 2007; 334: 129
        • Lomas J.
        Words without action? The production, dissemination, and impact of consensus recommendations.
        Annu Rev Public Health. 1991; 12: 41-65
        • Eddy D.M.
        Evidence-based medicine: a unified approach.
        Health Aff (Millwood). 2005; 24: 9-17
        • Fixsen D.N.
        • Naoom S.F.
        • Blase K.A.
        • et al.
        Implementation Research: A Synthesis of the Literature.
        University of South Florida, Tampa, Fla2005
        • Israel B.A.
        • Schulz A.J.
        • Parker E.A.
        • et al.
        Review of community-based research: assessing partnership approaches to improve public health.
        Annu Rev Public Health. 1998; 19: 173-202
        • Minkler M.
        Community-based research partnerships: challenges and opportunities.
        J Urban Health. 2005; 82: ii3-ii12
        • Van Olphen J.
        • Schulz A.
        • Israel B.
        • et al.
        Religious involvement, social support, and health among African-American women on the east side of Detroit.
        J Gen Intern Med. 2003; 18: 549-557
        • Levy J.I.
        • Brugge D.
        • Peters J.L.
        • et al.
        A community based participatory research study of multifaceted in-home environmental interventions for pediatric asthmatics in public housing.
        Soc Sci Med. 2006; 63: 2191-2203
      4. Edwards N. Scaling-up health innovations and interventions in public health: a brief review of the current state-of-the-science. Commissioned for the inaugural Conference to Advance the State of the Science and Practice on Scale-up and Spread of Effective Health Programs, Washington, DC, July 6–8, 2010.

        • Greenhalgh T.
        • Robert G.
        • Macfarlane F.
        • et al.
        Diffusion of innovations in service organizations: systematic review and recommendations.
        Milbank Q. 2004; 82: 581-629
        • Ferlie E.B.
        • Shortell S.M.
        Improving the quality of health care in the United Kingdom and the United States: a framework for change.
        Milbank Q. 2001; 79: 281-315
        • Della Penna R.
        • Martel H.
        • Neuwirth E.B.
        • et al.
        Rapid spread of complex change: a case study in inpatient palliative care.
        BMC Health Serv Res. 2009; 9: 245-251
        • Rabin B.A.
        • Brownson R.C.
        • Kerner J.F.
        • et al.
        Methodologic challenges in disseminating evidence-based interventions to promote physical activity.
        Am J Prev Med. 2006; 31: S24-S34
        • Davidson C.
        Knowledge translation: implications for evaluation.
        New Directions Eval. 2009; 124: 75-87
        • Green L.W.
        • Orleans C.T.
        • Ottoson J.M.
        • et al.
        Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control.
        Am J Prev Med. 2006; 31: S66-S81
      5. Bergman DA. Personal report. American Academy of Pediatrics Committee on Quality Improvement.

        • Dearing J.W.
        • Rogers E.M.
        • Meyer G.
        • et al.
        Social marketing and diffusion-based strategies for communicating with unique populations: HIV prevention in San Francisco.
        J Health Commun. 1996; 1: 343-363
        • Green L.W.
        • Gottlieb N.H.
        • Parecel G.S.
        Diffusion theory extended and applied.
        Adv Health Educ Promot. 1991; 3: 91-117
        • Freeman L.
        The Development of Social Network Analysis.
        Empirical Press, Vancouver, Canada2006
        • Luke D.A.
        • Harris J.K.
        Network analysis in public health: history, methods, and applications.
        Annu Rev Public Health. 2007; 28: 69-93
        • Hawe P.
        • Webster C.
        • Shiell A.
        A glossary of terms for navigating the field of social network analysis.
        J Epidemiol Community Health. 2004; 58: 971-975
        • Beck A.
        • Bergman D.A.
        • Rahm A.K.
        • et al.
        Using implementation and dissemination concepts to spread 21st-century well-child care at a health maintenance organization.
        Perm J. 2009; 13: 10-18
        • Bevan H.R.G.
        • Bate P.
        • Maher L.
        • et al.
        Using a design approach to assist large-scale organizational change: "10 high impact changes" to improve the National Health Service in England.
        J Appl Behav Sci. 2007; 43: 135-152
        • Bradley E.H.
        • Webster T.R.
        • Baker D.
        • et al.
        Translating research into practice: speeding the adoption of innovative health care programs.
        Issue Brief (Commonw Fund). 2004; : 1-12
        • Gardner K.L.
        • Dowden M.
        • Togni S.
        • Bailie R.
        Understanding uptake of continuous quality improvement in Indigenous primary health care: lessons from a multi-site case study of the Audit and Best Practice for Chronic Disease project.
        Implement Sci. 2010; 5: 21-26
        • Wen K.Y.
        • Gustafson D.H.
        • Hawkins R.P.
        • et al.
        Developing and validating a model to predict the success of an IHCS implementation: the Readiness for Implementation Model.
        J Am Med Inform Assoc. 2010; 17: 707-713
        • Bate P.
        • Robert G.
        • Bevan H.
        The next phase of healthcare improvement: what can we learn from social movements?.
        Qual Saf Health Care. 2004; 13: 62-66
        • Pastor M.
        • Ortiz R.
        Making Change: How Social Movements Work and How to Support Them.
        University of Southern California Press, Los Angeles, Calif2009
        • McCannon J.
        Update: 5 Million Lives campaign. Interview by Dan Pastorius.
        Nurs Manage. 2008; 39: 13-18
        • McCannon C.J.
        • Hackbarth A.D.
        • Griffin F.A.
        Miles to go: an introduction to the 5 Million Lives campaign.
        Jt Comm J Qual Patient Saf. 2007; 33: 477-484
        • McCannon C.J.
        • Perla R.J.
        Learning networks for sustainable, large-scale improvement.
        Jt Comm J Qual Patient Saf. 2009; 35: 286-291
      6. Chapter Alliance for Quality Improvement. 2010. Available at: http://www.aap.org/member/chapters/caqi/about.html. Accessed November 23, 2010.

        • Ovretveit J.
        • Staines A.
        Sustained improvement? Findings from an independent case study of the Jonkoping quality program.
        Qual Manag Health Care. 2007; 16: 68-83
      7. Commonwealth Fund. Tranforming safety net clinics to medical homes. October 14, 2010. Available at: http://www.commonwealthfund.org/Multimedia-Center.aspx. Accessed November 30, 2010.

        • Langley G.
        The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.
        2nd ed. Jossey-Bass, San Francisco, Calif2009
        • Fraser S.
        Undressing the Elephant: Why Good Practice Doesn’t Spread in Health Care.
        Lulu Press, Raleigh, NC2008
        • Van de Ven A.H.
        • Polley D.E.
        • Garud R.
        • et al.
        The Innovation Journey.
        Oxford University Press, Oxford1999
        • King T.M.
        • Tandon S.D.
        • Macias M.M.
        • et al.
        Implementing developmental screening and referrals: lessons learned from a national project.
        Pediatrics. 2010; 125: 350-360
        • Klein S.
        • McCarthy D.
        North Carolina’s ABCD program: using community care networks to improve the delivery of childhood developmental screening and referral to early intervention services.
        Issue Brief (Commonw Fund). 2009; 66: 1-28
        • de Koning H.
        • Verver J.P.
        • van den Heuvel J.
        • et al.
        Lean Six Sigma in healthcare.
        J Healthc Qual. 2006; 28: 4-11
        • Bodenheimer T.
        • Wagner E.H.
        • Grumbach K.
        Improving primary care for patients with chronic illness.
        JAMA. 2002; 288: 1775-1779
        • Bodenheimer T.
        • Wagner E.H.
        • Grumbach K.
        Improving primary care for patients with chronic illness: the chronic care model, part 2.
        JAMA. 2002; 288: 1909-1914
        • Toncich G.
        • Cameron P.
        • Virtue E.
        • et al.
        Institute for Health Care Improvement Collaborative Trial to improve process times in an Australian emergency department.
        J Qual Clin Pract. 2000; 20: 79-86
        • Besserman E.
        • Brennan M.
        • Brown 3rd, P.A.
        • et al.
        Multidisciplinary achievement: the collaborative approach to rapid cycle ICU and hospital change.
        Qual Manag Health Care. 1998; 6: 43-51
        • Tunis S.R.
        • Stryer D.B.
        • Clancy C.M.
        Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
        JAMA. 2003; 290: 1624-1632
        • Godwin M.
        • Ruhland L.
        • Casson I.
        • et al.
        Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity.
        BMC Med Res Methodol. 2003; 3: 28
        • Helms P.J.
        “Real world” pragmatic clinical trials: what are they and what do they tell us?.
        Pediat Allergy Immunol. 2002; 13: 4-9
        • Glasgow R.E.
        • Magid D.J.
        • Beck A.
        • et al.
        Practical clinical trials for translating research to practice: design and measurement recommendations.
        Med Care. 2005; 43: 551-557
        • Solberg L.I.
        • Glasgow R.E.
        • Unützer J.
        • et al.
        Partnership research: a practical trial design for evaluation of a natural experiment to improve depression care.
        Med Care. 2010; 48: 575-581
        • Donner A.
        • Klar N.
        Design and Analysis of Cluster Randomization Trials in Health Research.
        Arnold Publishers, London, England2000
        • Pawson R.
        • Tilley N.
        Realistic Evaluation.
        Sage Publishers, London, England1997
        • Thorpe K.E.
        • Zwarenstein M.
        • Oxman A.D.
        • et al.
        A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers.
        CMAJ. 2009; 180: E47-E57
        • Damschroder L.J.
        • Aron D.C.
        • Keith R.E.
        • et al.
        Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
        Implement Sci. 2009; 4: 50
        • Glasgow R.E.
        • Vogt T.M.
        • Boles S.M.
        Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
        Am J Public Health. 1999; 89 ([review]): 1322-1327
        • Imbens G.W.
        • Lemieux T.
        Regression discontinuity designs: a guide to practice.
        J Econometrics. 2007; 142: 615-635
        • Speroff T.
        • O’Connor G.T.
        Study designs for PDSA quality improvement research.
        Qual Manag Health Care. 2004; 13: 17-32
        • Victora C.G.
        • Habicht J.P.
        • Bryce J.
        Evidence-based public health: moving beyond randomized trials.
        Am J Public Health. 2004; 94: 400-405
      8. Dissemination and implementation research in health (R01). Available at: http://grants.nih.gov/grants/guide/pa-files/PAR-10-038.html. Accessed May 22, 2011.

        • Homer C.
        Exploring the business case for improving the quality of health care for children.
        Health Aff. 2004; 23: 159-166