Advertisement

Considering Context in Quality Improvement Interventions and Implementation: Concepts, Frameworks, and Application

  • Kathryn M. McDonald
    Correspondence
    Address correspondence to Kathryn M. McDonald, MM, Stanford University, Center for Health Policy, Center for Primary Care and Outcomes Research, 117 Encina Commons, Stanford, CA 94305-6019.
    Affiliations
    Stanford University School of Medicine, Stanford, Calif, and University of California at Berkeley School of Public Health, Berkeley, Calif
    Search for articles by this author

      Abstract

      Growing consensus within the health care field suggests that context matters and needs more concerted study for helping those who implement and conduct research on quality improvement interventions. Health care delivery system decision makers require information about whether an intervention tested in one context will work in another with some differences from the original site. We aimed to define key terms, enumerate candidate domains for the study of context, provide examples from the pediatric quality improvement literature, and identify potential measures for selected contexts. Key sources include the organizational literature, broad evaluation frameworks, and a recent project in the patient safety area on context sensitivity. The article concludes with limitations and next steps for developments in this area.

      Keywords

      Making mental connections is our most crucial learning tool, the essence of human intelligence; to forge links; to go beyond the given; to see patterns, relationships, context.Marilyn Ferguson
      HEALTH CARE DELIVERY is highly complex, from the patient–clinician encounter to the organizations in which these encounters occur, and is ultimately embedded in and affected by an even more varied environment. This situation makes delivering consistently high-quality care challenging for its participants and ripe for opportunities for improvement. Researchers and practitioners together develop and test quality improvement (QI) strategies, often inside of organizations, common test beds for QI research. Delivery organizations also implement strategies that have succeeded elsewhere. What evidence does an organization (eg, a children's hospital, a patient-centered medical home) need to know about whether a strategy researched in one place will work in another? Efforts to improve quality of health care occur in many different situations and may work better in some and worse in others.
      • Bradley E.H.
      • Nembhard I.M.
      • Yuan C.T.
      • et al.
      What is the experience of national quality campaigns? Views from the field.

      McDonald K, Chang C, Schultz E. Through the Quality Kaleidoscope: Reflections on the Science and Practice of Improving Health Care Quality. Closing the Quality Gap: Revisiting the State of the Science. Methods Research Report. Prepared by Stanford-UCSF Evidence-based Practice Center under Contract No. 290-2007-10062-I. Rockville, Md: Agency for Healthcare Research and Quality; February 2013. AHRQ Publication No. 13-EHC041-EF. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/509/1406/CQG-Reflections-on-Science-130219.pdf. Accessed June 13, 2013.

      McDonald KM, Chang C, Schultz E. Closing the Quality Gap: Revisiting the State of the Science. Summary Report. (Prepared by Stanford-UCSF Evidence-based Practice Center under Contract No. 290-2007-10062-I.) AHRQ Publication No. 12(13)-E017. Rockville, Md: Agency for Healthcare Research and Quality; January 2013. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/496/1375/ClosingtheQualityGap_SummaryReport_20130109.pdf. Accessed June 13, 2013.

      Just as medical treatment is indicated for some patients and not others, interventions aimed at quality improvement may be indicated for some organizations and not others. Appropriate selection of quality improvement strategies requires information about context. But what exactly is context, and how should it be measured to be useful in research and for interpretation in organizational decision making about quality improvement investments?
      Drawing from growing consensus that context matters and needs more concerted study for helping quality improvement decision makers, we sought to define key concepts for the study of context, offer approaches to measure and report context in pediatric quality improvement research, and assess limitations and next steps for future research in this area.
      Early research on improving quality and safety focused on efficacy, without much consideration of context, sparking debates about the relative value of different approaches.
      • Leape L.L.
      • Berwick D.M.
      • Bates D.W.
      What practices will most improve safety? Evidence-based medicine meets patient safety.
      • Shojania K.G.
      • Duncan B.W.
      • McDonald K.M.
      • Wachter R.M.
      Safe but sound: patient safety meets evidence-based medicine.
      The efficacy paradigm prioritizes internal validity over external validity, factoring out contextual variables, simplifying the job for an evaluator.
      • Goodman S.
      Confessions of a chagrined trialist.
      For concrete, easily packaged, and describable quality improvement interventions (QII), such a traditional evidence-based medicine approach, may be applicable: observe a system, introduce a perturbation (the intervention) to some participants but not others, and then observe again. The experiment is designed to determine if the changes in outcome are attributable to the perturbation (with statistical confidence limits). Yet most experiments are short on details about how the intervention works, including the influence of context. For more complex interventions developed to improve quality, the field is moving to incorporate complementary research paradigms that study how context enables, thwarts, and interacts with interventions.
      A useful alternative evaluation model comes from Pawson and Tilley's realist evaluation framework of context, mechanism, and outcome (ie, the CMO model), where “programs work (have successful ‘outcomes’) only insofar as they introduce the appropriate ideas and opportunities (‘mechanisms’) to groups in the appropriate social and cultural conditions (‘contexts’).”
      • Pawson R.
      • Tilley N.
      Realistic Evaluation.
      This model makes the context sensitivity of many QIIs explicit and applies whenever local adaptations are expected to influence outcome patterns. Defining context and considering when and how to study this concept effectively become paramount for assessing transferability of an intervention package from one context to another.
      • Berwick D.M.
      The science of improvement.

      What Is a QII?

      For the purpose of our focus on context sensitivity, a broad definition of a QII is used in order to inform research on efforts to improve quality of pediatric care comprehensively. On the basis of the Robert Woods Johnson Foundation's definition used for evaluating training programs in this domain, quality improvement is “the formal approach to the analysis of performance and systematic efforts to improve it.”

      Holmes SK. Quality improvement training: scanning the field and developing resources. 2011. Available at: http://www.rwjf.org/content/dam/web-assets/2011/12/quality-improvement-training. Accessed December 9, 2012.

      This definition of quality improvement is not restricted to any specific level of analysis (eg, performance could be assessed for a doctor's practice, an emergency department, a hospital, or a geographic area). An intervention is a newly created attempt to improve the performance of the specified unit over time, and it can be described in terms of what could potentially be packaged and replicated elsewhere.
      • Rubenstein L.V.
      • Hempel S.
      • Farmer M.M.
      • et al.
      Finding order in heterogeneity: types of quality-improvement intervention publications.
      Many QIIs have multiple components designed to change behaviors and actions of people, organizations, and even networks or systems of organizations.

      Cochrane Collaboration. Cochrane Effective Practice and Organisation of Care Group. 2012. Available at: http://epoc.cochrane.org/. Accessed December 9, 2012.

      • Shojania K.
      • McDonald K.
      • Wachter R.
      • Owens D.
      Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Volume 1, Series Overview and Methodology.
      As such, they are sometimes termed strategies, initiatives, or programs. Interventions can target any or all of the Institute of Medicine's quality domains (safety, effectiveness, efficiency, timeliness, equity, and patient-centeredness) where performance is lacking.
      • US Institute of Medicine; Committee on Quality of Health Care in America
      Crossing the Quality Chasm: A New Health System for the 21st Century.
      If the point is to improve health care quality, then all of these efforts fall under the QII research umbrella when considering ways to conceptualize, measure, and interpret potential contextual effects. For simplicity's sake, implementation elements that can be described, packaged, and replicated from one environment to another may be considered part of the QII. In this case, the mechanism in the realist CMO model is the QII and its implementation. Another tactic is to distinguish the content of a QII from its implementation, and consider the contextual influences on each (QII and implementation) separately and interactively.

      What Is Context?

      Simply put, context (ie, the C in the realist CMO model) is potentially everything else—anything that cannot be described as part of the QII or the outcomes of interest (ie, the M and the O in the CMO model). But only those contextual factors hypothesized to affect the success of the intervention need to be studied for a given QII, after careful consideration of potential factors and competing hypotheses. Which contextual factors are most likely to matter? Which factors may influence the mechanisms and outcomes favorably or unfavorably? How does one tell a priori? What parts of the QII may require adaptation from one local setting to the next, and one time period to another? The overarching question to ask at the outset of a QII evaluation is, what are the local conditions that could influence the outcomes of interest? The line between the QII and the local conditions may be clear initially and blur during implementation of the intervention. For better knowledge translation, those involved in QII implementation and evaluation need to anticipate potential scale-up and spread to other environments as they describe the QII versus the context in which it was tested. Any blurring between context and QII is often related to perspective. If QII developers believe that local leadership engagement is important to QII success, they may spend time up front gaining buy-in for the new program (the QII). If commitment of a certain package of resources is a prerequisite for implementing the intervention, then the presence of that commitment can be seen as a contextual variable (eg, money and personnel time committed). Alternatively, the QII may be designed with an initial leadership intervention component (eg, letter to the hospital CEO from the investigative team noting project backing from a major health plan in the area) aimed at gaining a particular level of commitment.
      More concretely, the Figure depicts 3 scenarios that have differing combinations of QII components and contexts, but are all effectively the same. The first scenario (labeled “original hospital”) reflects a study by McKee and colleagues,
      • McKee C.
      • Berkowitz I.
      • Cosgrove S.E.
      • et al.
      Reduction of catheter-associated bloodstream infections in pediatric patients: experimentation and reality.
      who describe an improvement model to reduce catheter-associated bloodstream infections at a children's hospital with 6 intervention components. Organizational commitment (ie, enthusiastic leadership, adequate resources) is implied as a contextual factor. In addition, the study authors describe a context responsive to catheter product issues occurring during a period of time when infection rates climbed. In one alternative hypothetical scenario (hospital A), a hospital might implement only 2 components (ie, the checklist and the dedicated cart) because their nurses have already received empowerment training and their physicians have already experienced the necessary type of insertion training. They may also have a real-time infection rate feedback system to the unit leadership. Their QII implementation is simplified in response to their local conditions. On the opposite end of the spectrum, another hospital (hospital B) might choose to implement the originally described QII along with extra interventions to compensate for limitations in their context.
      Figure thumbnail gr1
      FigureQuality improvement intervention (QII) and context distinctions vary depending on setting.
      Thus, context is what is left over after specifying the QII (with implementation components) and may in fact vary from one location (context) to another.
      • Kaplan H.C.
      • Brady P.W.
      • Dritz M.C.
      • et al.
      The influence of context on quality improvement success in health care: a systematic review of the literature.
      • Ovretveit J.
      Understanding the conditions for improvement: research to discover which context influences affect improvement success.
      Such a simplified definition should not mask the challenges inherent in understanding the interplay between a QII and the context, where components of a QII may be tailored to a particular context, and in turn, the context may adapt during implementation of the intervention. The active aspects of the interaction theoretically may differ in structure and impact depending on specifics of the situation, including whether a factor is viewed as endogenous or exogenous to the intervention. There is a need for clear and detailed specification of each part of the QII and its context in any evaluative project, as well as particulars about the ways that each influenced the other, as was reported qualitatively in the McKee et al study.
      • McKee C.
      • Berkowitz I.
      • Cosgrove S.E.
      • et al.
      Reduction of catheter-associated bloodstream infections in pediatric patients: experimentation and reality.

      Context Sensitivity

      Three main audiences increasingly care about context sensitivity of QIIs. Practitioners need to know what QIIs are suitable for implementation in their delivery settings and how to increase the chances of the intervention working. Policy makers need to create incentives to encourage adoption and spread of effective QIIs to the appropriate contexts. Researchers aim to understand context sensitivity in order to help practitioners and policy makers choose wisely among QIIs. The accretion of evidence on a QII results from conducting different types of studies to assess proof of concept, effectiveness in single settings, replication to other settings, and ultimately widespread implementation. These 4 types of research orientations elicit different key framing questions about context (Table 1). The remainder of the article introduces a sampling of commonly used frameworks for identifying relevant contextual domains and measurement tools available for some variables related to context.
      Table 1Contextual Considerations According to Research Phase
      Research PhaseTypical Approach/GoalContext Question
      Process Evaluation/Proof of Concept for a QIIDevelop logic model of mechanisms related to improving quality targetWhat are the enablers and barriers to potential interventions working in our context?
      Initial QII EvaluationUse logic model: design, implement, and test intervention in one setting to evaluate effectivenessWhat context factors should we describe, measure, and/or monitor?
      Further QII Evaluation for Promising InterventionsTest with more rigorous methods, expand to more than one setting to evaluate effectiveness and context sensitivityWhat context factors should we measure consistently across sites?
      Scale-up and Spread of Effective QIIDesign scale-up plan; implement and monitor over timeWhich context factors that influence effectiveness are fixed vs mutable?
      QII indicates quality improvement intervention.

      Contexts to Consider for Applications to Pediatric QII

      Regardless of the particular research question, we need ways to winnow down the long list of potential contexts surrounding an intervention to those that may exert influence on the success (or failure) of a QII. The organizational theory literature, several health care improvement frameworks and a recent interdisciplinary RAND-led project on patient safety practices provide some direction.
      • Foy R.
      • Ovretveit J.
      • Shekelle P.G.
      • et al.
      The role of theory in research to develop and evaluate the implementation of patient safety practices.
      • Ovretveit J.C.
      • Shekelle P.G.
      • Dy S.M.
      • et al.
      How does context affect interventions to improve patient safety? An assessment of evidence from studies of five patient safety practices and proposals for research.

      Shekelle PG, Pronovost PJ, Wachter RM, et al., and the PSP Technical Expert Panel. Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (Prepared by RAND Health under Contract No. HHSA-290-2009-10001C). AHRQ Publication No. 11-0006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2010. Available at: http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context.pdf. Accessed June 13, 2013.

      • Shekelle P.G.
      • Pronovost P.J.
      • Wachter R.M.
      • et al.
      Advancing the science of patient safety.

      Organizational Theory

      A rich source for generating pertinent contextual domains, organizational studies emphasize levels of action (eg, macro view of organizations and their environments, or micro view looking at behavior patterns within an organization), interrelationships (multilevel effects), and field observation for theory testing. For any given QII, therefore, we can start by asking what factors within the macro context might exert influence on the improvement effort, repeat this thought experiment for the micro context, and then hypothesize any interactions between these levels. Some macro analysis categories include: national/regional/local economy and market forces, urban/suburban/rural location, governmental and regulatory variables, and time (when the study was conducted, relevant external events during the study period).
      • Rousseau D.M.
      • Fried Y.
      Location, location, location: contextualizing organizational research.
      At the micro level, possible contextual factors relate to the organization and subunits or teams. The organization level factors potentially applicable to quality improvement include type of health care entity, life cycle stage of entity (eg, new vs established, recently restructured or acquired), structural factors (eg, disciplinary departments vs care delivery processes), financial support for quality improvement in form of actual allocations and/or dedicated human resources, organizational culture (dominant, subgroups, norms/values), management interest in QII, and leadership levels involved (top level, middle management), governance leadership, and learning climate.
      • Burns L.R.
      • Bradley E.H.
      • Weiner B.J.
      • Shortell S.M.
      Shortell and Kaluzny's Health Care Management Organization Design and Behavior.
      • Glisson C.
      • Hemmelgarn A.
      The effects of organizational climate and interorganizational coordination on the quality and outcomes of children's service systems.
      • Morris A.
      • Bloom J.R.
      Contextual factors affecting job satisfaction and organizational commitment in community mental health centers undergoing system changes in the financing of care.
      • Zazzali J.L.
      • Alexander J.A.
      • Shortell S.M.
      • Burns L.R.
      Organizational culture and physician satisfaction with dimensions of group practice.
      A front-line view focuses on what some have termed the clinical microsystem, where patient and provider encounters occur, and from which the following additional contexts may affect QII implementation success: staff factors (eg, focus on hiring decisions and integration into team, performance expectations, education and training, interdependence of care team), patient factors (focus on patient needs, focus on community needs), performance factors (data feedback systems process improvement strategies), and information factors (technology, communication).
      • Foster T.C.
      • Johnson J.K.
      • Nelson E.C.
      • Batalden P.B.
      Using a Malcolm Baldrige framework to understand high-performing clinical microsystems.
      • Nelson E.C.
      • Batalden P.
      • Homa K.
      • et al.
      Data and measurement in clinical microsystems, part 2: creating a rich information environment.
      The specific innovation processes contained in a QII typically occur at the micro level by individuals or groups of health care professionals; these micro level processes are supported, enhanced, or prevented by the macro conditions.

      Kanter, RM. When a Thousand Flowers Bloom: Structural, Collective, and Social Conditions for Innovation in Organizations. In Myers, PS, ed: Knowledge Management and Organizational Design. Newton, MA: Butterworth-Heinemann; 1996:93–132.

      The interactions between micro and macro levels can influence the QII's success during early stages of implementation and especially during scale-up and spread of the intervention. Behavioral and organizational context factors also influence sustainability of QII, with several intervention–context interactions suggested by researchers from the Department of Veterans Affairs Quality Enhancement Research Initiative on the basis of analysis of long-term effects of QIIs: intervention fit, intervention fidelity, intervention dose, and intervention target.
      • Bowman C.C.
      • Sobo E.J.
      • Asch S.M.
      • Gifford A.L.
      HIV/Hepatitis Quality Enhancement Research Initiative. Measuring persistence of implementation: QUERI series.
      A systematic review of QIIs for pediatric asthma care exemplifies the current limits on contextual analysis among these types of studies.
      • Bravata D.M.
      • Gienger A.L.
      • Holty J.E.
      • et al.
      Quality improvement strategies for children with asthma: a systematic review.
      The review of 79 QI interventions for asthma in children found that many different QIIs (eg, promotion of self-monitoring/self-management, patient education to increase asthma knowledge or inhaler technique without targeting patient behavior change, organizational change) improve process and outcomes of care. The only context factors available from all studies were time related (evaluations from 1974 to 2004, study duration from 4 weeks to 5 years), region of performance (most in the United States [n = 46], Australia [n = 7], the United Kingdom [n = 6], the Netherlands [n = 4], or Canada [n = 4]), and setting (eg, outpatient clinics, home, and school). Only 2 contextual factors were associated with QII outcomes; the longer the study for the self-management subgroup, the greater the expected reduction in asthma-related school absenteeism, and the more recent the study for some QII subgroups, the better the results. In the accompanying evidence report, the review provided a narrative synthesis of findings by setting.
      • Bravata D.M.
      • Sundaram V.
      • Lewis R.
      • et al.
      Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Volume 5, Asthma Care.
      For example, patient education QIIs targeted at children, their families, and other adults were subdivided into those carried out in part or fully in a school, doctor's office, or home environment. Some of the designs of these QIIs followed behavior change theory and intervened in these different environmental contexts as a result. Systematic reviews offer an opportunity to balance internal validity with applicability to specific contexts. Currently, however, primary QII studies often provide limited information to allow a rich analysis of context across studies.

      Existing Health Care Improvement Frameworks

      Two complementary frameworks developed for health care improvement applications are described briefly: the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework, and the Consolidated Framework for Implementation Research (CFIR), along with several pediatric QII applications.
      • Damschroder L.J.
      • Aron D.C.
      • Keith R.E.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
      • Glasgow R.E.
      • Vogt T.M.
      • Boles S.M.
      Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
      The RE-AIM framework was developed as a tool for planning, evaluation, and reporting of research translation into practice projects. With an orientation toward all phases of translation of complex interventions, it steps one through contextual factors at stages of program or intervention analysis applicable to QIIs. It has been applied widely to health promotion programs, with pediatric applications mostly school-based public health programs (eg, to increase physical activity).
      • Janssen M.
      • Toussaint H.M.
      • van Mechelen W.
      • Verhagen E.A.
      Translating the PLAYgrounds program into practice: a process evaluation using the RE-AIM framework.
      Applications to systems-oriented improvement interventions include use of RE-AIM to develop and evaluate a Web-based behavior and self-management tool within the patient-centered medical home setting.
      • Glasgow R.E.
      • Dickinson P.
      • Fisher L.
      • et al.
      Use of RE-AIM to develop a multi-media facilitation tool for the patient-centered medical home.
      Some sample contextual factors elicited from the framework application are practice workflow considerations (“adoption” context), and integration with National Committee for Quality Assurance recommendations and clinical pathways (“maintenance” context within the framework).
      • Glasgow R.E.
      • Dickinson P.
      • Fisher L.
      • et al.
      Use of RE-AIM to develop a multi-media facilitation tool for the patient-centered medical home.
      Users of RE-AIM often write up their findings from mixed-methods research for each of the evaluation domains (reach, effectiveness, adoption, implementation, maintenance). This framework is particularly applicable to the scale-up and spread phase of QIIs (Table 1).
      The CFIR comprises constructs from 19 published implementation theories.
      • Damschroder L.J.
      • Aron D.C.
      • Keith R.E.
      • et al.
      Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
      The intent is complementary to the needs of pediatric QII researchers and practitioners—a structure for identifying candidate constructs to potentially measure or otherwise observe during different phases of QII research and adoption. The framework has 5 topic areas: interventions characteristics, outer setting, inner setting, characteristics of individuals, and process. The first and last topics relate more to the intervention and implementation, while the other 3 more closely tie to context domains. The distinctions between intervention, context (inner, outer settings, and individuals acting in each), and the process of implementation can be transient depending on circumstances, so that the entire framework may be useful for identifying relevant contexts.
      • Curran G.M.
      • Bauer M.
      • Mittman B.
      • et al.
      Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact.
      This basic structure also corresponds with the PARiHS framework key domains of evidence, context, and facilitation.
      • Kitson A.L.
      • Rycroft-Malone J.
      • Harvey G.
      • et al.
      Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges.
      As these different frameworks build on each other with some developed from basic research findings from social sciences,
      • Greenhalgh T.
      • Robert G.
      • Macfarlane F.
      • et al.
      Diffusion of innovations in service organizations: systematic review and recommendations.
      it is not surprising to see alignment with the organizational literature's approach (eg, the micro level captures inner setting and individuals, and macro level relates to the outer setting). The CFIR domains are expanded and explained with detailed appendix materials and mappings to original sources to support application for future QII research. It seems that the CFIR is too new to have a published application of a pediatric QII. Table 2 supplies an application based on information available from a recently published QII on reducing pediatric identification band errors.
      • Phillips S.C.
      • Saysana M.
      • Worley S.
      • Hain P.D.
      Reduction in pediatric identification band errors: a quality collaborative.
      Table 2Application of the Consolidated Framework for Implementation Research (CFIR) to a Pediatric Quality Improvement Intervention (QII)
      Pediatric Identification (ID) Band Errors Quality Collaborative

      Goals: Spread QII from a single children's hospital across 6 sites to reduce pediatric patient ID band error rates. Decrease ID band error rate by 50%. Demonstrate learning across institutions drives results.
      CFIR Context DomainsDomain Description

      Example from Pediatric QII Collaborative
      OUTER SETTING
       Patient Needs and ResourcesThe extent to which patient needs, barriers and facilitators known and prioritized
      • Issues with ID band discomfort and resultant removal identified by baseline survey. One size/type of ID band not good for all children. In response, some hospitals worked with vendors to develop bands for vulnerable patients (skin integrity, difficulty keeping band in place in NICU patients).
      • Tailoring original site materials (patient and family educational handouts about the importance of the patient ID band).
       CosmopolitanismDegree organization is networked with other external organizations
      • The collaborative included some variety in organizations
       Peer PressureMimetic or competitive pressure to implement QII
      • No information on this context.
       External Policy and IncentivesExternal strategies to spread interventions
      • Improving patient ID first goal of the Joint Commission's National Patient Safety Goals since 2003.
      INNER SETTING
       Structural CharacteristicsSocial architecture, age, maturity, and size of an organization
      • Freestanding children's hospitals; children's hospitals within academic medical centers; community hospitals. Data on size included.
       Networks and CommunicationsNature and quality of social networks, formal and informal communication
      • Six collaborating organizations, monthly collaborative calls to prioritize “key drivers” (reasons for problem) using baseline data and employee survey information. Used drivers to brainstorm potential interventions.
       CultureNorms, values, and basic assumptions of organization
      • Noted need for further work to understand a specific culture impediment (tendency for caregivers to assume patient is already correctly banded).
       Implementation ClimateCapacity for change, shared receptivity to QII, extent to which use QII rewarded, supported, and expected within organization
      • One site used audits as part of a manager incentive program
      • Sharing stories of band errors and patient impact helped staff understand importance of patient ID bands.
       Readiness for ImplementationTangible indicators of organizational commitment to QII
      • Leadership engagement noted on safety walk-rounds.
      • For all the institutions, collaboration among nurses, physicians, and quality professionals ensured collaborative was a priority. Data sharing at all levels.
      INDIVIDUAL CHARACTERISTICS
       Knowledge and Beliefs About the InterventionIndividuals' attitudes toward and familiarity with QII
      • Survey used to identify attitudes at outset (eg, belief that checking the band not needed because “I know my patient”).
       Self-efficacyIndividual belief in their own capabilities to act
      • Engaging all levels strengthened desire to change and will to sustain gains made.
       Individual Stage of ChangePhase an individual is in, in terms of progresses toward skilled, enthusiastic, and sustained use of QII
      • Each “failure” was an opportunity to provide just-in-time education to participants (staff, family) about importance of ID bands to safe care.
       Individual Identification With OrganizationBroad construct related to individual's perception of and commitment to the organization
      • No information on this context.
       Other Personal AttributesBroad construct including tolerance of ambiguity, intellectual ability, motivation, values, competence, capacity, and learning style
      • No information on this context.

      Recent Interdisciplinary Patient Safety Project

      This project reviewed the relative merits of different methods for evaluating context-sensitive effectiveness of interventions based on literature reviews to identify contextual factors and structured input from an interdisciplinary panel of experts comprising international patient safety leaders, clinicians, policy makers, social scientists, and methodologists.

      Shekelle PG, Pronovost PJ, Wachter RM, et al., and the PSP Technical Expert Panel. Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (Prepared by RAND Health under Contract No. HHSA-290-2009-10001C). AHRQ Publication No. 11-0006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2010. Available at: http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context.pdf. Accessed June 13, 2013.

      Four key domains of context with associated factors were prioritized (Table 3), with one domain corresponding to outer settings in the CFIR structure and 3 domains connected to the inner setting. The expert panel judged the importance of assessing each factor relative to 5 representative patient safety practices separately. Table 3 provides the panel assessments of potential importance of these context factors for interventions to improve medication reconciliation (ie, the QII, with patient safety as the target for improvement). The project also recommended combined research approaches that simultaneously seek rigor in effectiveness estimation, report key contextual factors, and, if possible, assess contextual influences on the outcomes for different research goals (eg, effectiveness, implementation experience, and adoption or spread).
      Table 3RAND Interdisciplinary Patient Safety Practices Project: Prioritized Contexts
      Context Domain

      [CFIR Analog]
      FactorsImportance to Specific QII?

      Example: Medication Reconciliation
      Y indicates yes; N, no, not expected to influence effectiveness of medication reconciliation interventions; based on technical expert panel input (see RAND report for details19).
      1. Structural Organizational Characteristics

      [Inner Setting: Structural Characteristics]
      • Size
      • Location
      • Financial status
      • Academic status
      • Organizational complexity
      • Date of study
      • Volume
      • Existing quality/safety infrastructure
      • Space/physical environment
      • Experience with information technology
      • Physician ownership
      • Y
      • N
      • N
      • Y
      • N
      • N
      • N
      • Y
      • N
      • N
      • N
      2. External Factors

      [Outer Setting: Peer Pressure; External Policy and Incentives]
      Regulatory requirement

      Payments or penalties

      Local sentinel event

      Marketplace competition

      Competing demands
      Y

      Y

      N

      N

      N
      3. Patient Safety Culture, Teamwork, Leadership

      [Inner Setting: Culture; Network/Communication; Readiness for Implementation]
      • Patient safety culture—organizational level
      • Patient safety culture—unit level
      • Teamwork—organizational level
      • Teamwork—unit level
      • Leadership—organizational level
      • Leadership—unit level
      • Y
      • Y
      • N
      • Y
      • Y
      • Y
      4. Implementation and Management Tools

      [Inner Setting: Implementation Climate]
      • Staff education and training
      • Designated staff time to implement
      • Use of audit and feedback
      • Person responsible for implementation
      • Internal incentives
      • Local tailoring or iterative process
      • Help desk support
      • Extent of project management
      • Timeline of implementation
      • Implementation process—one unit at a time or all at once
      • Y
      • Y
      • Y
      • Y
      • N
      • Y
      • N
      • N
      • N
      • N
      Y indicates yes; N, no, not expected to influence effectiveness of medication reconciliation interventions; based on technical expert panel input (see RAND report for details

      Shekelle PG, Pronovost PJ, Wachter RM, et al., and the PSP Technical Expert Panel. Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (Prepared by RAND Health under Contract No. HHSA-290-2009-10001C). AHRQ Publication No. 11-0006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2010. Available at: http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context.pdf. Accessed June 13, 2013.

      ).

      Which Context (or Contexts) to Measure?

      Deciding which contextual factors to include in a study is based on theoretical and practical considerations. From a theory-based perspective, a logic model can help prioritize the contextual factors that are deemed most likely to enhance or limit the QII's success with respect to the specific research goals. It is also worthwhile (for future adopters and systematic reviewers) to describe any contextual factors that may not vary in the tested environment, but could be different and have an effect in another implementation setting. Practically, the key decision driver is feasibility with respect to measurability (measurement tool availability for the construct), its burden to those assessed when measurement is based on survey methods and costs to the evaluation team. Some contexts are relatively easy to measure or specify, such as location (rural or urban; country), organizational type (academic or community; nursing home, outpatient clinic, hospital), and applicable regulatory requirements (public reporting of immunization rates). A systematic review by Kaplan et al identified 47 QII articles and classified 66 contextual factors and the specific measures used in the primary studies.
      • Kaplan H.C.
      • Brady P.W.
      • Dritz M.C.
      • et al.
      The influence of context on quality improvement success in health care: a systematic review of the literature.
      Other contexts pose measurement challenges or choices with potential tradeoffs. The patient safety practices project identified some measurement tools for 4 context areas (teamwork, leadership, patient safety culture, and organizational complexity) based on an extensive literature search in health care databases.

      Shekelle PG, Pronovost PJ, Wachter RM, et al., and the PSP Technical Expert Panel. Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (Prepared by RAND Health under Contract No. HHSA-290-2009-10001C). AHRQ Publication No. 11-0006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2010. Available at: http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context.pdf. Accessed June 13, 2013.

      Because of the limited measures netted from this source for organizational complexity measures, the team also conducted a limited search of the organizational literature. Many options exist for teamwork, leadership, and culture, though some are setting specific and more or less labor intensive, while others are still under development. The patient safety practices expert group supported several specific measures for the types of applications in that project, and these choices may be applicable (or adaptable with further measure development effort) to the pediatric QII domain as well: for teamwork and leadership, the Shortell et al ICU Nurse–Physician Questionnaire, for leadership alone, either the Leadership Practice Inventory, or the Practice Environment Scale.
      • Shortell S.M.
      • Rousseau D.M.
      • Gillies R.R.
      • et al.
      Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse–physician questionnaire.
      • Tourangeau A.E.
      • McGilton K.
      Measuring leadership practices of nurses using the Leadership Practices Inventory.
      • Lake E.T.
      Development of the practice environment scale of the Nursing Work Index.
      In addition, the Multifactor Leadership Questionnaire (MLQ), which is used to assess transformational learning style in nurse executives, has been developed and tested in children’s hospitals.
      • Bass B.M.
      • Avolio B.J.
      Improving Organizational Effectiveness Through Transformational Leadership.
      • Dunham-Taylor J.
      Nurse executive transformational leadership found in participative organizations.
      • Avolio B.J.
      • Bass B.M.
      • Jung D.I.
      Multifactor Leadership Questionnaire Technical Report.
      Consensus was lacking for a choice among the patient safety culture measures, and more general measures of organizational culture applicable to quality improvement were not assessed. Since that project's report, a comparison showed similar reliability and predictive validity of 2 widely used culture surveys: Safety Attitudes Questionnaire (SAQ) and Hospital Survey on Patient Safety Culture (HSOPS).
      • Etchegaray J.M.
      • Thomas E.J.
      Comparing two safety culture surveys: safety attitudes questionnaire and hospital survey on patient safety.
      Few options were identified for organizational complexity, and the only ones applied to health care were limited to the domain of care coordination, where other measures resources exist.
      • Shortell S.M.
      • Rousseau D.M.
      • Gillies R.R.
      • et al.
      Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse–physician questionnaire.
      • McDonald K.
      • Schultz E.
      • Albin L.
      • et al.
      Care Coordination Atlas.
      • Sicotte C.
      • Pineault R.
      • Lambert J.
      Medical team interdependence as a determinant of use of clinical resources.

      Reporting Contexts in Evaluations of QII

      An important step in strengthening the research base on the role of context is uptake of the context elements of SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for publishing studies of QIIs.

      SQUIRE: Standards for Quality Improvement Reporting Excellence. Available at: http://squire-statement.org/guidelines. Accessed December 9, 2012.

      • Ogrinc G.
      • Mooney S.E.
      • Estrada C.
      • et al.
      The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration.
      Specifically, these guidelines are divided into 19 items (eg, abstract, analysis, funding sources), of which 6 explicitly note contextual factors. The guidelines for SQUIRE item 5, “intended improvement,” should include “who (champions, supporters) and what (events, observations) triggered the decision to make changes, and why now (timing).” Item 8, “setting,” should specify “how elements of the local care environment considered most likely to influence change/improvement in the involved site or sites were identified and characterized.” Item 9, “planning the intervention,” should include information on factors contributing to the choice of the intervention (ie, “suitability of the intervention for the specific setting”), as well as factors related to those implementing the intervention (ie, “roles, qualification and training”). Item 11, “methods of evaluation,” should describe the “contribution of … context factors to effectiveness of the intervention.” Item 13, “outcomes,” should characterize “relevant elements of setting or settings (for example, geography, physical resources, organizational culture, history of change efforts), and structures and patterns of care (eg, staffing, leadership) that provided context for the intervention” and present “evidence regarding the strength of association between observed changes/improvements and intervention components/context factors.” Finally, Item 17, “interpretation,” should draw inferences, “paying particular attention to components of the intervention and context factors that helped determine the intervention's effectiveness (or lack thereof), and types of settings in which this intervention is most likely to be effective.” These items collectively offer an initial context sensitivity checklist for researchers.

      Conclusions and Next Steps

      Efficient and effective means to incorporate the domain of context into research on QIIs has received relatively minimal attention in health care, even though the salience of this broad topic is well understood by practitioners and policy makers.
      • Batalden P.
      • Davidoff F.
      • Marshall M.
      • et al.
      So what? Now what? Exploring, understanding and using the epistemologies that inform the improvement of healthcare.
      • Batalden P.
      • Bate P.
      • Webb D.
      • McLoughlin V.
      Planning and leading a multidisciplinary colloquium to explore the epistemology of improvement.
      This article should be looked on as a sampling of materials to offer potential, but not exhaustive, starting points for further development of domain lists and measurement options for a given construct.
      Evaluations of complex interventions directed at improving quality need to consider QIIs in terms of core transferable elements and adaptive aspects that respond to a local context.
      • Greenhalgh T.
      • Robert G.
      • Macfarlane F.
      • et al.
      Diffusion of innovations in service organizations: systematic review and recommendations.
      Teasing apart the interrelationships requires a more dedicated focus on defining context and developing a strategy to assess its influence on and interactions with interventions. Researchers and practitioners whose attention is directed to child health already have a solid appreciation for issues of translating evidence from one domain to another, knowing that what works for adults will not necessarily apply to children. Complex QII are analogous: they may translate from one context to another, but such a conclusion is premature without considering the logic behind the likelihood of intervention mobility, the possible effects of context, and the need for triangulating the evidence from different types of research studies and practical experiences. Understanding context influence is akin to diagnosis. Differential diagnosis of context is an ongoing process to assure that no important context is missed, and this review of resources brings together concepts, domains and measures to consider the influence of specific contexts on QIIs. Further systematic and collaborative work among those who evaluate new QIIs and those who decide whether to implement a promising QII could lead to the development of context evaluation tools tailored to pediatric applications for research and quality improvement.
      Funders and researchers may consider developing repositories of measurement tools for the context domains of importance to QII, noting any that are particularly relevant to pediatric applications. For the national priority area of care coordination, the Agency for Healthcare Quality and Research sponsored a project to develop a Care Coordination Measures Atlas, a resource that includes a measurement framework, identified measures with acceptable performance characteristics and maps of these measures to framework domains.
      • McDonald K.
      • Schultz E.
      • Albin L.
      • et al.
      Care Coordination Atlas.
      A similar resource would be useful for those involved in QII evaluations from proof of concept through spread of successful interventions with widespread applicability (ie, intervention exhibits limited context sensitivity or workarounds to the local context are well understood). Finally, reporting context and the QII in ways aligned with the SQUIRE guidelines

      SQUIRE: Standards for Quality Improvement Reporting Excellence. Available at: http://squire-statement.org/guidelines. Accessed December 9, 2012.

      • Ogrinc G.
      • Mooney S.E.
      • Estrada C.
      • et al.
      The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration.
      is crucial to advancing knowledge about what interventions work, and in what ways context sensitivity may play a role in success or failure of a particular QII.

      References

        • Bradley E.H.
        • Nembhard I.M.
        • Yuan C.T.
        • et al.
        What is the experience of national quality campaigns? Views from the field.
        Health Services Res. 2010; 45: 1651-1669
      1. McDonald K, Chang C, Schultz E. Through the Quality Kaleidoscope: Reflections on the Science and Practice of Improving Health Care Quality. Closing the Quality Gap: Revisiting the State of the Science. Methods Research Report. Prepared by Stanford-UCSF Evidence-based Practice Center under Contract No. 290-2007-10062-I. Rockville, Md: Agency for Healthcare Research and Quality; February 2013. AHRQ Publication No. 13-EHC041-EF. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/509/1406/CQG-Reflections-on-Science-130219.pdf. Accessed June 13, 2013.

      2. McDonald KM, Chang C, Schultz E. Closing the Quality Gap: Revisiting the State of the Science. Summary Report. (Prepared by Stanford-UCSF Evidence-based Practice Center under Contract No. 290-2007-10062-I.) AHRQ Publication No. 12(13)-E017. Rockville, Md: Agency for Healthcare Research and Quality; January 2013. Available at: http://effectivehealthcare.ahrq.gov/ehc/products/496/1375/ClosingtheQualityGap_SummaryReport_20130109.pdf. Accessed June 13, 2013.

        • Leape L.L.
        • Berwick D.M.
        • Bates D.W.
        What practices will most improve safety? Evidence-based medicine meets patient safety.
        JAMA. 2002; 288: 501-507
        • Shojania K.G.
        • Duncan B.W.
        • McDonald K.M.
        • Wachter R.M.
        Safe but sound: patient safety meets evidence-based medicine.
        JAMA. 2002; 288: 508-513
        • Goodman S.
        Confessions of a chagrined trialist.
        BMJ Qual Saf. 2011; 20: i97-i98
        • Pawson R.
        • Tilley N.
        Realistic Evaluation.
        Sage, Thousand Oaks, Calif1997
        • Berwick D.M.
        The science of improvement.
        JAMA. 2008; 299: 1182-1184
      3. Holmes SK. Quality improvement training: scanning the field and developing resources. 2011. Available at: http://www.rwjf.org/content/dam/web-assets/2011/12/quality-improvement-training. Accessed December 9, 2012.

        • Rubenstein L.V.
        • Hempel S.
        • Farmer M.M.
        • et al.
        Finding order in heterogeneity: types of quality-improvement intervention publications.
        Qual Saf Health Care. 2008; 17: 403-408
      4. Cochrane Collaboration. Cochrane Effective Practice and Organisation of Care Group. 2012. Available at: http://epoc.cochrane.org/. Accessed December 9, 2012.

        • Shojania K.
        • McDonald K.
        • Wachter R.
        • Owens D.
        Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Volume 1, Series Overview and Methodology.
        Agency for Healthcare Research and Quality, Rockville, MdAugust 2004 (Publication 04-0051-1)
        • US Institute of Medicine; Committee on Quality of Health Care in America
        Crossing the Quality Chasm: A New Health System for the 21st Century.
        National Academies Press, Washington, DC2001
        • McKee C.
        • Berkowitz I.
        • Cosgrove S.E.
        • et al.
        Reduction of catheter-associated bloodstream infections in pediatric patients: experimentation and reality.
        Pediatr Crit Care Med. 2008; 9: 40-46
        • Kaplan H.C.
        • Brady P.W.
        • Dritz M.C.
        • et al.
        The influence of context on quality improvement success in health care: a systematic review of the literature.
        Milbank Q. 2010; 88: 500-559
        • Ovretveit J.
        Understanding the conditions for improvement: research to discover which context influences affect improvement success.
        BMJ Qual Saf. 2011; 20: i18-i23
        • Foy R.
        • Ovretveit J.
        • Shekelle P.G.
        • et al.
        The role of theory in research to develop and evaluate the implementation of patient safety practices.
        BMJ Qual Saf. 2011; 20: 453-459
        • Ovretveit J.C.
        • Shekelle P.G.
        • Dy S.M.
        • et al.
        How does context affect interventions to improve patient safety? An assessment of evidence from studies of five patient safety practices and proposals for research.
        BMJ Qual Saf. 2011; 20: 604-610
      5. Shekelle PG, Pronovost PJ, Wachter RM, et al., and the PSP Technical Expert Panel. Assessing the Evidence for Context-Sensitive Effectiveness and Safety of Patient Safety Practices: Developing Criteria (Prepared by RAND Health under Contract No. HHSA-290-2009-10001C). AHRQ Publication No. 11-0006-EF. Rockville, MD: Agency for Healthcare Research and Quality; December 2010. Available at: http://www.ahrq.gov/research/findings/final-reports/contextsensitive/context.pdf. Accessed June 13, 2013.

        • Shekelle P.G.
        • Pronovost P.J.
        • Wachter R.M.
        • et al.
        Advancing the science of patient safety.
        Ann Intern Med. 2011; 154: 693-696
        • Rousseau D.M.
        • Fried Y.
        Location, location, location: contextualizing organizational research.
        J Organization Behav. 2001; 22: 1-13
        • Burns L.R.
        • Bradley E.H.
        • Weiner B.J.
        • Shortell S.M.
        Shortell and Kaluzny's Health Care Management Organization Design and Behavior.
        6th ed. Delmar Cengage Learning, Clifton Park, NY2012
        • Glisson C.
        • Hemmelgarn A.
        The effects of organizational climate and interorganizational coordination on the quality and outcomes of children's service systems.
        Child Abuse Neglect. 1998; 22: 401-421
        • Morris A.
        • Bloom J.R.
        Contextual factors affecting job satisfaction and organizational commitment in community mental health centers undergoing system changes in the financing of care.
        Mental Health Services Res. 2002; 4: 71-83
        • Zazzali J.L.
        • Alexander J.A.
        • Shortell S.M.
        • Burns L.R.
        Organizational culture and physician satisfaction with dimensions of group practice.
        Health Services Res. 2007; 42: 1150-1176
        • Foster T.C.
        • Johnson J.K.
        • Nelson E.C.
        • Batalden P.B.
        Using a Malcolm Baldrige framework to understand high-performing clinical microsystems.
        Qual Saf Health Care. 2007; 16: 334-341
        • Nelson E.C.
        • Batalden P.
        • Homa K.
        • et al.
        Data and measurement in clinical microsystems, part 2: creating a rich information environment.
        Joint Commission J Qual Saf. 2003; 29: 5-15
      6. Kanter, RM. When a Thousand Flowers Bloom: Structural, Collective, and Social Conditions for Innovation in Organizations. In Myers, PS, ed: Knowledge Management and Organizational Design. Newton, MA: Butterworth-Heinemann; 1996:93–132.

        • Bowman C.C.
        • Sobo E.J.
        • Asch S.M.
        • Gifford A.L.
        HIV/Hepatitis Quality Enhancement Research Initiative. Measuring persistence of implementation: QUERI series.
        Implement Sci. 2008; 3: 21
        • Bravata D.M.
        • Gienger A.L.
        • Holty J.E.
        • et al.
        Quality improvement strategies for children with asthma: a systematic review.
        Arch Pediatr Adolesc Med. 2009; 163: 572-581
        • Bravata D.M.
        • Sundaram V.
        • Lewis R.
        • et al.
        Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies. Volume 5, Asthma Care.
        Agency for Healthcare Research and Quality, Rockville, Md2007
        • Damschroder L.J.
        • Aron D.C.
        • Keith R.E.
        • et al.
        Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.
        Implement Sci. 2009; 4: 50
        • Glasgow R.E.
        • Vogt T.M.
        • Boles S.M.
        Evaluating the public health impact of health promotion interventions: the RE-AIM framework.
        Am J Public Health. 1999; 89: 1322-1327
        • Janssen M.
        • Toussaint H.M.
        • van Mechelen W.
        • Verhagen E.A.
        Translating the PLAYgrounds program into practice: a process evaluation using the RE-AIM framework.
        J Sci Med Sport. July 21, 2012;
        • Glasgow R.E.
        • Dickinson P.
        • Fisher L.
        • et al.
        Use of RE-AIM to develop a multi-media facilitation tool for the patient-centered medical home.
        Implement Sci. 2011; 6: 118
        • Curran G.M.
        • Bauer M.
        • Mittman B.
        • et al.
        Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact.
        Med Care. 2012; 50: 217-226
        • Kitson A.L.
        • Rycroft-Malone J.
        • Harvey G.
        • et al.
        Evaluating the successful implementation of evidence into practice using the PARiHS framework: theoretical and practical challenges.
        Implement Sci. 2008; 3: 1
        • Greenhalgh T.
        • Robert G.
        • Macfarlane F.
        • et al.
        Diffusion of innovations in service organizations: systematic review and recommendations.
        Milbank Q. 2004; 82: 581-629
        • Phillips S.C.
        • Saysana M.
        • Worley S.
        • Hain P.D.
        Reduction in pediatric identification band errors: a quality collaborative.
        Pediatrics. 2012; 129: e1587-e1593
        • Lake E.T.
        Development of the practice environment scale of the Nursing Work Index.
        Res Nurs Health. 2002; 25: 176-188
        • Shortell S.M.
        • Rousseau D.M.
        • Gillies R.R.
        • et al.
        Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse–physician questionnaire.
        Med Care. 1991; 29: 709-726
        • Tourangeau A.E.
        • McGilton K.
        Measuring leadership practices of nurses using the Leadership Practices Inventory.
        Nurs Res. 2004; 53: 182-189
        • Avolio B.J.
        • Bass B.M.
        • Jung D.I.
        Multifactor Leadership Questionnaire Technical Report.
        Mind Garden, Redwood City, Calif1995
        • Bass B.M.
        • Avolio B.J.
        Improving Organizational Effectiveness Through Transformational Leadership.
        Sage Publications, Thousand Oaks, Calif1994
        • Dunham-Taylor J.
        Nurse executive transformational leadership found in participative organizations.
        J Nurs Admin. 2000; 30: 241-250
        • Etchegaray J.M.
        • Thomas E.J.
        Comparing two safety culture surveys: safety attitudes questionnaire and hospital survey on patient safety.
        BMJ Qual Saf. 2012; 21: 490-498
        • Sicotte C.
        • Pineault R.
        • Lambert J.
        Medical team interdependence as a determinant of use of clinical resources.
        Health Services Res. 1993; 28: 599-621
        • McDonald K.
        • Schultz E.
        • Albin L.
        • et al.
        Care Coordination Atlas.
        Agency for Healthcare Research and Quality, Rockville, MdJanuary 2011
        • Ogrinc G.
        • Mooney S.E.
        • Estrada C.
        • et al.
        The SQUIRE (Standards for Quality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration.
        Qual Saf Health Care. 2008; 17: i13-i32
      7. SQUIRE: Standards for Quality Improvement Reporting Excellence. Available at: http://squire-statement.org/guidelines. Accessed December 9, 2012.

        • Batalden P.
        • Bate P.
        • Webb D.
        • McLoughlin V.
        Planning and leading a multidisciplinary colloquium to explore the epistemology of improvement.
        BMJ Qual Saf. 2011; 20: i1-i4
        • Batalden P.
        • Davidoff F.
        • Marshall M.
        • et al.
        So what? Now what? Exploring, understanding and using the epistemologies that inform the improvement of healthcare.
        BMJ Qual Saf. 2011; 20: i99-i105