Abstract
Objective
Context is a critical determinant of the effectiveness of quality improvement programs. We assessed the role of contextual factors in influencing the efforts of 5 diverse quality improvement projects as part of the Pediatric Quality Measure Program (PQMP) directed by the Agency for Health Care Research and Quality.
Methods
We conducted a mixed methods study of 5 PQMP grantees involving semistructured interviews followed by structured worksheets to identify influential contextual factors. Semistructured interviews and worksheets were completed between August and October 2020. Participants were comprised of PQMP grantee teams (2–4 team members per team for a total of 15 participants). Coding and analysis was based on the Tailored Implementation for Chronic Diseases (TICD) framework.
Results
Despite heterogeneity in the process and outcome targets of the PQMP initiatives, professional interactions, incentives and resources, and capacity for organizational change were the domains most commonly identified as influential across the grantees. While social, political, and legal factors was not commonly referenced as an important domain, payer or funder policies (a factor within this domain) was highlighted as one of the most influential factors. Overall, the incentives and resources domain was identified as the most influential.
Conclusions
We found that using a determinant framework, such as the TICD, is valuable in facilitating comparisons across heterogeneous projects, allowing us to identify key contextual factors influencing the implementation of pediatric quality measures across a diverse range of clinical topics and settings. Future quality improvement work should account for this and include resources to support infrastructure development in addition to program implementation.
Keywords
Context is a critical determinant of the effectiveness of any quality improvement (QI) program.
1
,2
By systematically assessing the role of contextual factors in a program's implementation, we can gain a better understanding of how the intervention worked, with lessons for others conducting similar efforts.3
Historically, implementation science frameworks have been used to provide a conceptual underpinning to analyze the influence of contextual factors for a range of interventions, including pediatric QI, primary care practice transformation, prevention and wellness projects, among others.3
, 4
, - Stover AM
- Haverman L
- van Oers HA
- et al.
Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings.
Qual Life Res. 2020; ([e-pub ahead of print])https://doi.org/10.1007/s11136-020-02564-9
5
, 6
As part of the first phase of the Pediatric Quality Measure Program (PQMP) directed by the Agency for Health Care Research and Quality (AHRQ), 7 Centers of Excellence developed new and enhanced pediatric quality measures covering a range of clinical conditions. These quality measures were developed to increase the portfolio of measures available to public and private purchasers of children's health care services under the Children's Health Insurance Program Reauthorization Act of 2009, Title IV, Section 401. In the second phase of the PQMP, 6 grantees assessed the feasibility and usability of the newly developed measures for QI. Each of the grantees launched and led one or more quality improvement collaboratives (QICs) to support QI activities targeting their respective measures.
QICs are a commonly used QI or implementation strategy that brings together multiple health care organizations to focus on the same goal and includes opportunities to share learnings and best practices in support of improving performance.
7
Given that the focus of each grantee's QIC was tied to one of the quality measures developed in the first PQMP phase, the QI work supported by the QICs reflected a variety of clinical conditions, measure types (process, outcome, experience), and data sources (claims, electronic health record [EHR], survey). Depending on the measure, the QICs engaged with individuals and organizations at different, and sometimes multiple, levels of the health care system—spanning patient, provider, health system, health plan, and state levels. Despite differences in the clinical focus and the level of the health care system at which each QIC was operating, all of the QICs aimed to improve systems, processes, and/or long-term outcomes of pediatric care among QIC participants.The purpose of this analysis was to use an implementation science determinants framework to identify which, if any, contextual factors were common barriers or facilitators to improving pediatric care across a group of diverse QICs conducted as part of phase II of the AHRQ PQMP program.
8
Methods
Overview
We conducted a mixed methods study to identify the factors that facilitated or impeded the successful implementation of system and process changes within the PQMP QICs using the Tailored Implementation for Chronic Diseases (TICD) framework.
8
The TICD is a comprehensive, integrated checklist of determinants of implementation success organized into 7 domains that include factors that might impact implementation. The 7 domains include: guideline or quality measure factors, individual health professional factors, patient factors, professional interactions, incentives and resources, capacity for organizational change, and social/political/legal factors. Consolidating other determinants frameworks, the TICD offers specificity and flexibility as it can be applied across different levels of organizations or systems, which was a key component of this analysis necessitated by how the QICs were organized.We carried out data collection in 2 phases: first, we conducted semistructured interviews to obtain information about the QICs and respective system and process changes implemented by the QIC participants; second, we collected information to capture details about contextual factors influencing implementation of the system and process changes using structured worksheets based on the TICD domains. The first phase of data collection was used to inform the second phase. This paper focuses on the findings from the second phase of data collection.
Selection of QIC Projects
While some grantees executed more than one QIC to support QI targeted to multiple quality measures, for the purpose of this evaluation, we focused on activities related to one QIC and one measure from 5 of the grantees. We excluded one grantee because neither of their QICs advanced to a phase of active QI during our evaluation period. In consultation with the grantees and AHRQ, we identified specific system and process changes implemented by QIC participants as part of the QI activities within each QIC. Selection was driven by measures/system changes that progressed the most during the period of performance. Characteristics of the included measures and QICs are highlighted in Table 1. We focused our inquiry on key contextual barriers and facilitators of implementation of these specific system and process changes.
Table 1QICs Featured in the IS Analysis
| PQMP Grantee | QIC Goals | QIC Participants | QIC Duration | Measure Targeted for Improvement | Intervention Selection Process | Main System and Process Changes |
|---|---|---|---|---|---|---|
| CEPQM | Develop and implement a shared intervention across sites to garner shared learnings for reducing hospital readmissions | Hospitals (n = 5) | 36 months | Pediatric 30-day all-condition hospital readmissions (NQF # 2393) | All QIC participants agreed to implement the same intervention through a consensus-based decision-making process | Parental confidence assessment: Hospitals developed and implemented a parental confidence assessment—focused on assessing parents’ confidence in discharge instructions—that could be administered to parents before their child was discharged from the hospital |
| IMPLEMENT | - Successfully establish and continue engagement in the collaborative - Improve asthma outcomes, specifically the number of child emergency department visits for asthma | Primary care practices (n = 11) | 12 months, plus 9 months extension | Rate of emergency department visit use for children managed for persistent asthma | Grantee presented QIC participants with a menu of intermediate measures; QIC participants selected from these measures and then developed their own interventions | Electronic health record (EHR) improvements for child asthma-related process measures: Participating practices focused mostly on improvements related to implementing decision support features and improving the sites’ documentation capabilities to help capture and track the process measures. Strategies included: reconfigured visit report pull from EHR, smart forms, dot/smart phrases, asthma patient identification |
| NCINQ II | Improve metabolic monitoring in children and adolescents prescribed an antipsychotic | Health plans (n = 5) | 18 months | Metabolic monitoring for children and adolescents on antipsychotics (NQF # 2800) | Grantee presented QIC participants with a menu of intervention options that could be tailored to participant specifications | Health plan-level metabolic monitoring improvement strategies: All the plans focused on implementing safe and judicious practices for antipsychotic prescribing among youth—they selected the strategy that was best suited to their context. Strategies included: member outreach, standing lab order, data exchange, provider outreach, incentive program |
| P-HIP | Improve participating hospitals’ performance on counseling parents/caregivers regarding restricting access to potentially lethal means of suicide from their child/ adolescent treated for suicidality prior to discharge home and documentation in the medical record that the counseling was completed | Hospitals (n = 8) | 12 months | Children/adolescents admitted to the hospital for dangerous self-harm or suicidality, should have documentation in the hospital record that their caregiver was counseled on how to restrict the child's/adolescent's access to potentially lethal means of suicide (eg, firearms, medications, car, etc.) prior to discharge | Grantee presented QIC participants with a menu of intervention options that could be tailored to participant specifications | Implementing a standardized documentation strategy for counseling through their EHR: Teams introduced a statement in the discharge template about counseling on the restriction of lethal means to serve as a reminder/prompt, and to ensure that documentation was completed |
| Q-METRIC | - Improve care for children with sickle cell disease and specifically improve rates of appropriate antibiotic prophylaxis for children with sickle cell anemia - Successfully identify target patient population - Continued engagement in the collaborative | State (n = 1) Health plans (n = 3) Clinics (n = 2) | 12 months | Appropriate antibiotic prophylaxis for children with sickle cell anemia (NQF # 3166) | Grantee presented QIC participants with a menu of intervention options that could be tailored to participant specifications | Developing patient registries/databases for children with sickle cell anemia: Sites developed approaches to identifying children with sickle cell anemia in their clinics or plans |
QIC indicates quality improvement collaborative; IS, implementation science; and PQMP, Pediatric Quality Measure Program.
Semistructured Interviews
In August 2020, we conducted semistructured telephone interviews (supplemental file available upon request) with each of the grantees to elicit details about the nature of the QIC they convened. Grantees self-selected team members (from 2 to 4 members) to participate in the interviews. All teams included the principal investigator. Each interview was led by an experienced researcher (E.C. or M.J.). Interviews were recorded and transcribed. Interview notes and transcripts were independently reviewed by 2 research team members (M.J. and M.K.), with each extracting descriptive information about the QICs and system and process changes implemented. Results were compared and discrepancies resolved through consensus. Extracted information was used to prepopulate the first section of the worksheet for grantee review and confirmation and to provide background for the research team to better understand how each QIC functioned.
Worksheets
Between September and October 2020, each grantee was asked to complete a structured worksheet to identify relevant contextual factors that impacted QIC participants’ implementation of system/process changes (supplemental file available upon request). Worksheet design was informed by the approach of Tomoaia-Cotisel et al,
5
in which authors used structured worksheets to collect information on contextual factors from 14 distinct research teams. Wording leveraged supplementary materials published by Flottorp et al,8
the developers of the TICD checklist. Worksheets were tailored to each grantee by incorporating descriptive details of their QICs from the interviews. Grantees were asked to elaborate on how relevant factors in each TICD domain applied in their QIC context. Separately, grantees were asked to select the contextual factors they deemed “most influential” overall in the ability of QIC participants to successfully make system and process changes (allowing for several). Grantees completed the worksheets using free text. Typically, worksheets were completed by the same grantee team members as those who participated in the semistructured interviews.All 5 worksheets were collected from grantees and independently coded by 2 research team members (M.J. and M.K.). The research team performed deductive content analysis of free text data from the worksheets using the TICD as a template. Specifically, contextual factors were coded using TICD domains, subdomains, and individual factors as a priori codes, noting whether factors were barriers and/or facilitators. Coding results were reviewed and compared. Any coding discrepancies were discussed and resolved through consensus (E.C., M.J., and M.K.). Factors that were indicated by grantees as “most influential,” that is, those that had the most impact on implementation efforts, among their QIC participants were noted. Domains with the highest proportion of factors identified as relevant or “most influential” across grantee QIC participants were considered most salient and are referred to as such in the results. The team identified quotations from the worksheet responses to illustrate key themes or observations, particularly among the most salient domains.
This work was a component of a learning collaborative effort associated with the grantees’ cooperative agreement awards, interviewees were only grantees and asked only about topics associated with the specific QICs established under their cooperative agreement awards. Interviewees were fully informed of how the information would be used and no sensitive information was collected.
Results
We interviewed a total of 15 participants (2–4 per grantee team). All grantee teams included the principal investigator. Additional participants included one or more of the following depending on the grantee team: project manager, implementation lead, QI lead, and/or lead learning collaborative facilitator. In some cases, the same person served multiple roles. These interviewees were also responsible for completing the structured worksheets.
There were contextual factors within each of the TICD domains that were identified as important barriers and facilitators of successful implementation of system and process changes across the 5 QICs as shown in the data collected from the grantees’ worksheets (Table 2).
Indicates factor was identified by grantee as being relevant as a barrier or facilitator for one or more of their quality improvement collaborative participants.
Table 2Factors Identified by Grantees
| Factor | Definition | CEPQM | IMPLEMENT | NCINQ II | P-HIP | Q-METRIC |
|---|---|---|---|---|---|---|
| 1. Guideline factors | ||||||
| A. Quality measure | ||||||
| Quality of evidence supporting the measure | How confident we are in the evidence base, reliability, and validity of the quality measure | |||||
| Strength of the quality measure | How confident we are that the desirable effects of improving the quality measure outweigh the undesirable effects | |||||
| Clarity | The clearness of the target population, the settings in which the quality measure is to be used, and the recommended change strategy | |||||
| Cultural appropriateness | The extent to which the quality measure is suitable in the social context where it is being implemented | |||||
| Accessibility of the quality measure | How accessible the specifications are | |||||
| Source of the quality measure | The organization(s) and people that developed the quality measure | |||||
| Consistency with other practices | The extent to which the quality measure and associated change strategy are consistent with other guidelines with which the targeted health care professionals might be familiar | |||||
| B. Recommended intervention | ||||||
| Feasibility | The extent to which implementing the quality measure and recommended change strategy is practical | |||||
| Accessibility of the intervention | The extent to which the quality measure and recommended change strategy is accessible | |||||
| C. Recommended behavior | ||||||
| Compatibility | The extent to which the recommended behaviors associated with the change strategy fits with current practices | |||||
| Effort | The amount of effort required to change or adhere | |||||
| Trialability | The ability to try out the recommended behaviors associated with the change strategy | |||||
| Observability | The degree to which benefits of the change strategy are visible | |||||
| 2. Individual health professional factors | ||||||
| A. Knowledge and skills | ||||||
| Domain knowledge | The extent to which the targeted health care professionals have pre-existing knowledge or expertise about the targeted condition/practice | |||||
| Awareness and familiarity with the quality measure | The extent to which the targeted health care professionals are aware of and familiar with the change strategy | |||||
| Knowledge about own practice | The extent to which the targeted health care professionals are aware of their own practice in relationship to the change strategy | |||||
| Skills needed to adhere | The extent to which the targeted health professionals have skills that they need to adhere | |||||
| B. Cognitions | ||||||
| Agreement with the quality measure | The extent to which the targeted health care professionals agree with the quality measure and associated change strategies | |||||
| Attitudes toward interventions in general | The perceptions that the targeted health care professionals have regarding improvement change strategies in general | |||||
| Expected outcome | The extent to which the targeted health care professionals believe that adherence with the change strategy will lead to desired outcomes | |||||
| Intention and motivation | The extent to which the targeted health care professionals intend to adhere and are motivated to do so | |||||
| Self-efficacy | The targeted health care professionals’ self-perceived competence or confidence in their abilities | |||||
| Learning style | The preferred ways in which the targeted health care professionals learn | |||||
| Emotions | The extent to which emotions affect adherence | |||||
| C. Professional behavior | ||||||
| Nature of the behavior | Characteristics of the behavior, including: frequency of performance for a patient, frequency of performance for a population of patients, the degree of habit or automaticity, whether it is within a sequence of other behaviors that have to be performed, and whether it is performed by one person or by different people | |||||
| Capacity to plan change | The extent to which the targeted health care professionals have the capacity to plan necessary changes in order to adhere | |||||
| Self-monitoring or feedback | The extent to which the targeted health care professionals have the capacity for self-monitoring or feedback to reinforce adherence with the change strategy | |||||
| 3. Patient and family factors | ||||||
| Patient and family needs | Real or perceived needs and demands of the patient/family | |||||
| Patient and family beliefs and knowledge | Patients’/families’ beliefs or knowledge or ability to learn, or the targeted health care professionals’ ability or perceived ability to inform or teach patients/families necessary knowledge and skills | |||||
| Patient and family preferences | Patients’/families’ values in relationship to professional values or those in the change strategy | |||||
| Patient and family motivation | The targeted health care professionals’ ability or perceived ability to motivate patients/families to adhere | |||||
| Patient and family behavior | Patient/family behaviors that motivate or demotivate adherence with the change strategy | |||||
| 4. Professional interactions | ||||||
| Communication and influence | The extent to which the targeted health care professionals’ adherence is influenced by professional opinions and communication | |||||
| Team processes | The extent to which professional teams or groups have the skills needed to adhere and interact in ways that facilitate or hinder adherence | |||||
| Referral processes | Processes for transferring patients and communication between different levels of care, between health and social services, and between the targeted health care professionals and targeted patients | |||||
| 5. Incentives and resources | ||||||
| Availability of necessary resources | The extent to which the resources that are needed to adhere are available | |||||
| Financial incentives and disincentives | The extent to which patients/families, individual health professionals and organizations have financial incentives or disincentives to adhere | |||||
| Nonfinancial incentives and disincentives | The extent to which patients/families, individual health professionals and organizations have nonfinancial incentives or disincentives to adhere | |||||
| Information system | The extent to which the information system facilitates or hinders adherence | |||||
| Quality assurance and patient safety systems | The extent to which existing quality assurance or patient safety systems facilitate or hinder adherence | |||||
| Continuing education system | The extent to which the continuing education system facilitates or hinders adherence | |||||
| Assistance for clinicians | The extent to which clinicians have the assistance they need to adhere | |||||
| 6. Capacity for organizational change | ||||||
| Mandate, authority, accountability | The mandate, authority, and mechanisms of accountability for making necessary changes | |||||
| Capable leadership | The extent to which clinical leaders or managers are capable of making necessary changes | |||||
| Relative strength of supporters and opponents | The extent of support and/or opposition to necessary changes | |||||
| Regulations, rules, policies | The extent to which organizational regulations, rules, or policies facilitate or hinder necessary changes | |||||
| Priority of necessary change | The relative priority given to making necessary changes | |||||
| Monitoring and feedback | The extent to which monitoring and feedback are needed at the organizational level and available to sustain necessary changes (including evaluations of improvement programs) | |||||
| Assistance for organizational changes | The extent to which external support is needed and available for necessary changes | |||||
| 7. Social, political, and legal factors | ||||||
| Economic constraints on the health care budget | Limits on the total health care budget or its growth | |||||
| Contracts | The extent to which contracts may affect implementation of necessary changes | |||||
| Legislation | The extent to which legislation may affect implementation of necessary changes | |||||
| Payer or funder policies | The extent to which payer or funder policies may affect implementation of necessary changes | |||||
| Malpractice liability | The extent to which malpractice liability may affect implementation of necessary changes | |||||
| Influential people | The extent to which influential people may affect implementation of necessary changes | |||||
| Corruption | The extent to which corruption may affect implementation of necessary changes | |||||
| Political stability | The extent to which political stability may affect implementation of necessary changes |
† Indicates factor was identified by grantee as being among the “most influential” factors for one or more of their quality improvement collaborative participants.
‡ Indicates domain was identified as most salient during the analysis. Domains with the highest proportion of factors identified as relevant or “most influential” across grantees were considered most salient.
§ Definitions were adapted using the supplementary Tailored Implementation for Chronic Diseases materials published by Flottorp et al
8
.Most Salient Domains
The “incentives and resources,” “professional interactions,” and “capacity for organizational change” domains were identified as most salient across the QICs. Importantly, contextual factors within these domains were identified as important despite the heterogeneity in the level of the health care system where changes were made, the types of system and process changes implemented, and the complexity of the clinical areas of focus.
Incentives and Resources
Contextual factors within this domain were identified as important barriers and facilitators of successful implementation of system and process changes universally by grantees in their worksheet responses. Not only was this domain commonly discussed by grantees in free text responses, but several factors under this domain were also designated as “most influential.” The 2 factors most commonly identified as having an impact on QIC implementation efforts were: the availability of necessary resources and information systems. Availability of necessary resources was reported as both a barrier and facilitator. For example, in the CEPQM QIC, as related to their strategy for developing and implementing a parental confidence discharge assessment, the grantee commented that “ultimately, resources—staff, QI expertise, and IRB expertise—ended up being critically important for getting this intervention off the ground. Even if there is leadership buy-in, without the additional resources, hospitals were unable to participate.” Another grantee, Q-METRIC, reported that, within their QIC, without adequate resources—financial and staffing—it was challenging to make short-term progress and even more difficult to sustain long-term improvements in appropriate antibiotic prophylaxis for children with sickle cell anemia. This grantee noted that a lack of extrinsic incentives exacerbated the issue of resource availability, commenting “Without incentives or an extrinsic requirement to report, it is unlikely that adequate organizational resourcing and support will be made available to improve performance rates.” The grantee supporting the NCINQ QIC had a similar observation, noting, “One of the most impactful factors in implementing the measures and strategies [for improving metabolic monitoring] was the existence of state mandate and existing infrastructure/mechanism for reporting … this created an incentive for providers and health plans to focus resources on this small, but highly vulnerable population….” Information systems were also noted as a key factor. In nearly all cases, information systems were facilitators to implementation efforts, particularly in cases where the information systems were able to be provided centrally by the QIC. For example, both the CEPQM and NCINQ QICs used their existing information systems to run performance reports to monitor their QIC participants’ progress. Others, like the P-HIP and IMPLEMENT QICs leveraged and built out their information systems as part of the QIC activities to support the sustainability of improvement efforts.
Professional Interactions
Within this domain, the most commonly cited factor was team processes, which is the extent to which professional teams have the skills needed to facilitate or impede implementation. Although the clinical focus of each QIC varied (Table 1), all of the QICs were focused on making system and process changes that involved members of different departments and/or disciplines. Within the P-HIP QIC which focused on caring for children and adolescents admitted to the hospital for dangerous self-harm or suicidality, the grantee noted: “Setting expectations and clearly defining roles for conducting and documenting the counseling was essential given the many staff and departments involved…. Teams accomplished this in a variety of ways: having a dedicated role staffed 24/7 that was responsible for the counseling; …required training program for new residents who were responsible for educating parents/caregivers; and creating accountability for doing both the counseling and the documentation.” Another grantee, highlighted an innovative approach that one health plan participating in the NCINQ QIC undertook to ensure a team-based approach for metabolic monitoring for children and adolescents on antipsychotics: “having a pharmacist on their QI team, which the other health plans had not done, but learned was a valuable approach through the QIC and explored incorporating it in their own QI work.” The pharmacist had developed a corporate-wide strategy to encourage safer monitoring that involved, “standing lab orders … and pharmacy soft edits (to encourage discussion of alternate treatment options).”
Capacity for Organizational Change
Contextual factors within this domain were the most commonly identified barriers and facilitators of successful implementation of system and process changes, specifically capable leadership; priorities of necessary change; mandate, authority, accountability; and monitoring and feedback. Referencing the importance of capable leadership, grantees identified this factor as both a facilitator and a barrier. Grantees stated that having engaged and “bought-in” leaders improved resource allocation and focus on the QIC efforts. This was noted specifically by the grantee team supporting the NCINQ QIC. On the other hand, grantees reported that those participants with less involved leaders suffered from fewer resources, and at one of the participating sites, the project risked being de-prioritized (IMPLEMENT). Grantees also noted that priorities of necessary change, impacted their QICs’ success. The IMPLEMENT QIC, which focused on EHR improvements for child asthma-related processes, reported that this factor was both a facilitator and a barrier. They shared that participants’ “ability to modify or customize an EHR requires time and resources… Unless the particular measure is a priority for an organization, changes to the EHR may require a practice to ‘wait their turn in line’.” Similarly, other grantees reported that QIC participants struggled with competing priorities drawing resources and support away from their QI work as part of the QIC. On the other hand, within the P-HIP QIC, priorities of necessary change was identified as a facilitator. The grantee speculated that the pressure the QIC participants felt to reduce rates of suicide among children led to this QI work being prioritized by participating organizations.
Other Important Domains
While the other domains of the TICD were less consistently reported as relevant among grantees, there were individual contextual factors within these domains that appeared to be critical barriers and facilitators of successful implementation of system and process changes in both the qualitative and quantitative (Table 2) data provided by grantees, though findings were more heterogeneous. The importance of these factors appeared to be dependent on the specific context of the QIC, and there were few cross-QIC patterns based broadly on clinical topic (eg, behavioral health) or QIC participant composition (eg, health system level).
Social, political, and legal factors were the least referenced domain; however, 2 grantees reported that payer or funder policies was among their “most influential” factors impacting their QICs. The fact that the IMPLEMENT QIC was supported as part of the AHRQ PQMP program made the collaborative stand-out among other QI project options, facilitating recruitment of primary care practice participants: “…this group came together because of the unique opportunity to test the usability of a new pediatric quality measure.” Conversely, the Q-METRIC QIC attributed some of their limited measure uptake to a lack of funder policies, “it is very difficult to convince systems or plans to invest resources when other required priorities exist.”
Within the domain of Guideline Factors, the most commonly cited factors were quality of evidence supporting the measure and source of the quality measure. While the domain overall was not identified as one of the most influential among grantees, the specific factor of quality of evidence supporting the measure was identified among the “most influential” factors impacting implementing success. This is not surprising given that a core criterion of a PQMP measure is having a robust evidence base.
Among the most commonly cited factors related to Individual Health Professional Factors were domain knowledge, awareness and familiarity with the quality measure, and intention and motivation. Nearly all grantees noted that dedicated and motivated health professionals who were already familiar with the quality measure were well-positioned to facilitate implementation within their QIC participating sites. Nature of the behavior and capacity to plan change were also highlighted as key factors, with one grantee in particular noting that their QIC's strategy to “hardwire changes” into their team processes served as a facilitator for easier adoption.
Finally, the most commonly cited factor within the domain of patient factors was patient and family beliefs and knowledge. This factor was reported to be important particularly among QICs where system and process changes required engagement directly with patients and families, and principally when timing was an important part of the process. For example, participants in the CEPQM QIC administered their parental discharge readiness assessment near the time of discharge, when parents may have been reluctant to acknowledge their lack of confidence in the discharge instructions to expedite taking their child home. In another case, the grantee found that the counseling strategy implemented by participants in the P-HIP QIC worked best within the first 3 days of admission, allowing parents time to adjust to the circumstances.
Discussion
Our study sought to identify common influential contextual factors across a set of 5, highly heterogeneous pediatric QI initiatives. These QI initiatives differed in terms of type of participants, clinical topics, measures of focus, and specific system and process changes being implemented. While each initiative supported participants’ QI efforts using a QIC model, the differing approaches used by these QIC's to support improvement varied and added to the heterogeneity. Within QICs, participants were generally given the freedom to identify system and process changes that would be best suited for their own setting—meaning specific system and process changes varied within as well as across QICs. In addition, the timeframe allowed for QIC activity varied across these initiatives with some QIC's occurring over 12 months and others over 36 months. While this amount of heterogeneity generates substantial complexity in the evaluation, it allowed us to identify a number of key contextual factors that are common barriers and facilitators of pediatric quality initiatives across the board, regardless of QI approach or clinical focus.
The key contextual factors across domains that seemed to transcend the heterogeneity across QICs and were consistently highlighted as important barriers and facilitators included team processes, availability of necessary resources, information systems, capable leadership and priorities of necessary change. The need for organizational systems and infrastructure to support improvement was consistently identified in the QI initiatives, a finding that is aligned with the broader literature demonstrating the critical role of organizational, systemic, and cultural factors affecting implementation.
5
,9
Additionally, these findings reinforce the central need for available resources to enable successful implementation, consistent with other studies.10
, - Higgins A
- Murphy R
- Downes C
- et al.
Factors impacting the implementation of a psychoeducation intervention within the mental health system: a multisite study using the consolidation framework for implementation research.
BMC Health Serv Res. 2020; 20 (Published 2020 Nov 9): 1023https://doi.org/10.1186/s12913-020-05852-9
11
, 12
, 13
Not surprisingly, because QI, at its core, is data-driven, having a data infrastructure to monitor changes in processes and outcomes of care related to the target quality metric was identified as important in all QICs. By creating a centralized and automated structure, information systems provide a powerful tool for organizations to support staff in carrying out their QI initiatives. As noted by Karltun et al, a consistent information and knowledge management infrastructure is crucial for clinical QI as it creates reinforcing enabling conditions, even across diverse health care settings.
14
Given its relevance across all QICs, information systems clearly represent a key factor for similar QI initiatives to consider—and leverage—as a way to systematically support implementation.Observations from our evaluation also provide insights not only into the important role that extrinsic incentives, such as state mandates or required reporting of quality measures, play in procuring resources, but also into how external sponsors (like QICs or state agencies) can provide infrastructure for data collection and reporting of pediatric quality measures. In addition, reflections on the role of payer or funder policies by some grantees underscore the role that entities within the sociopolitical environment play in either facilitating or hindering successful improvement. Particularly important for pediatric QI is the observation that measures in the CMS Child Core Set or those that are aligned with QI focused incentive programs received greater attention by QIC participant leadership.
Ensuring the local context is set up to support QI efforts targeting pediatric quality measures is critical and, ideally, should be addressed before beginning the QI initiative. Based on our analysis, it is clear that resources—human, infrastructure, financial—are needed to support effective pediatric focused QI activities. However, securing, and having dedicated resources can be challenging. There is an opportunity for Federal and state level governments to play a more substantial role in providing resources, whether through the creation of grants to support more pediatric QI work (eg, PQMP-like opportunities), by advancing value-based payment opportunities through Medicaid and CHIP that incentivize engagement in pediatric QI work, or through supporting registries that link varied data sources and can be used to track key quality measures. Lastly, the decision to adequately resource a QI effort is directly linked to whether that initiative was prioritized by the organization, and, in many organizations, pediatric quality is not identified as a top priority. Prioritization of pediatric quality by payers and health systems is an important first step in securing the resources needed for effective pediatric QI work.
There were some notable limitations to this analysis. First, we collected data from the individuals who managed and led the QICs, so our findings may not represent the full range of experiences from the QIC participants. The retrospective nature of the project may also have led to underreporting of some factors. Also, due to the variation both in how the QICs defined success and in the maturity of the projects, we were not able to adequately assess success as measured by improvement in the pediatric quality measure across the projects, so we focused on those factors that influenced the implementation of system and process changes. Finally, the small sample size and high heterogeneity precludes inferential analysis, such as assessing associations between influential contextual factors and QIC characteristics.
Despite the difficulties inherent in the retrospective data collection and analysis used in this study, it is important to make sense out of complex implementation projects to facilitate future implementation efforts across disparate sites. This is particularly important in pediatrics. Compared to the adult care setting, pediatrics is characterized by less funding and fewer resources, for a number of reasons including its comparatively smaller and less burdened population, as well as the reality that a higher proportion of children come from low-income families and communities, compared with adults.
15
As we found, these resource limitations can have real impact on the ability to carry out effective QI. While QI in pediatrics has been an active area for some time,16
this study is the first of its kind to explore commonly salient contextual factors that affect QI functioning across heterogeneous health system actors, clinical foci, measures, and change strategies. Based on our findings, as well as those in other studies, ensuring adequacy of resources at both an individual and collective (QIC) level will be essential to supporting effective activities in the future.7
, 9
, 17
Conclusions
We found that using a determinant framework is valuable in facilitating comparisons and learning across heterogeneous QI projects. Through the use of semistructured interviews and worksheets based on the TICD framework, we were able to identify key contextual factors influencing the implementation of pediatric quality measures across a diverse range of clinical topics and settings. We found that factors related to the professional interactions, incentives and resources, and capacity for organizational change domains appeared to transcend measure topics and clinical settings as important drivers of implementation efforts.
Acknowledgments
The authors extend their gratitude to the individuals from the Pediatric Quality Measures Program grantee teams featured in this article; the authors' team is very grateful for their contributions, including technical guidance, support with data collection, and review of this manuscript. The authors also thank Robyn Sagatov, PhD, MHS, RDN from the Agency for Healthcare Quality and Research for her insightful reviews.
Financial statement: This project was conducted under contract to Agency for Healthcare Research and Quality (contract number HHSA290201400003I/75Q80119F32002).
The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
This article is published as part of a supplement sponsored by the US Department of Health and Human Services, the Centers for Medicare and Medicaid Services, and the Agency for Healthcare Research and Quality.
References
- Understanding the conditions for improvement: research to discover which context influences affect improvement success.BMJ Qual Saf. 2011; 20: i18-i23
- The influence of context on quality improvement success in health care: a systematic review of the literature.Milbank Q. 2010; 88: 500-559https://doi.org/10.1111/j.1468-0009.2010.00611.x
- Considering context in quality improvement interventions and implementation: concepts, frameworks, and application.Acad Pediatr. 2013; 13: S45-S53https://doi.org/10.1016/j.acap.2013.04.013
- Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings.Qual Life Res. 2020; ([e-pub ahead of print])https://doi.org/10.1007/s11136-020-02564-9
- Context matters: the experience of 14 research teams in systematically reporting contextual factors important for practice change.Ann Fam Med. 2013; 11: S115-S123https://doi.org/10.1370/afm.1549
- A mixed methods approach to evaluate partnerships and implementation of the Massachusetts Prevention and Wellness Trust Fund.Front Public Health. 2018; 6 (Published 2018 Jun 5): 150https://doi.org/10.3389/fpubh.2018.00150
- How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review.Implement Sci. 2020; 15: 27https://doi.org/10.1186/s13012-020-0978-z
- A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice.Implement Sci. 2013; 8 (Published 2013 Mar 23): 35https://doi.org/10.1186/1748-5908-8-35
- Contextual factors that influence quality improvement implementation in primary care: the role of organizations, teams, and individuals.Health Care Manage Rev. 2018; 43: 261-269https://doi.org/10.1097/HMR.0000000000000194
- Factors impacting the implementation of a psychoeducation intervention within the mental health system: a multisite study using the consolidation framework for implementation research.BMC Health Serv Res. 2020; 20 (Published 2020 Nov 9): 1023https://doi.org/10.1186/s12913-020-05852-9
- Avoiding unnecessary hospitalisation for patients with chronic conditions: a systematic review of implementation determinants for hospital avoidance programmes.Implement Sci. 2020; 15 (Published 2020 Oct 21): 91https://doi.org/10.1186/s13012-020-01049-0
- Why do we fail to deliver evidence-based practice in critical care medicine?.Curr Opin Crit Care. 2017; 23: 400-405https://doi.org/10.1097/MCC.0000000000000436
- Key ingredients for implementing intensive outpatient programs within patient-centered medical homes: a literature review and qualitative analysis.Healthc (Amst). 2016; 4: 22-29https://doi.org/10.1016/j.hjdsi.2015.12.005
- Knowledge management infrastructure to support quality improvement: a qualitative study of maternity services in four European hospitals [published correction appears in Health Policy. 2020 May;124:575].Health Policy. 2020; 124: 205-215https://doi.org/10.1016/j.healthpol.2019.11.005
- Basic Facts About Low-Income Children: Children Under 9 Years, 2019.National Center for Children in Poverty, Bank Street College of Education, New York2021
- Quality improvement in pediatrics: past, present, and future.Pediatr Res. 2017; 81: 156-161https://doi.org/10.1038/pr.2016.192
- The influence of contextual factors on healthcare quality improvement initiatives: a realist review.Syst Rev. 2020; 9 (Published 2020 Apr 26): 94https://doi.org/10.1186/s13643-020-01344-3
Article info
Publication history
Accepted:
August 21,
2021
Received:
February 2,
2021
Footnotes
The authors have no conflicts of interest to disclose.
Identification
Copyright
Copyright © 2021 by Academic Pediatric Association
