Pediatricians and those who provide health care to children strive to provide the highest possible quality of care. Only recently, however, have there been opportunities for health care providers to measure more adequately the extent to which the optimal quality of care is being provided and to engage in quality improvement (QI) to address identified gaps.
1
, Agency for Healthcare Research and Quality. National Healthcare Disparities Report, 2012. National Healthcare Disparities Reports 2013; http://www.ahrq.gov/research/findings/nhqrdr/nhdr12/index.html. Accessed Aug. 5, 2013.
2
, 3
, Child and Adolescent Health Measurement Initiative. Data Resource Center for Child and Adolescent Health. Available at: http://www.childhealthdata.org/home. Accessed Sep. 18, 2013.
4
, Leatherman S, McCarthy D. Quality of Health Care for Children and Adolescents: A Chartbook. 2004; Available at: http://www.commonwealthfund.org/~/media/Files/Publications/Chartbook/2004/Apr/Quality%20of%20Care%20for%20Children%20and%20Adolescents%20%20A%20Chartbook/leatherman_pedchartbook_700%20pdf.pdf. Accessed Sep. 18, 2013.
5
Also contributing to this focus on QI are heightening demands for accountability and improvement from payers across the public and private spectrum.Secretary of the U.S. Department of Health and Human Services. 2012 Annual Report on the Quality of Care for Children in Medicaid and CHIP. 2012; Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/Downloads/2012-Ann-Sec-Rept.pdf. Accessed Sep. 18, 2013.
6
The certifying, accrediting, and academic communities have also increased expectations that physicians acquire theoretical and hands-on experience with quality improvement.7
This supplement is intended to make readers aware of key developments in QI policy, practice, education, and evaluation research. Our goal is to stimulate additional sharing of lessons learned, whether through research publications or other means, and to encourage health care providers and researchers to become full participants in the current national movement toward the triple aim of better care, better population health, and more affordable care.
8
The supplement is divided into 3 sections. The first section includes commentaries by leaders in pediatrics and quality improvement, including the President of the National Initiative for Children’s Healthcare Quality (NICHQ),
9
the President and Chief Quality Officer of the American Academy of Pediatrics (AAP),10
the Director of the Child and Adolescent Health Measurement Initiative (CAHMI),11
the President of AcademyHealth,12
leaders of quality improvement and research networks in ambulatory care,13
current and past Senior Vice Presidents for quality improvement at the American Board of Pediatrics,14
and a senior leader at the Patient-Centered Outcomes Research Institute (PCORI).15
The second section includes 4 articles concerning critical topics in quality improvement evaluation research methods.16
, 17
, 18
, 19
The concluding section presents examples of QI activities in a variety of settings, including hospitals,20
emergency medicine,21
NICUs,22
and several networks and states.23
, 24
, 25
This introduction provides a brief overview of and commentary on the contributions to the supplement focused on QI evaluation methods and on QI activities. Moreover, in 2014, 5 papers focused on education in QI will be published as an addendum to this supplement, with an overview led by John Co.26
QI Evaluation Research Methods
There is a clear and urgent need for more rigorous study of the extent to which QI activities, including those that are pediatric-focused, are effective, for which populations, and under which conditions.
27
, 28
To foster more rigorous study, the Agency for Healthcare Research and Quality has for the past 3 years helped support an annual conference focusing on pediatric QI.29
Four papers stemming from interactive educational sessions at the conferences are included in this supplement. Papers focused on other study designs taught at the Academic Pediatric Association meetings are readily available in the literature (eg, statistical process controlAcademic Pediatric Association. Quality Improvement Research Methods meetings. Quality Improvement SIG. Available at http://www.ambpeds.org/specialInterestGroups/sig_quality_improvement.cfm. Accessed Sep. 18, 2013.
30
, 31
and stepped wedge32
). In our view, these methodological papers taken together provide excellent guidance for evaluative research in pediatric quality improvement interventions.The focus of the article by Parry et al. concerning the term evaluation may cause discomfort for those who prefer to use the term research, while gratifying those who recognize the need for rigorous evaluation of QI interventions but avoid the term research owing to ambiguities about the protection of research in human subjects.
16
, 33
In our view, however, QI implementation research is a form of intervention researchU.S. Department of Health and Human Services. Quality Improvement Activities: FAQs. Available at http://answers.hhs.gov/ohrp/categories/1569. Accessed Aug. 30, 2013.
34
, 35
that is by definition evaluative. It aims to answer questions of whether, why, how, and for whom specific forms of QI work. Parry and colleagues argue for adoption of a formative, theory-driven, evaluation approach in pediatric health care QI that asks the question “How and in what contexts does the new model work or can it be amended to work?” The formative part of their recommendations reflects the fact that improvement initiatives are typically iterative in nature. Dealing with this staged approach to QI requires a formal framework such as that used by Kirkpatrick36
in medical education or similar stepwise approaches. A second major contribution of Parry and colleagues is their urgent call for the improvement and academic evaluation communities to extend and enhance the evaluative framework by establishing more standard taxonomies and refining specific methods to enhance the predictability of quality improvement interventions, a strategy that will improve the likelihood of publishing pediatric QI studies and seeing effective, context-sensitive QI interventions spread widely.Two reports address specific study designs that will enhance the internal validity of QI evaluation studies. Garrison and Mangione-Smith explicate the rationales behind, and the approaches to, cluster-randomized controlled trials (CRTs), the strongest designs for most health care QI intervention evaluation studies.
17
QI interventions are typically intended to change the behaviors of organizations or groups of actors (eg, health care policy entities, health care delivery systems, clinical practices, and microsystems). Thus, CRTs more closely approximate the real world of QI implementation and allow the rigors of randomization, blinding and potential avoidance of cross-contamination, and clustering of results at the patient and unit levels to enhance the value of QI interventions in terms of internal validity and publishability. The Garrison and Mangione-Smith paper provides numerous examples of situations in which CRTs have an advantage over randomized controlled trials, but it also notes the real methodological challenges of CRTs in pediatric QI, such as potentially decreased statistical power (due to clustering) and the need for at least 6 to 8 clusters for sufficient powering. In addition, the paper notes logistical considerations, including study participants’ concerns about the ethics of withholding a health system change intervention. Willingness of providers and other entities in health care to be randomized can be another challenge.Quasi-experimental designs provide an alternative to randomized trials, and Penfold, in this supplement, argues that the interrupted time series (ITS), when well done, is the best of the quasi-experimental study designs.
18
He enumerates the strengths of ITS, such as controlling for the effects of secular trends and other threats to internal validity and inference. No method is perfect, and ITS results are subject to a couple of noteworthy biases; Penfold also notes them.CRT and ITS designs provide substantial help in addressing potential threats to internal validity. However, addressing internal validity is not enough at this stage in the development of QI interventions and their evaluation. As Parry and colleagues imply and others have stated explicitly, “… heterogeneity is not always noise”; that is, understanding the impact of context on the results of QIs, or “context-sensitivity,” is essential. McDonald’s paper in this supplement addresses how to begin to ensure that context sensitivity is addressed in QI evaluation studies.
19
McDonald’s use of the realist evaluation model of Context, Mechanism, Outcome is helpful even as our understanding of context is evolving. Depending on the stage of QI intervention development (see Parry, this issue), aspects of context may become part of the mechanism that influences the QI outcome and may influence the transferability of any QI strategies found to be effective in one setting.McDonald carefully dissects some work she and others have done to attempt to understand context-sensitivity toward the eventual goal of “clear and detailed specification of each part of the QI intervention and its context in any evaluative project, as well as particulars about the ways that each influenced each other…” and discusses the recently developed Consolidated Framework for Implementation Research.
37
Developing theoretical perspectives on and measuring, and analyzing the effects of variations in context are not merely abstract concerns; McDonald specifies at least 3 main audiences that care increasingly about the context-sensitivity of QI interventions: practitioners (who need to know which QIs are suitable in their delivery settings), policymakers (who need to create incentives and infrastructures to encourage adoption and spread of effective QIs), and researchers (who aim to understand context-sensitivity to help practitioners and policymakers choose wisely among QIs).QI in Clinical Settings and Networks
One of the most striking phenomena in QI in the past decade has been the rapid development of multisite networks conducting large QI projects, as illustrated in the final section of this supplement.
20
, 21
, 22
, 23
, 24
, 25
Collectively, they provide a rich array of varied experiences in QI. There are many commonalities. Among the networks described in the first 4 articles,20
, 21
, 22
, 23
a significant factor for developing multisite, multidisciplinary networks is the relative rarity of the child health problems being addressed. The remaining reports24
, 25
deal with more common problems but in the context of working in primary care, a setting that may lack some of the supports available in academic medical centers. Implicit or explicit in all reports is the need for a robust infrastructure to support the QI activities, including major investments in data-collection systems, the necessary clinical and QI expertise, the effort of compiling the evidence for changes in practice, and the need for training and ongoing support or coaching of those in the field. A variety of strategies have been developed for the latter, including regular phone conferences, newsletters, webinars, and meetings. In addition, these reports indicate the significant role of allies in this effort, in terms of support from professional organizations, governmental agencies, and major health care providers such as hospitals. Among the organizations cited is the American Academy of Pediatrics, which has served as a convener and funder, as well as fostering the networks to meet maintenance of certification requirements. To develop and sustain this level of activity requires substantial resources, including financial support, but also the commitment and energy of the participants.These reports also raise some issues that need to be considered further. The first of these is the effect of this effort on improving care and, eventually, child health. Many networks are still too new for this assessment and, clearly, some have demonstrated significant success in addressing specific problems such as catheter-associated blood and other nosocomial infections.
23
As Shah et al22
indicate for the neonatal networks, which have a longer trajectory, not all QI efforts ameliorate the health problem being addressed. In view of the resources required, the reports raise the question of how to make greater efforts to understand the effectiveness of QI, as described above, and sustainability, a particularly important issue addressed by Shaw et al24
in terms of the ability of individual private practices to maintain QI activities. The significant effort in developing evidence reviews and comparisons of alternative approaches to problems, along with robust data systems and more standardized protocols, provides an excellent milieu for many types of research. However, there appears to be confusion in the field, as indicated in these reports, as to what are QI, health services or comparative-effectiveness research, and QI research. These distinctions have implications for both the conduct of QI activities that might not require institutional review board review, as well as the standards for publication of these activities, as noted by Simon et al.20
The conduct of health services and comparative effectiveness research is well established, but is different from QI research. As Shah et al22
and Shaw et al24
note, more work is needed before there is an understanding of the process of QI and the development of less laborious and more rapid methods for QI, as well as an understanding of what works for whom. This effort most certainly will require mixed-methods research to understand the effect of context on QI efforts19
and designs differing from the traditional decision analysis and comparative effectiveness paradigms.Conclusion
The fields of quality improvement and quality-improvement intervention research are in their infancy. Our hope is that the papers in this supplement are widely read and rapidly disseminated, discussed, and tested by our pediatric colleagues, and that the development and testing of rigorous but relevant study designs toward “generalized causal inference”
38
progresses rapidly.Acknowledgments
Supported by a Contract to the Academic Pediatric Association from the Agency for Healthcare Research and Quality, and a Grant from the American Board of Pediatrics Foundation . The views in this article do not necessarily reflect those of the U.S. Department of Health and Human Services or the Agency for Healthcare Research and Quality, a component of the U.S. Department of Health and Human Services.
References
Agency for Healthcare Research and Quality. National Healthcare Disparities Report, 2012. National Healthcare Disparities Reports 2013; http://www.ahrq.gov/research/findings/nhqrdr/nhdr12/index.html. Accessed Aug. 5, 2013.
- Annual report on health care for children and youth in the United States: trends in racial/ethnic, income, and insurance disparities over time, 2002-2009.Acad Pediatr. 2013; 13: 191-203
Child and Adolescent Health Measurement Initiative. Data Resource Center for Child and Adolescent Health. Available at: http://www.childhealthdata.org/home. Accessed Sep. 18, 2013.
Leatherman S, McCarthy D. Quality of Health Care for Children and Adolescents: A Chartbook. 2004; Available at: http://www.commonwealthfund.org/~/media/Files/Publications/Chartbook/2004/Apr/Quality%20of%20Care%20for%20Children%20and%20Adolescents%20%20A%20Chartbook/leatherman_pedchartbook_700%20pdf.pdf. Accessed Sep. 18, 2013.
Secretary of the U.S. Department of Health and Human Services. 2012 Annual Report on the Quality of Care for Children in Medicaid and CHIP. 2012; Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/Downloads/2012-Ann-Sec-Rept.pdf. Accessed Sep. 18, 2013.
- Improving timely childhood immunizations through pay for performance in Medicaid-managed care.Health Serv Res. 2010; 45: 1934-1947
- Maintenance of certification: the profession’s response to physician quality.Ann Fam Med. 2011; 9: 196-197
- The triple aim: care, health, and cost.Health Aff (Millwood). 2008; 27: 759-769
- A tall order: improve child health.Acad Pediatr. 2013; 13 (this issue)
- The American Academy of Pediatrics and Quality Improvement.Acad Pediatr. 2013; 13 (this issue)
- Engaging families (and ourselves) in quality improvement: an optimistic and developmental perspective.Acad Pediatr. 2013; 13 (this issue)
- The role of policy in quality improvement.Acad Pediatr. 2013; 13 (this issue)
- Practice based quality improvement/research networks: full speed forward.Acad Pediatr. 2013; 13 (this issue)
- Quality Improvement and Maintenance of Certification.Acad Pediatr. 2013; 13 (this issue)
- Disparities and quality: the next phase for high performing pediatric care.Acad Pediatr. 2013; 13 (this issue)
- Recommendations for evaluation of health care improvement initiatives.Acad Pediatr. 2013; 13 (this issue)
- Cluster randomized trials for health care quality improvement research.Acad Pediatr. 2013; 13 (this issue)
- Use of interrupted time series analysis in evaluating pediatric health care quality improvements.Acad Pediatr. 2013; 13 (this issue)
- Considering context in quality improvement interventions and implementation: concepts, frameworks, and application.Acad Pediatr. 2013; 13 (this issue)
- Quality improvement research in the inpatient setting: role of the pediatric research in inpatient settings (PRIS) network.Acad Pediatr. 2013; 13 (this issue)
- Quality improvement in pediatric emergency medicine.Acad Pediatr. 2013; 13 (this issue)
- Quality improvement initiatives in neonatal intensive care unit networks: achievements and challenges.Acad Pediatr. 2013; 13 (this issue)
- Pediatric collaborative networks for quality improvement and research.Acad Pediatr. 2013; 13 (this issue)
- The National Improvement Partnership Network: state-based partnerships that improve primary care quality.Acad Pediatr. 2013; 13 (this issue)
- Nine states' use of collaboratives to improve children's health care quality in Medicaid and CHIP.Acad Pediatr. 2013; 13 (this issue)
PLEASE INSERT REF TO THE JAN 2014? ISSUE SUPPLEMENT.
- Organising a manuscript reporting quality improvement or patient safety research.BMJ Qual Saf. 2013; 22: 777-785
- Strategies for addressing barriers to publishing pediatric quality improvement research.Pediatrics. 2011; 128: e678-e686
Academic Pediatric Association. Quality Improvement Research Methods meetings. Quality Improvement SIG. Available at http://www.ambpeds.org/specialInterestGroups/sig_quality_improvement.cfm. Accessed Sep. 18, 2013.
- Applying control charts to quality improvement.J Healthcare Qual. 1996; 18: 32-41
- How does lean work in emergency care? a case study of a lean-inspired intervention at the Astrid Lindgren Children’s Hospital, Stockholm, Sweden.BMC Health Serv Res. 2012; 12: 28
- Effectiveness of yearly, register based screening for chlamydia in the Netherlands: controlled trial with randomised stepped wedge implementation.BMJ. 2012; 345: e4316https://doi.org/10.1136/bmj.e316
U.S. Department of Health and Human Services. Quality Improvement Activities: FAQs. Available at http://answers.hhs.gov/ohrp/categories/1569. Accessed Aug. 30, 2013.
- What is population health intervention research?.Revue Canadienne de Santé Publique. 2009; 100: I8-I14
- Common definition for categories of clinical research: a prerequisite for a survey on regulatory requirements by the European Clinical Research Infrastructures Network (ECRIN).Trials. 2009; 10
- Evaluating Training Programs: The Four Levels.San Francisco: Berrett-Koehler. 1998;
- Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.Implement Sci. 2009; 7: 50
- Experimental and Quasi-Experimental Designs for Generalized Causal Inference.Houghton Mifflin, Boston2002
Article Info
Publication History
Accepted:
September 23,
2013
Received:
September 23,
2013
Identification
Copyright
Published by Elsevier Inc.
ScienceDirect
Access this article on ScienceDirectLinked Article
- ErratumAcademic PediatricsVol. 14Issue 4
- PreviewIn the Commentary by McCormick et al (“Quality Improvement in Pediatric Health Care: Introduction to the Supplement” Vol. 13 No. 6S, November/December 2013 Supplement), one of the author affiliations is listed incorrectly. The MGH Center for Child and Adolescent Health Research Policy should be “MGH Center for Child and Adolescent Health Research and Policy.”
- Full-Text
- Preview
