If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Pediatric Quality Measures Program (PQMP) was established in response to the Children's Health Insurance Program Reauthorization Act of 2009, aiming to measure and improve health care quality and outcomes for the nation's children. This brief report describes the PQMP 2.0 and its components. PQMP 2.0 established a priori research questions (Research Foci) and endeavored to assess usability and feasibility of measures through measure implementation and quality improvement initiatives. The Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare and Medicaid Services (CMS) awarded 6 grants to Centers of Excellence (COEs), and a contract to facilitate collaboration and learning across the COEs. The 6 COEs partnered with stakeholders from multiple levels (eg, state, health plan, hospital, provider, family) to field test real-world implementation and refinement of pediatric quality measures and quality improvement initiatives. The PQMP Learning Collaborative (PQMP-LC) consisted of AHRQ, CMS, the 6 COEs, and L&M Policy Research, LLC. The PQMP-LC completed literature reviews, key informant interviews, and data collection to develop reports to address the Research Foci; aided with development of measure implementation and quality improvement toolkits; conceptualized an implementation science framework, analysis, and roadmap; and facilitated dissemination of learnings and products. The various products are intended to support the uptake of PQMP measures and inform future pediatric measurement and improvement work.
The second phase of the Pediatric Quality Measures Program engaged six Centers and multi-level stakeholders to test pediatric quality measures in real-world settings to assess usability and feasibility. Additionally, the PQMP Learning Collaborative allowed for the dissemination of generalizable findings.
The History of PQMP
The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA) presented an unprecedented opportunity to measure and improve health care quality and outcomes for the nation's children, particularly those enrolled in Medicaid and the Children's Health Insurance Program (CHIP). Together, Medicaid and CHIP serve more than 45.1 million children annually, representing approximately 1 in 2 children in the United States.
Medicaid and CHIP play a key role in ensuring that low-income children receive high-quality health care and coverage.
When CHIPRA was enacted more than a decade ago, the Agency for Healthcare Research and Quality (AHRQ) and the Centers for Medicare & Medicaid Services (CMS) began working together to implement selected provisions of the legislation related to children's health care quality. As part of that collaboration and in keeping with the overall Department of Health and Human Services’ National Strategy for Quality Improvement in Health Care strategy, the AHRQ Pediatric Quality Measures Program (PQMP) was established. The goal was to increase the portfolio of evidence-based, consensus-built, pediatric quality measures available to public and private purchasers of children's health care services and to continue to improve and strengthen the Core Set of Children's Health Care Quality Measures for Medicaid and CHIP (Child Core Set).
The initial phase “PQMP 1.0,” initiated in 2011, aimed to improve children's quality of care through the development of new pediatric quality measures. Seven PQMP Centers of Excellence, funded through cooperative agreement grants, successfully developed measures, many of which were endorsed by the National Quality Forum, across a variety of domains (eg, perinatal care, child clinical preventive services, management of acute conditions, management of chronic conditions, patient-reported outcomes, availability of services).
Beginning in 2016, the second phase “PQMP 2.0,” focused on disseminating and implementing the measures developed by the Centers of Excellence (COE) and those existing in the Child Core Set. The PQMP 2.0 prioritized assessing the feasibility and usability of the pediatric quality measures at various levels (state, health plan, and provider) in real-world settings to further strengthen the connection between measurement and improvement. This Brief Report describes the PQMP 2.0 and its components, providing additional detail on how the PQMP 2.0 aimed to improve the evidence-base to support performance monitoring and quality improvement (QI) efforts for children in Medicaid and CHIP.
PQMP 2.0 Goals
PQMP 2.0 was designed to support and strengthen pediatric quality measurement and QI based on learnings from QI demonstration projects that implemented and evaluated the use of measures in a variety of real-world settings. PQMP 2.0 focused on assessing the feasibility and usability of PQMP or existing Child Core Set measures within the Medicaid and CHIP patient populations at the state, health plan, hospital, and provider levels. These activities were intended to support performance monitoring and QI through: 1) field testing to inform refinement, data collection, and reporting of measures; and 2) use of performance data on measures to define QI goals and test multilevel improvement strategies.
Notably, the PQMP 2.0, through the QI demonstration projects, sought to answer a set of important questions, or Research Foci (RF). These questions are shown in Table 1. The RF were established by AHRQ and CMS as part of the broader goal to build the evidence-base for quality measurement and improvement in pediatrics. The RF were developed to build upon lessons learned in PQMP 1.0 and advance the evidence base regarding implementation of measures, including contextual factors in real world settings that can hinder or facilitate uptake of measures and QI. The foci outlined key research areas in QI where more evidence was needed to facilitate progress.
Table 1PQMP 2.0 Research Foci
Using quality measures at multiple levels
How can the same measure be used to evaluate quality of care between multiple levels (ie, state, health plan, and provider levels) to ascertain how improvement at one level drives overall improvement at the state level?
What are the appropriate uses for each measure and each level of measurement, given a measure's “intended use” by developer/steward? What are the different standards and criteria that should be applied to the development and use of measures used for payment versus QI? How do we determine measures can be appropriately used/aggregated at multiple levels (state, health plan, and provider levels) and be “folded up or down”?
Making performance comparisons
How might relative performance be compared at each level, such as between different provider groups/organizations (eg, Federally Qualified Health Centers, pediatric group practice, multi-specialty group practice) or between different Accountable Care Organizations/managed care health plans?
What are evidence-based and scientifically sound methods for benchmarking progress on these measures? What level of improvement can be expected for measures—and in what time frame—taking into account different QI approach(es) undertaken at different levels? For example, if improvement is likely to be more rapid at the provider-level, what are the implications for performance targets established at the state-level?
Identifying measurement challenges/successes
What measurement (eg, data collection, reporting, QI) challenges and successes are identified at different levels (eg, state, health plan, practice, provider, patient levels)?
Assessing intermediate progress
For those measures for which improvement is unlikely to be seen within the CMS annual reporting cycle (calendar year), how might “intermediate” progress be measured at other levels which would predict improvement at the state-level, with a high predictive value?
QI indicates quality improvement; CMS, Centers for Medicare and Medicaid Services.
The PQMP 2.0 COEs participated in a Learning Collaborative (PQMP-LC) led by AHRQ in collaboration with CMS and L&M Policy Research from 2019 to 2021. The PQMP-LC created a partnership within and between the COEs, focused on improving understanding of best practices for dissemination and implementation of pediatric quality measures. The PQMP-LC aimed to build capacity and sustainability for performance monitoring and QI efforts focused on the Medicaid and CHIP patient populations at the state, health plan, hospital, and provider levels. The PQMP-LC also set out to identify generalizable lessons learned through the implementation of the selected pediatric quality measures by looking across the COEs’ experiences with regard to contextual factors that served as both barriers and facilitators to uptake and use of the measures.
Six COEs were selected to participate in PQMP 2.0. Table 2 lists each of the COEs’ principal investigator and institution. The components of the PQMP 2.0 are described in this paper and depicted in Figure.
Table 2PQMP 2.0 Centers of Excellence (COEs) and Projects
COE Institution/Project Title
Quality Measure Domain(s)
Michael D. Cabana
Children's Hospital at Montefiore/IMPLEmenting MEasures NeTwork (IMPLEMENT) for Child Health Network
Emergency room use for asthma
Sickle cell treatment
Gary L. Freed
University of Michigan/Quality Measurement, Evaluation, Testing, Review, and Implementation Consortium (Q-METRIC)
Emergency room use for asthma
Overuse of imaging
Sickle cell treatment
Kaiser Permanente Washington/Pediatric Hospital Care Improvement Project (P-HIP)
Mental health care in hospital settings
Transitions of care in hospital settings
Sarah H. Scholle
National Committee for Quality Assurance/National Collaborative for Innovation in Quality Measurement: Implementing and Improving (NCINQ II)
Depression in children and adolescents
Safe and judicious use of anti-psychotic medications in children and adolescents
Mark A. Schuster/Sara Toomey
Boston Children's Hospital/Center of Excellence for Pediatric Quality Measurement (CEPQM)
Consumer Assessment of Healthcare Providers and Systems Child Hospital Survey (Child HCAHPS)
Elizabeth A. Shenkman
University of Florida/Child Health Quality (CHeQ) Partnership
Linkage between dental prevention and treatment
Safe and judicious use of anti-psychotic medications in children and adolescents
Importantly, the PQMP-LC provided a forum for cross-collaboration among the COEs, feedback from experts and stakeholders in pediatric quality care measurement, and sharing of learnings and evidence. Throughout the PQMP 2.0, stakeholders, specifically end users, were involved in testing both the quality measures and QI strategies. The COEs were required to involve a state Medicaid or CHIP agency, External Quality Review Organizations (when applicable), and at least one provider organization that served Medicaid and CHIP beneficiaries. In addition, the COEs engaged other key stakeholders such as, health plans, hospitals, patients, and families or caregivers in their activities.
Field-Testing and Refining Measures
The COEs each field-tested and refined their quality measures to bolster understanding regarding appropriate use of a measure across multiple levels (one of the key research questions of the PQMP) including at the state, health plan, hospital, and the provider level (ie, health system, hospital, clinic). Their activities focused on assessing the performance of the measures to determine their feasibility and usability across levels and also how measures would fare at levels for which they were not originally developed.
Specific activities to test and refine the measures varied across the COEs; examples from Quality Measurement, Evaluation, Testing, Review, and Implementation Consortium, IMPLEmenting MEasures NeTwork, and Child Health Quality are included here. The Quality Measurement, Evaluation, Testing, Review, and Implementation Consortium team's activities focused on issues they identified related to data collection and reporting. In order to use a state-level measure for measuring health plan performance, the team developed a model to attribute Medicaid-enrolled patients at the state level to the health plan level. They also developed a chart abstraction tool to both ease and make data collection efforts more uniform for one of their measures. The IMPLEmenting MEasures NeTwork team assessed changes to variable construction and measure reporting due to the move from International Classification of Diseases (ICD)-9 to ICD-10 codes during their grant timeline. Additionally, this team analyzed provider-level variation in performance on 2 related PQMP asthma measures to assess usability at the provider level. The Child Health Quality team refined data collection methods by linking electronic health record data and claims data from 2 different states. Working with partner health plans, they also tested the impact of using varied measure specifications and coding practices on measure results. These activities helped the COEs refine the quality measures prior to implementing the measures and pursuing QI efforts.
Testing Quality Improvement Strategies
Each of the COEs completed QI demonstration projects aimed at improving performance for their quality measure(s) in partnership with stakeholders at multiple levels. Most of the COEs established QI Learning Collaboratives to encourage organizations to work together and learn from each other as they implemented the QI initiatives or interventions.
The activities and strategies to engage participants differed across the various Learning Collaboratives; examples from Center of Excellence for Pediatric Quality Measurement (CEPQM), National Collaborative for Innovation in Quality Measurement (NCINQ), and Pediatric Hospital Care Improvement Project (P-HIP) are provided here. CEPQM analyzed and identified drivers of improvement at participating sites and subsequently created primary and secondary driver diagrams used to conceptualize the improvement activities at each of the sites. CEPQM also worked with the participating sites to implement different improvement strategies during the grant timeline, such as adding a tool to the discharge process to test whether the use of the tool impacted hospital revisit rates and family experiences in care. Another COE, NCINQ, held 3 individual monthly calls with each of the Learning Collaborative participants to support their implementation efforts early on in the project, offered coaching calls with all the participants to support shared learning, and provided in person training. The NCINQ team synthesized learnings and informed the development of a change package created by the COE. P-HIP created a virtual Learning Collaborative for their participating sites, which included hosting a series of topic focused group calls, where sites could share experiences, lessons, and resources; the COE also provided one-on-one site level coaching as part of their Learning Collaborative. P-HIP also created a change package that includes lesson learned and best practices collected from the sites.
Research Foci, Implementation Science, and Dissemination
The PQMP-LC was established to ensure coordination, sharing of learning and best practices and cross-COE synthesis to identify generalizable findings that could be disseminated to promote use of pediatric measures. The PQMP-LC key activities focused on addressing the RF, supporting quality measurement and improvement, grounding the PQMP work in implementation science principles, and supporting the dissemination of COEs’ findings to support measure uptake and improvement efforts.
The COEs projects aimed to address the RF through the measure implementation process; the goal of the PQMP-LC was to synthesize and augment the COEs’ findings. Some of this was accomplished through literature reviews, key informant interviews, and synthesis of findings from the COEs’ demonstration projects to create an in-depth blend of existing evidence and on-the-ground experience. Published literature proved to be somewhat limited, increasing the value of the lessons conveyed through real-world involvement and understanding (gathered through key informant interviews described in more detail in this supplement [Reference will be included in the supplement]). The PQMP-LC is developing lessons learned related to the RF through the COEs’ efforts, including data collection, tool development, empirical analyses, and stakeholder engagement so that others involved in measure refinement and implementation efforts may benefit.
Toolkit Development and Dissemination
The PQMP-LC collaborated to identify key elements for inclusion in measure toolkits designed to facilitate measure uptake and QI initiatives by providing measure-specific resources, such as measure specifications, driver diagrams, strategic roadmaps, screening and communication flow diagrams, and improvement data.
After finalization of toolkit elements, the COEs developed draft toolkits based on their experiences with measure implementation and QI projects. The PQMP-LC provided a venue for the COEs to review and provide feedback on one another's’ toolkits. Another valuable resource facilitated by the PQMP-LC included solicitation of expert feedback regarding the appropriateness, usefulness, and alignment of measure toolkit materials and promotion of measure uptake from the Medicaid Medical Directors Network as well as experts at CMS. L&M provided technical assistance in the toolkit development, packaging the toolkits and creating language to guide users accessing the toolkits online.
Implementation Science Framework, Analysis, and Roadmap
To provide an introduction and overarching guide to the toolkits, the PQMP-LC developed an implementation roadmap to be posted to a webpage to support broader dissemination and uptake of the toolkits. To accomplish this, the PQMP-LC worked closely with 2 subject matter experts to identify an implementation science framework suitable for analyzing contextual factors across the COE projects. Also working closely with the subject matter experts and based on the selected framework, the PQMP-LC drafted an initial interview guide to gather information about contextual factors that influence pediatric quality measure implementation and QI efforts from the COEs. A follow-up worksheet was also created and completed by the COEs. The findings from the PQMP-LC's data analysis are discussed elsewhere in this supplement.
Central to the CHIPRA legislation and goal of the PQMP-LC is the public dissemination of key scientific findings and lessons learned concerning measure feasibility and usability to a broad stakeholder audience.
End users were involved throughout PQMP 2.0. The Medicaid Medical Directors Network provided feedback about the RF and toolkits. As part of the development of this journal supplement, the PQMP-LC again prioritized input from multiple stakeholders to provide a broad perspective. An additional avenue for dissemination included updating and enhancing the information available about PQMP activities and products on the AHRQ website https://www.ahrq.gov/pqmp/index.html. Importantly, a public webinar was organized—featuring experts from federal and state government as well as provider organizations—on the future of quality measurement and improvement in pediatrics, with a focus on lessons learned from recent quality initiatives and the shifting dynamics of pediatric care and research.
The PQMP 2.0 allowed for a real-world examination of PQMP and CMS Child Core Set measures across broad settings to drive both performance measurement but also QI. Implementation of these measures required a clear understanding of the context in which users were operating and how contextual factors can interact to serve as both barriers and facilitators to improving quality of care.
The challenges faced by the COEs with their respective stakeholder groups or collaborative in implementation of quality measurement and improvement served as perhaps the most important learnings. The COEs were able to highlight, and in some cases overcome, challenges that were identified to impact both measurement and use of measures to drive improvement across multiple levels.
One of the key challenges the COEs faced that affected quality measurement and QI was access to data. In order for the data to be useful, it needs to be accurate, complete, and must be collected regularly to assess intermediate progress and to evaluate the success of QI efforts. In some cases, data are only collected annually, or may not be collected reliably. Policy, infrastructure, and other issues were sometimes barriers to obtaining and linking data from multiple sources.
The complexity of measure specifications also presents a challenge. For quality measures to be comparable, they need to be able to be measured in a standardized way; however, there is variation in the types of data various entities collect. Additionally, when International Classification of Diseases codes change, measure specifications need to be updated. With respect to measure implementation, another challenge with comparison is ensuring that comparisons are fair—deciding whether risk adjustment is appropriate, if it can be done without masking disparities, and determining appropriate benchmarking strategies. There were also challenges pertaining to the use of measures across different levels, including patient attribution challenges and diminishing sample size when trying to use measures at lower levels.
The lessons learned, disseminated through this supplement, the toolkits, and findings from the demonstration projects, not only support the uptake of the subset of measures tested in PQMP 2.0 but also inform future pediatric measurement and improvement work more broadly. The long-term vision for the PQMP is that these and other evidence-based pediatric measures are used more broadly to support the delivery of high-quality health care to children, and these PQMP findings can further evidence-based quality measurement and improvement in pediatrics.
We must continue to select and prioritize measures that will drive meaningful changes in health outcomes for children and also address disparities in care. PQMP continues to work towards strengthening the Child Core Set and producing evidence-based measures for health plans and health systems by identifying gaps and continuing work on development and implementation of valid measures and tools that are aligned with stakeholders’ needs. However, equally important to reaching the vision of PQMP is ensuring a pipeline for collaborative innovation with multilevel stakeholders to test new methods and measures that continue to drive improvements in care. This will foster learning opportunities and facilitate national dissemination of lessons learned and best practices to support state Medicaid and CHIP programs in improving pediatric quality of care.
Financial statement: The Pediatric Quality Measures Program 2.0 was funded by AHRQ and CMS .
This article is published as part of a supplement sponsored by the US Department of Health and Human Services, the Centers for Medicare and Medicaid Services, and the Agency for Healthcare Research and Quality.
Disclaimer: The views expressed in this article are those of the authors, and no official endorsement by the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare and Medicaid Services (CMS), or the Department of Health and Human Services (DHHS) is intended or should be inferred.
The authors have no conflict of interest to disclose.