Advertisement

Developing Measures for Pediatric Quality: Methods and Experiences of the CHIPRA Pediatric Quality Measures Program Grantees

      Abstract

      Background

      Monitoring quality is an important way of understanding how the health care system is serving children and families. The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA) Pediatric Quality Measures Program (PQMP) funded efforts to develop and enhance measures to assess care for children and adolescents. We describe the processes used by the PQMP grantees to develop measures to assess the health care of children and adolescents in Medicaid and the Children's Health Insurance Program.

      Methods

      Key steps in the measures development process include identifying concepts, reviewing and synthesizing evidence, prioritizing concepts, defining how measures should be calculated, and measure testing. Stakeholder engagement throughout the process is critical. Case studies illustrate how PQMP grantees adapted the process to respond to the nature of measures they were charged to develop and overcome challenges encountered.

      Results

      PQMP grantees used varied approaches to measures development but faced common challenges, some specific to the field of pediatrics and some general to all quality measures. Major challenges included the limited evidence base, data systems difficult or unsuited for measures reporting, and conflicting stakeholder priorities.

      Conclusions

      As part of the PQMP, grantees were able to explore innovative methods to overcome measurement challenges, including new approaches to building the evidence base and stakeholder consensus, integration of alternative data sources, and implementation of new testing methods. As a result, the PQMP has developed new quality measures for pediatric care while also building an infrastructure, expertise, and enhanced methods for measures development that promise to provide more relevant and meaningful tools for improving the quality of children's health care.

      Keywords

      There is increasing demand for measures that can be used in quality improvement and accountability initiatives.
      • Panzer R.J.
      • Gitomer R.S.
      • Greene W.H.
      • et al.
      Increasing demands for quality measurement.
      The Affordable Care Act introduced delivery system reforms that use quality reporting for payment incentives. Additionally, the Children's Health Insurance Program Reauthorization Act (CHIPRA) called for states to voluntarily report on the quality of care for children in Medicaid and Children's Health Insurance Program (CHIP).

      Children’s Health Insurance Program Reauthorization Act of 2009, Pub. L 111-3, 123 Stat 8.

      Because of the paucity of meaningful quality measures for children's health care,
      • Kavanagh P.L.
      • Adams W.G.
      • Wang C.J.
      Quality indicators and quality assessment in child health.
      CHIPRA provided for historic investment in measures development in this area through establishment of the Pediatric Quality Measures Program (PQMP). Seven Centers of Excellence (CoEs) (Table 1) received funding to enhance existing measures or develop new ones for the Core Set of Children's Health Care Quality Measures for Medicaid/CHIP

      Centers for Medicare and Medicaid Services. CHIPRA initial core set of children’s health care quality measures. Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/CHIPRA-Initial-Core-Set-of-Childrens-Health-Care-Quality-Measures.html. Accessed March 20, 2014.

      and other public or private programs.

      Agency for Healthcare Research and Quality. CHIPRA Pediatric Quality Measures Program. Available at: http://www.ahrq.gov/policymakers/chipra/pqmpback.html. Accessed March 20, 2014.

      Broad measure domains and settings were designated in the CHIPRA legislation; within those domains, measure topics were assigned to the CoEs by the Centers for Medicare and Medicaid Services (CMS) and the Agency for Healthcare Research and Quality (AHRQ), with input from the CoEs and the public.
      • Agency for Healthcare Research and Quality
      Priority setting for the Children’s Health Insurance Program Reauthorization Act (CHIPRA) Pediatric Quality Measures Program: a notice by the Agency for Healthcare Research and Quality.
      Table 1AHRQ-CMS CHIPRA CoEs Under the Pediatric Quality Measures Program
      AbbreviationFull Name
      CAPQuaMMount Sinai Collaboration for Advancing Pediatric Quality Measures
      CEPQMCoE for Pediatric Quality Measurement at Boston Children's Hospital
      CHOP CoECoE at the Children's Hospital of Philadelphia/University of Pennsylvania
      COE4CCNCoE on Quality of Care Measures for Children with Complex Needs
      NCINQNational Collaborative for Innovation in Quality Measurement
      PMCoEPediatric Measurement CoE
      Q-METRICQuality Measurement, Evaluation, Testing, Review, and Implementation Consortium
      AHRQ-CMS CHIPRA indicates Agency for Healthcare Research and Quality–Centers for Medicare and Medicaid Services Children's Health Insurance Program Reauthorization Act of 2009; CoE, center of excellence.
      Here we describe the measures development process and the particular methods used by the PQMP CoEs. We describe criteria used for evaluating measures, outline steps in the development process, and present 2 case studies in detail and 5 additional case studies in the Online Appendix in order to highlight variations in methods and challenges. We discuss lessons learned to inform future measures work.

      Measures Development Process

      Quality measures in the health care context are tools that characterize a structure, process or outcome relevant to quality goals such as safety, effectiveness, efficiency, and patient-centeredness.

      Centers for Medicare and Medicaid Services. Quality measures. Available at: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/index.html?redirect=/QUALITYMEASURES/. Updated April 2013. Accessed October 2013.

      Measures serve purposes as varied as monitoring performance or describing trends, driving internal quality activities, or holding entities accountable. For the PQMP, CHIPRA called for measures for monitoring care of children enrolled in Medicaid/CHIP and in public and private health care organizations.

      Children’s Health Insurance Program Reauthorization Act of 2009, Pub. L 111-3, 123 Stat 8.

      This section delineates the general measures development process while describing the specific charges to the PQMP grantees.
      The measures development process begins by identifying and prioritizing opportunities for measurement and progresses through refining opportunities based on evidence review, definition, testing, and vetting of the measures with stakeholders (Figure). The process is designed to assess a measure's adherence to desirable attributes related to the importance, scientific soundness, feasibility, and usability of the measures (Table 2). For the PQMP, the CoEs convened a work group to determine what attributes constituted a good pediatric measure in the context of Medicaid/CHIP reporting.
      • Mistry K.B.
      • Chesley F.
      • LLanos K.
      • et al.
      Advancing Children's Health Care and Outcomes Through the Pediatric Quality Measures Program.
      For example, CHIPRA asked for measures at the state, plan, and provider levels that could identify disparities by racial/ethnic, socioeconomic, and special health care needs and that would be understandable to families and providers. Although the attributes chosen emphasize measure characteristics important to Medicaid/CHIP, they align with those used by other national evaluation programs, including the Blueprint for the CMS Measures Management System,

      Centers for Medicare and Medicaid Services. A blueprint for the CMS measures management system, v9.1. 2012. Available at: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/MeasuresManagementSystemBlueprint.html. Accessed September 2013.

      the National Committee for Quality Assurance's HEDIS measures,

      National Committee for Quality Assurance. HEDIS desirable attributes. Available at: http://www.ncqa.org/tabid/415/Default.aspx. Accessed October 19, 2013.

      and the National Quality Forum's Consensus Development Process.

      National Quality Forum. The consensus development process. Available at: http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx. Accessed October 14, 2013.

      Table 2
      AttributeDescription
      Step in the measures development process: Evidence/guideline review
       ImportanceThis criterion describes burden of the condition on children, families and society; fiscal burden; potential for quality improvement; and effect on a child’s development. The measure’s importance to Medicaid/CHIP as well as the existence of health care disparities also are considered.
       Grounded in scientific evidenceQuality measures seek to describe meaningful structure, processes or outcomes of care. The construct being measured should be grounded in clinical evidence.
      Step in the measures development process: Testing
       Scientific soundness of the measureTwo components comprise this criterion:

      Validity is the extent to which the measure represents the construct being evaluated.

      Reliability is the extent to which the measure results are reproducible when conditions remain the same.
       FeasibilityThe extent to which the data required are readily available, are retrievable from record systems without undue burden, and can be implemented for measurement.
       Identification of disparitiesThis criterion describes a measure’s ability to highlight differences in performance by race/ethnicity, socioeconomic status, special health care needs, and other characteristics important for quality improvement.
       Health Information TechnologyTo facilitate the move towards health information technology, the PQMP added a criterion that assesses a measure’s potential to be reported using electronic health record systems.
      Step in the measures development process: Public Comment, Focus Groups, Stakeholder Usability Interviews, etc.
       UnderstandabilityA quality measure is unlikely to change behavior unless it is clear to its users. Measures developed by the PQMP must be understandable to health care purchasers, families and health care providers.
       UsabilityCurrent use of measure.

      Testing to see if users understand the measure.

      Identification and Prioritization of Measure Concepts

      To identify potential measure concepts within topics, a measurement framework or clinical algorithm that reflects professional consensus about how a condition should best be managed often is created.
      • Lewandowski R.E.
      • Acri M.C.
      • Hoagwood K.E.
      • et al.
      Evidence for the management of adolescent depression.
      This algorithm can help to identify those steps that may have the greatest effects on health outcomes, or where unexplainable variation exists. It also is important at this stage to conduct an environmental scan of published literature or measure repositories to identify existing measures in a topic area. It can reveal where new measures are unnecessary or expose opportunities for building upon existing measures.
      Once measure concepts are chosen, it is useful to seek input from stakeholders to prioritize them. A participatory stakeholder process, in fact, is a required part of priority setting for measures development under CHIPRA.

      Children’s Health Insurance Program Reauthorization Act of 2009, Pub. L 111-3, 123 Stat 8.

      For measures in child health broadly and the PQMP specifically, stakeholders include children and families, pediatricians and other clinicians, state Medicaid/CHIP, federal policy makers, and public and private health plans as well as research and clinical experts in relevant fields.

      Evidence/Guideline Review

      The evidence/guideline review has 2 goals: first, it should demonstrate the importance of a measure topic area (in terms of number of individuals affected, gaps in care, disparities, financial impact on health systems); and second, it should establish the evidence or rationale for how a measure addresses a structure or process of care or an outcome that is influenced by a structure or process of care.
      Relevant clinical guidelines are useful starting places because they represent a consensus statement from key clinical leaders; however, attention should be given to the Institute of Medicine's recommended standards for guideline development, including use of systematic evidence reviews, grading of evidence, and input from conflict-free experts.
      • Committee on Pediatric Health and Health Care Quality Measures
      • Board on Children, Youth, and Families, Board on Health Care Services; Institute of Medicine and National Research Council of the National Academy
      Child and Adolescent Health and Health Care Quality: Measuring What Matters.
      Measure developers may need to conduct their own systematic reviews when such reviews and guidelines are lacking.

      Measure Specification and Testing

      Measure specifications are instructions for obtaining data and calculating rates. Primary components are the denominator, numerator, and exclusions. Precise specification is important to ensure measures are being implemented correctly and that results support reliable and valid comparisons. The scientific soundness, feasibility and precision of a measure are determined during testing. Testing also can reveal a measure's ability to identify health care disparities and whether it can be reported using health information technology. Testing is often conducted by applying measures in the settings in which they will be implemented or a suitably representative data source, such as the Medicaid Analytic extract.

      Centers for Medicare and Medicaid Services. Medicaid Analytic eXtract (MAX) general information. Available at: http://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Data-and-Systems/MedicaidDataSourcesGenInfo/MAXGeneralInformation.html. Accessed March 2014.

      The PQMP grantees are developing measures for numerous settings, including clinical practices, health plans, and state Medicaid/CHIP.

      Stakeholder Engagement Methods

      Techniques for explicitly engaging stakeholders include convening them on advisory panels, conducting stakeholder interviews or focus groups, and posting measures broadly for public comment. Incorporating input from multiple and diverse stakeholders throughout the process is critical to establishing a measure's importance, understandability, and usability.

      Case Studies

      Producing measures that achieve or balance the desirable attributes is complex. The following examples illustrate how PQMP grantees developed measures for 2 topics. The cases describe some challenges that affect the field of pediatrics in particular and others that encumber quality measures in general.

      Measures of Well-Care for Adolescents

      The National Collaborative for Innovation in Quality Measurement (NCINQ) was asked to develop measures assessing the content of well-care visits for adolescents.

      Evidence/Guideline Review

      NCINQ prepared summaries addressing the importance of and existing guidelines for 15 areas of well care for adolescents, ranging from sexually transmitted infection screenings

      Berlan ED, Ireland AM, Morton S, et al. Variations in measurement of sexual activity based on EHR definitions. Pediatrics. In press.

      to depression screening.
      • Lewandowski R.E.
      • Acri M.C.
      • Hoagwood K.E.
      • et al.
      Evidence for the management of adolescent depression.
      For some topics, the environmental scan revealed that measures existed, and the team did not pursue further work. Conflicts often existed between consensus-based, evidence-informed guidelines such as Bright Futures
      and the recommendations of the US Preventive Services Task Force (USPSTF), whose process relied on systematic evidence reviews

      US Preventive Services Task Force. Procedure manual. July 2008. AHRQ Publication 08-05118-EF. Available at: http://www.uspreventiveservicestaskforce.org/uspstf08/methods/procmanual.pdf. Accessed May 22, 2014.

      and often found insufficient evidence to recommend for or against services. NCINQ convened a multistakeholder advisory panel to prioritize topics for measures development and testing. The panel recommended 3 areas: chlamydia identification and follow-up, depression screening, and tobacco use and follow-up. At the time of development, only the first 2 screening topics were recommended by the USPSTF, although all were supported by consensus-based recommendations.

      Field Testing

      Because the proposed well care measures depend on detailed clinical data not available in administrative claims, the NCINQ testing approach focused on demonstrating the feasibility and scientific soundness of the measures using electronic health record (EHR) data. The team assessed availability and completeness of coded data, comparability of data extracted from electronic data versus manual review of the electronic record, and the logic for calculating the results. NCINQ's 2-stage approach involved surveying testing sites to assess the availability of structured fields and then limiting data collection to sites with structured fields. NCINQ's testing revealed an overall lack of agreement between computer- and manually extracted data, with computer-extracted data proving less informative for measure reporting across most concepts.

      Gardner W, Morton S, Byron SC, et al. Using computer-extracted data from electronic health records to measure the quality of adolescent well-care. Health Serv Res. In press.

      Although structured data fields existed, they were not routinely used. On the basis of this finding, NCINQ has incorporated interviews with clinical staff to understand clinical work flow and sample data reports as part of the vetting of potential test sites for subsequent measure testing in EHRs.

      Stakeholder Input and Finalization

      NCINQ worked with its multistakeholder advisory panels to weigh the results of testing and public comment. Of the 3 measure concepts tested, NCINQ submitted 2 measures for the PQMP: Sexual Activity Status Among Adolescents and Tobacco Use and Help With Quitting Among Adolescents. The latter was recommended for use in federal or other programs by the core set recommending body. NCINQ did not submit the Depression Screening Among Adolescents measure because of a similar measure in development; the team is focused on depression management and remission for this topic area.

      Measures of Care Coordination for Children With Complex Needs

      The Center of Excellence on Quality of Care Measures for Children With Complex Needs (COE4CCN) was charged with developing measures to assess care coordination for children with complex medical needs.

      Evidence Review, Development of Concepts, and Stakeholder Engagement

      COE4CCN developed a conceptual framework for care coordination for children with complex needs. The framework outlined the ideal state for care coordination: information is collected, synthesized, and shared by all individuals caring for the child, including family members, the primary care provider, subspecialists, care coordinators, and school nurses. The framework illuminated how care coordination relates to both short- and long-term outcomes, such as emergency department utilization and health-related quality of life.
      COE4CCN summarized the sparse evidence in the pediatric literature assessing links between care coordination and outcomes. To address the gaps in evidence, the team turned to literature addressing the frail elderly population, as they have many similarities to children with complex needs regarding need for well-coordinated care. From the review, the team developed 42 quality indicators that were supported by varying levels of evidence.
      COE4CCN convened a multistakeholder Delphi panel to prioritize concepts for specification. Panelists scored each indicator on validity, meaning the concept was supported by adequate levels of evidence and/or expert consensus, and feasibility, meaning the data needed to assess the concept were obtainable at reasonable cost and burden. The panelists then discussed all 42 indicators in person and could privately rescore indicators after discussion and seeing median scores. Thirty-four of the 42 indicators progressed to the next phase. The proposed data sources were parent/caregiver survey, medical records, and claims data.

      Specification and Testing

      COE4CCN sought to develop specifications that support reliable implementation. However, the team was unable to specify 3 of the medical records-based indicators because information needed was not present in sample charts reviewed. Overall, the specification process resulted in a 54-item Family Experiences with Coordination of Care survey, an electronic Web-based medical record abstraction tool, and 1 administrative measure.
      In testing, COE4CCN is seeking to assess the feasibility of care coordination measures, but the team has encountered challenges. To date, caregivers have completed the family experiences survey. The team will next abstract the children's medical records and access Medicaid data. Testing has involved meeting requirements from 4 institutional review boards. The team asked office-based providers to report on their own performance using standardized medical record abstractions, but participation has been low. To address this challenge, COE4CCN has moved to a model of central abstraction, which may be a more feasible yet more costly approach to data collection.

      Finalization

      Given the evidence limitations, COE4CCN will next conduct analyses to examine whether higher scores on care coordination process measures relate to better outcomes such as decreased hospitalizations, fewer missed school days, and better functional status. This step is particularly important when measures are based on low levels of evidence yet will be considered for accountability purposes.

      Discussion

      Despite their work in widely divergent content areas, these case studies followed similar steps in evidence review, measure specification, testing, and stakeholder engagement. These and other PQMP grantees demonstrated innovative approaches to address limits to the evidence base, overcome data challenges, and meaningfully engage stakeholders in developing the measures.

      Addressing Limitations in Evidence

      In many areas, we do not have sufficient evidence about effective clinical or organizational interventions to build quality measures for child health.
      Many existing guidelines for pediatric care are based primarily on expert consensus
      • Palmer R.H.
      • Miller M.R.
      Methodologic challenges in developing and implementing measures of quality for child health care.
      and therefore do not rise to the level of trustworthiness recommended by the Institute of Medicine. Features of the pediatric population that contribute to this problem, such as a developmental trajectory marked by rapid change, mean evidence will remain a challenge. Grantees used several approaches to address this problem. COE4CCN drew on evidence from a relevant adult population and is assessing whether their measures demonstrate a relationship to key outcomes. Mount Sinai Collaboration for Advancing Pediatric Quality Measures (CAPQuaM) developed an approach to gain stakeholder consensus amid conflicting evidence. As these examples illustrate, alternative approaches for using and applying the available evidence may be needed for different kinds of measures. The current national investment in comparative effectiveness research,

      Agency for Healthcare Research and Quality. Effective health care program. Available at: http://effectivehealthcare.ahrq.gov/index.cfm/what-is-comparative-effectiveness-research1/. Accessed December 18, 2013.

      such as the recently funded National Pediatric Learning Health System,

      Patient-Centered Outcomes Research Institute. A national pediatric learning health system (PEDSNet). Available at: http://pfaawards.pcori.org/node/20/datavizwiz/detail/1556. Accessed May 29, 2014.

      promise to support and enhance measures development. In particular, such efforts could be designed to consider questions such as timing and periodicity of follow-up and alternative thresholds for designating high- or low-quality care.

      Addressing Limitations in Data Sources

      One aim of the PQMP is to stimulate use of new health information technologies. The hope is to move beyond measures that rely on administrative data, which are feasible to collect yet lack the clinically rich information required to describe content of care and other important aspects of child health. Like NCINQ, several grantees designed measures for use in EHRs but found currently available EHRs often contain nonextractable information. The limitations of computer-extracted data derived both from technical challenges (eg, the lack of structured fields) as well as from the lack of existing work flows for implementing and documenting quality processes.
      One solution to data limitations is a more prospective approach to designing and testing quality measures. Instead of testing measures using existing data sources, measures could be developed in concert with quality improvement efforts that include effective clinical work flows and patient-data storage systems. Ideally, such an approach would result in measures that clinical teams view as meaningful for care (thus increasing adoption and completion of required data). Efforts such as certifying EHR technology to ensure systems have the necessary technological functionality to meet meaningful use criteria

      Centers for Medicare and Medicaid Services. Certified EHR technology. Available at: http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Certification.html. Updated October 17, 2013. Accessed May 21, 2014.

      will be helpful in this regard and could greatly improve information available to consumers, purchasers, and policymakers.

      Engaging Diverse Stakeholders

      Grantees benefited from stakeholder engagement in several ways. Given a large number of measure opportunities, the Quality Measurement, Evaluation, Testing, Review, and Implementation Consortium worked with stakeholders to prioritize measures that balanced implementation burden against likely impact on quality outcomes for children with sickle cell disease. Stakeholders helped to evaluate importance in the face of differing levels of evidence, as illustrated in NCINQ and CAPQuaM examples. Stakeholders suggested new measure concepts, such as the CoE at the Children's Hospital of Philadelphia/University of Pennsylvania Newborn Duration measure. When developing a survey of family experiences with pediatric hospital care, the CoE for Pediatric Quality Measurement at Boston Children's Hospital found that engaging youth and families in survey content, item wording, and labeling of information greatly enhanced the survey. Finally, Pediatric Measurement CoE's establishment of a relationship with the American Board of Pediatrics Maintenance of Certification function facilitated rapid adoption of measures for attention-deficit hyperactivity disorder into the Board's Improvement Module.

      Conclusions

      As illustrated, the PQMP grantees followed a common approach to measures development but varied it to meet the challenges of creating pediatric measures in complex medical domains. On the basis of this experience, we have several recommendations for future measures development. First, we need continued investment in research to generate the empirical evidence required for measures development. We are seeing some movement in this area: spurred by national incentive programs, broader adoption of EHRs is making quality measurement more feasible, particularly in areas of patient- and family-centeredness and person-reported outcomes.
      • Estabrooks P.A.
      • Boyle M.
      • Emmons K.M.
      • et al.
      Harmonized patient-reported data elements in the electronic health record;supporting meaningful use by primary care action on health behaviors and key psychosocial factors.
      In addition, engaging stakeholders at every stage of measures development is a critical step in overcoming challenges related to priority setting, evidence, implementation, and adoption. Ways to meaningfully involve diverse stakeholders, particularly families and other end-users of measures, should be formally incorporated in all measures development activities.
      The PQMP has developed an infrastructure to advance pediatric quality measurement across many topics. Grantees are testing newly developed measures to connect processes to outcomes of care. Grantees are developing methods to utilize new data sources. And grantees are forging partnerships with important stakeholders, including states, providers, and families. The bi-directional measures development process has served to educate both grantees and stakeholders themselves. This program will result in new or improved measures in over 30 key areas, including perinatal care, inpatient care, mental health, oral health, and health outcomes. Further, measures will assess care for special populations, including medically complex children and those in the foster care system. This foundation should contribute to important advances in quality measurement as well as improvement in the health of children.

      Acknowledgments

      Supported in part by grants U18HS20518 (CAPQuaM), U18HS20513 (CEQM), U18HS20508 (CHOP CoE), U18HS20506 (COE4CCN), U18HS20503 (NCINQ), U18HS20498 (PMCoE), and U18HS20516 (Q-METRIC) from the AHRQ . The content is solely the responsibility of the authors and does not necessarily represent the official views of the AHRQ.

      Supplementary Data

      References

        • Panzer R.J.
        • Gitomer R.S.
        • Greene W.H.
        • et al.
        Increasing demands for quality measurement.
        JAMA. 2013; 310: 1971-1980
      1. Children’s Health Insurance Program Reauthorization Act of 2009, Pub. L 111-3, 123 Stat 8.

        • Kavanagh P.L.
        • Adams W.G.
        • Wang C.J.
        Quality indicators and quality assessment in child health.
        Arch Dis Child. 2009; 94: 458-463
      2. Centers for Medicare and Medicaid Services. CHIPRA initial core set of children’s health care quality measures. Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/CHIPRA-Initial-Core-Set-of-Childrens-Health-Care-Quality-Measures.html. Accessed March 20, 2014.

      3. Agency for Healthcare Research and Quality. CHIPRA Pediatric Quality Measures Program. Available at: http://www.ahrq.gov/policymakers/chipra/pqmpback.html. Accessed March 20, 2014.

        • Agency for Healthcare Research and Quality
        Priority setting for the Children’s Health Insurance Program Reauthorization Act (CHIPRA) Pediatric Quality Measures Program: a notice by the Agency for Healthcare Research and Quality.
        Fed Reg. 2010; 75: 75469-75471
      4. Centers for Medicare and Medicaid Services. Quality measures. Available at: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/QualityMeasures/index.html?redirect=/QUALITYMEASURES/. Updated April 2013. Accessed October 2013.

        • Mistry K.B.
        • Chesley F.
        • LLanos K.
        • et al.
        Advancing Children's Health Care and Outcomes Through the Pediatric Quality Measures Program.
        Acad Pediatr. 2014; 14: S19-S26
      5. Centers for Medicare and Medicaid Services. A blueprint for the CMS measures management system, v9.1. 2012. Available at: http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/MMS/MeasuresManagementSystemBlueprint.html. Accessed September 2013.

      6. National Committee for Quality Assurance. HEDIS desirable attributes. Available at: http://www.ncqa.org/tabid/415/Default.aspx. Accessed October 19, 2013.

      7. National Quality Forum. The consensus development process. Available at: http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx. Accessed October 14, 2013.

        • Lewandowski R.E.
        • Acri M.C.
        • Hoagwood K.E.
        • et al.
        Evidence for the management of adolescent depression.
        Pediatrics. 2013; 132: e996-e1009
        • Committee on Pediatric Health and Health Care Quality Measures
        • Board on Children, Youth, and Families, Board on Health Care Services; Institute of Medicine and National Research Council of the National Academy
        Child and Adolescent Health and Health Care Quality: Measuring What Matters.
        National Academies Press, Washington, DC2011
      8. Eden J. Wheatley B. McNeil B. Sox H. Committee on Reviewing Evidence to Identify Highly Effective Clinical Services and Board, Health Care Services; Institute of Medicine; National Research Council of the National Academy. Knowing What Works in Health Care: A Roadmap for the Nation. National Academies Press, Washington, DC2008
      9. Eden J. Levit L. Berg A. Morton S. Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Board on Health Care Services; Institute of Medicine; National Research Council of the National Academy. Finding What Works in Health Care: Standards for Systematic Reviews. National Academies Press, Washington, DC2011
      10. Centers for Medicare and Medicaid Services. Medicaid Analytic eXtract (MAX) general information. Available at: http://www.cms.gov/Research-Statistics-Data-and-Systems/Computer-Data-and-Systems/MedicaidDataSourcesGenInfo/MAXGeneralInformation.html. Accessed March 2014.

      11. Berlan ED, Ireland AM, Morton S, et al. Variations in measurement of sexual activity based on EHR definitions. Pediatrics. In press.

      12. Hagan J.F. Shaw J.S. Duncan P.M. Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents. 3rd ed. American Academy of Pediatrics, Elk Grove Village, Ill2008
      13. US Preventive Services Task Force. Procedure manual. July 2008. AHRQ Publication 08-05118-EF. Available at: http://www.uspreventiveservicestaskforce.org/uspstf08/methods/procmanual.pdf. Accessed May 22, 2014.

      14. Gardner W, Morton S, Byron SC, et al. Using computer-extracted data from electronic health records to measure the quality of adolescent well-care. Health Serv Res. In press.

        • Palmer R.H.
        • Miller M.R.
        Methodologic challenges in developing and implementing measures of quality for child health care.
        Ambul Pediatr. 2001; 1: 39-52
      15. Agency for Healthcare Research and Quality. Effective health care program. Available at: http://effectivehealthcare.ahrq.gov/index.cfm/what-is-comparative-effectiveness-research1/. Accessed December 18, 2013.

      16. Patient-Centered Outcomes Research Institute. A national pediatric learning health system (PEDSNet). Available at: http://pfaawards.pcori.org/node/20/datavizwiz/detail/1556. Accessed May 29, 2014.

      17. Centers for Medicare and Medicaid Services. Certified EHR technology. Available at: http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Certification.html. Updated October 17, 2013. Accessed May 21, 2014.

        • Estabrooks P.A.
        • Boyle M.
        • Emmons K.M.
        • et al.
        Harmonized patient-reported data elements in the electronic health record;supporting meaningful use by primary care action on health behaviors and key psychosocial factors.
        J Am Med Inform Assoc. 2012; 19: 575-582