Advertisement

Young and Reckless? Greater Standardization and Transparency of Performance Is Needed for Pediatric Performance Measures

  • John Matthew Austin
    Correspondence
    Address correspondence to J. Matthew Austin, PhD, Department of Anesthesiology and Critical Care Medicine, School of Medicine, Johns Hopkins University, 750 E Pratt St, 15th Floor, Baltimore, MD 21202.
    Affiliations
    Department of Anesthesiology and Critical Care Medicine, Johns Hopkins University, Baltimore, Md

    Armstrong Institute for Patient Safety and Quality, Johns Hopkins Medicine, Baltimore, Md
    Search for articles by this author
  • Marlene R. Miller
    Affiliations
    Department of Pediatrics, School of Medicine, Johns Hopkins University, Baltimore, Md

    Department of Health Policy and Management, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Md
    Search for articles by this author
  • Peter J. Pronovost
    Affiliations
    Department of Anesthesiology and Critical Care Medicine, Johns Hopkins University, Baltimore, Md

    School of Nursing, Johns Hopkins University, Baltimore, Md

    Department of Health Policy and Management, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Md
    Search for articles by this author
      When the Children's Health Insurance Program Reauthorization Act (CHIPRA) was passed in 2009, the secretary of the Department of Health and Human Services was tasked with identifying an initial core set of measures that state Medicaid programs and the Children's Health Insurance Program (CHIP) could voluntarily use to reflect the quality of health care being delivered to children enrolled in these 2 programs. This effort was overdue for this joint federal–state program, as each state had developed its own quality measures, definitions, and performance thresholds, limiting the ability to compare Medicaid programs among states.
      To identify a core set of measures, the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare and Medicaid Services (CMS), and the 2009 Subcommittee of the AHRQ National Advisory Council on Healthcare Research and Quality (SNAC) partnered to establish criteria for assessing and recommending measures for the initial measure set. The SNAC searched existing measures for ones that reflected desirable attributes, such as importance, research evidence, feasibility, and understandability. They recommended 25 measures for the initial measure set encompassing both physical and mental health, including chronic conditions such as asthma and diabetes.

      Centers for Medicare and Medicaid Services. 2013 core set of children’s health care quality measures for Medicaid and CHIP. Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/Downloads/ChildCoreMeasures.pdf. Accessed December 7, 2013.

      The CHIPRA legislation also directed the Secretary of Health and Human Services to establish a CHIPRA Pediatric Quality Measures Program (PQMP). Under this program, measures developed could replace or be added to the initial core set, effective January 1, 2013, and annually thereafter. The developed measures are designed to be evidence based, identify disparities in care, report data in a standard format, and be responsive to multiple domains of health care quality.
      • Agency for Healthcare Research and Quality
      Recommendations to Improve Children’s Health Care Quality Measures: Background Report on the 2012 Process.

      Challenges

      The SNAC has been helpful by providing a standard set of measures to assess the quality of care provided to children enrolled in state Medicaid and CHIP programs. Nevertheless, use of the standard set is voluntary, and the number publicly reported varies by state. Of the 22 measures in the federal fiscal year (FFY) 2012 measure set, only 2 were reported by all 50 states and the District of Columbia. For FFY 2012, the median number of measures reported by states was 14, and only 2 states reported on all 22 core measures.
      • Sebelius K.
      The Department of Health and Human Services 2013 Annual Report on the Quality of Care for Children in Medicaid and CHIP.
      Many states also choose to report Healthcare Effectiveness Data and Information Set (HEDIS) measures for their Medicaid CHIP program, but there are concerns about possible variation in the data collection methods used for HEDIS measures.

      National Committee for Quality Assurance. HEDIS compliance audit program. Available at: http://www.ncqa.org/tabid/205/Default.aspx. Accessed December 7, 2013.

      There is also variation across states in how HEDIS results are publicly reported (eg, on a Web site vs an annual report).
      The voluntary nature of the SNAC core measure set and HEDIS measures provide the flexibility states need to measure areas that address the concerns of patients they serve; however, the paucity of common measures among states and the range of reporting methods for pediatric-focused programs limits the ability to evaluate the quality of these programs across states. CMS has made an effort to present measures that use a standardized approach for measurement; however, this does not preclude states from using alternative or additional measures in their own reports.
      Innovation in measurement is laudable and beneficial, assuming that the measures and reports are valid and reliable for informing consumer decision making and guiding provider improvement efforts. Yet there are no standards for reporting on the performance of performance measures, and there is concern that many measures are not up to task.

      Damberg CL, Hyman D, France J. Do public reports of provider performance make their data and methods available and accessible? Med Care Res Rev. Jul 9, 2013. [Epub ahead of print].

      The validity and reliability of most measures that use administrative data, such as the Hospital Acquired Conditions used by CMS, are often unknown or poor.
      • Berenson R.A.
      • Pronovost P.J.
      • Krumholz H.M.
      Achieving the potential of health care performance measures.
      HEDIS measures, which are used by many states, are an exception to this rule when the HEDIS specifications are followed.

      Agency for Healthcare Research and Quality. Established child health care quality measures—HEDIS: child health care quality toolbox. Available at: http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/chtoolbx/measures/measure4.html. Accessed February 1, 2014.

      The concern about validity appears particularly acute for measures constructed from administrative data. Although administrative data systems in health care were primarily designed for reimbursement, stakeholders have used these data to measure the quality of health care services. Administrative data are inexpensive and widely available; however, many criticize this source as inaccurate compared to chart review. For example, the Johns Hopkins Hospital recently received conflicting reviews from 2 state agencies for its performance on central line–associated bloodstream infections during the same time period. The hospital was congratulated when this infection was measured using more accurate clinical data from the Centers for Disease Control and Prevention and criticized when measured using less accurate administrative data. The agreement was 13% between the 2 data sources in positively identifying cases with an infection. The majority of measures in the initial core set are derived exclusively from administrative data, raising potential concerns about the validity and reliability of these measures.

      Recommendations

      We need greater transparency on the performance of measures used in public reports, given the use of nonstandardized measures and measures derived from administrative data. The performance of performance measures is important as we evaluate what measures to include in reports, which to give greater credence to, where better science is needed, and what the trade-offs are in cost and performance between current measures and better measures. The need for greater transparency of measure performance also holds true for adult-related measures.
      The SNAC outlined criteria for evaluating candidate measures for the initial core set, including a measure's validity, feasibility, and understandability, with the scores for each criterion reported publicly. Using this approach as a model, we recommend that future evaluations of performance measures include a specific evaluation against such criteria and that the results be made transparent.
      For performance measures and public reports to be useful and meaningful to stakeholders, we need uniform scientific standards and the transparent reporting of performance against those standards, with a specific focus on reporting on measures' validity and reliability. Without standards and transparency, stakeholders have no way to judge the utility of public reports of provider performance. True innovation in reporting requires that we start from a common book of truth. We have previously discussed the need for a health care organization to serve the transparency function, similar to the Securities and Exchange Commission's function in financial markets. The pediatric Medicaid quality measures provide yet another example of the need for such an entity.

      References

      1. Centers for Medicare and Medicaid Services. 2013 core set of children’s health care quality measures for Medicaid and CHIP. Available at: http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-Topics/Quality-of-Care/Downloads/ChildCoreMeasures.pdf. Accessed December 7, 2013.

        • Agency for Healthcare Research and Quality
        Recommendations to Improve Children’s Health Care Quality Measures: Background Report on the 2012 Process.
        AHRQ, Rockville, Md2012 (Available at:) (Accessed December 7, 2013)
        • Sebelius K.
        The Department of Health and Human Services 2013 Annual Report on the Quality of Care for Children in Medicaid and CHIP.
        Department of Health and Human Services, Washington, DC2013 (Available at:) (Accessed December 7, 2013)
      2. National Committee for Quality Assurance. HEDIS compliance audit program. Available at: http://www.ncqa.org/tabid/205/Default.aspx. Accessed December 7, 2013.

      3. Damberg CL, Hyman D, France J. Do public reports of provider performance make their data and methods available and accessible? Med Care Res Rev. Jul 9, 2013. [Epub ahead of print].

        • Berenson R.A.
        • Pronovost P.J.
        • Krumholz H.M.
        Achieving the potential of health care performance measures.
        Robert Wood Johnson Foundation, 2013 (Available at:) (Accessed February 4, 2014)
      4. Agency for Healthcare Research and Quality. Established child health care quality measures—HEDIS: child health care quality toolbox. Available at: http://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/chtoolbx/measures/measure4.html. Accessed February 1, 2014.