Advertisement
Academic Pediatrics
Advanced SearchSave search
Article| Volume 14, ISSUE 5, SUPPLEMENT , S90-S96, September 01, 2014

Patient Reported Outcomes as Indicators of Pediatric Health Care Quality

DOI:https://doi.org/10.1016/j.acap.2014.06.002
Patient Reported Outcomes as Indicators of Pediatric Health Care Quality
Previous ArticleAddressing Electronic Clinical Information in the Construction of Quality Measures
Next ArticleSystematic Evidence-Based Quality Measurement Life-Cycle Approach to Measure Retirement in CHIPRA

      Abstract

      Health care reform has increased demand for pediatric health care quality evaluations, particularly those that assess the impact of care on patient and population health outcomes. Many of today's most common childhood conditions are characterized by symptoms, behaviors, and functional limitations that are best assessed as patient reported outcomes (PROs). Although they remain greatly underutilized, PROs have the potential to improve pediatric health care quality assessment at the point of care and through system-level performance evaluations. The functions, benefits, and challenges of these PRO applications are described and illustrated in case examples. Although challenges remain, numerous methodological and technical innovations facilitate the use of PROs as health care quality metrics. These include advances in PRO measure development methodologies, the integration of PRO measures into electronic health records, and developing consensus among providers that PROs provide valuable information that can be used to enhance patient care. Although additional work is needed to address remaining methodological challenges, pediatric PROs are increasingly recognized as valuable indicators of health care quality in the clinical environment and as measures of organization- and provider-level performance.

      Keywords

      In the current context of health care reform, there is a critical need to evaluate the degree to which pediatric health services increase the likelihood of desired outcomes for individuals and the population. However, multiple barriers impede the measurement of children's health and health care quality.
      • Institute of Medicine; National Research Council
      Child and Adolescent Health and Health Care Quality: Measuring What Matters Committee on Pediatric Health and Health Care Quality Measures.
      National Academies Press, Washington, DC2011
      These include the fragmentation of data sources that provide information about child health status and health care quality, a lack of standardized metrics, and inconsistencies in data collection methods.
      • Institute of Medicine; National Research Council
      Child and Adolescent Health and Health Care Quality: Measuring What Matters Committee on Pediatric Health and Health Care Quality Measures.
      National Academies Press, Washington, DC2011
      , 
      • Broderick J.E.
      • Morgan-DeWitt E.
      • Rothrock N.
      • et al.
      Advances in patient reported outcomes: the NIH PROMIS measures.
      eGEMs. 2013; 1 (Article 12)
      Recognizing the need to improve pediatric health care quality measurement, the Children's Health Insurance Program Reauthorization Act (CHIPRA) of 2009 mandated specification of an initial core set of quality measures for voluntary use by Medicaid and Children's Health Insurance Program (CHIP) programs, health insurance issuers, managed care entities, and providers.

      US Department of Health and Human Services; Agency for Healthcare Research and Quality. Pediatric Quality Measures Program (PQMP) centers of excellence grant awards. March 2012. Available at: http://www.ahrq.gov/legacy/chipra/pqmpfact.htm. Accessed November 12, 2013.

      Although CHIPRA required that the core set cover a full array of health care quality domains, the measures primarily focus on children's receipt of services (eg, percentage of children who receive preventive dental services or well-child checkups at recommended ages). The core set instruments provide reliable and valid measures of services provided and family experiences of care.
      • Institute of Medicine; National Research Council
      Child and Adolescent Health and Health Care Quality: Measuring What Matters Committee on Pediatric Health and Health Care Quality Measures.
      National Academies Press, Washington, DC2011
      , 
      • Donabedian A.
      The quality of care: how can it be assessed?.
      JAMA. 1988; 260: 1743-1748
      However, despite long-standing recognition that health care quality evaluations should consider the effects of care on the health status of patients and populations, only 5 of the 24 core measures reflect patient health outcomes.
      • Donabedian A.
      The quality of care: how can it be assessed?.
      JAMA. 1988; 260: 1743-1748
      CHIPRA mandated establishment of the Pediatric Quality Measures Program (PQMP) to expand the portfolio of evidence-based measures available to public and private purchasers of children's health care services, providers, and consumers.

      US Department of Health and Human Services; Agency for Healthcare Research and Quality. Pediatric Quality Measures Program (PQMP) centers of excellence grant awards. March 2012. Available at: http://www.ahrq.gov/legacy/chipra/pqmpfact.htm. Accessed November 12, 2013.

      In particular, the program is expanding the availability of pediatric outcome metrics that can be used to evaluate health care quality at the state, health plan, and provider levels. Patient outcomes include physiological functions, symptoms, health perceptions, quality of life, health behaviors, and health care experiences.

      Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

      Many of today's most common childhood conditions (eg, overweight/obesity, developmental delay, attention-deficit/hyperactivity disorder) are best characterized by symptoms, behaviors, and functional limitations. Therefore, relevant and valid pediatric health care quality evaluations should include measures of these outcome types. Because many symptoms, indicators of functioning, and behaviors are experienced by children or observed in their day-to-day lives, their measurement requires direct questioning of the child or an individual who has adequate knowledge of the child's experiences.
      • Bevans K.
      • Forrest C.B.
      The reliability and validity of children's self-reported health status.
      in: Ungar W. Economic Evaluation in Child Health. Oxford University Press, Oxford, UK2010: 33-54
      , 
      • Patrick D.L.
      • Burke L.B.
      • Powers J.H.
      • et al.
      Patient-reported outcomes to support medical product labeling claims: FDA perspective.
      Value Health. 2007; 10: S125-S137
      As such, they exemplify patient reported outcomes (PROs). PROs are any report of the status of a patient's health condition, health behavior, or health care experience that comes directly from the patient, without interpretation by a clinician or anyone else.
      • Patrick D.L.
      • Burke L.B.
      • Powers J.H.
      • et al.
      Patient-reported outcomes to support medical product labeling claims: FDA perspective.
      Value Health. 2007; 10: S125-S137
      , 

      US Department of Health and Human Services; Food and Drug Administration. Guidance for industry patient report outcome measures: use in medical product development to support labeling claims. December 2009. Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. Accessed November 12, 2013.

      Over the past decade, several initiatives were established to improve PRO measurement and generate easily administered, interpretable, and scientifically robust PRO measures (PROMs).
      • Cella D.
      • Yount S.
      • Rothrock N.
      • et al.
      The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
      Med Care. 2007; 45: S3-S11
      In 2004, the National Institutes of Health (NIH) launched the Patient Reported Outcome Measurement Information System (PROMIS). The cooperative group of research sites and centers that comprise PROMIS developed a unique mixed-methods PROM development process, a growing set of PROMs, and an informatics platform that enables Web-based PROM administration.
      • Cella D.
      • Yount S.
      • Rothrock N.
      • et al.
      The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
      Med Care. 2007; 45: S3-S11
      PROMIS has made substantial contributions to the advancement of PRO science by establishing measurement and psychometric standards for PROMs

      National Institutes of Health. PROMIS® instrument development and psychometric evaluation scientific standards. Available at: http://www.nihpromis.org/Documents/PROMIS_Standards_050212.pdf. Accessed October 26, 2013.

      that expand on previous recommendations for PROM development and utilization.
      • Patrick D.L.
      • Burke L.B.
      • Powers J.H.
      • et al.
      Patient-reported outcomes to support medical product labeling claims: FDA perspective.
      Value Health. 2007; 10: S125-S137
      , 

      US Department of Health and Human Services; Food and Drug Administration. Guidance for industry patient report outcome measures: use in medical product development to support labeling claims. December 2009. Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. Accessed November 12, 2013.

      , 
      • Matza L.S.
      • Patrick D.L.
      • Riley A.W.
      • et al.
      Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force.
      Value Health. 2013; 16: 461-479
      To date, PROMIS measures have been developed to assess pediatric global health
      • Forrest C.B.
      • Bevans K.B.
      • Pratiwadi R.
      • et al.
      Development of the PROMIS® pediatric global health (PGH-7) measure.
      Qual Life Res. 2014; 23: 1221-1231
      and 18 physical, mental, and social health outcomes by child and parent report.
      • Forrest C.B.
      • Bevans K.B.
      • Tucker C.
      • et al.
      The Patient Reported Outcome Measurement Information System (PROMIS®) for children and youth: application to pediatric psychology.
      J Pediatr Psychol. 2012; 37 ([commentary]): 614-621
      , 
      • Bevans K.B.
      • Gardner W.
      • Pajer K.
      • et al.
      Qualitative development of the PROMIS® pediatric stress response item banks.
      J Pediatr Psychol. 2013; 38: 173-191
      , 
      • Irwin D.E.
      • Stucky B.
      • Langer M.M.
      • et al.
      An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales.
      Qual Life Res. 2010; 19: 595-607
      Researchers at a CHIPRA PQMP Center of Excellence (the Children's Hospital of Philadelphia) applied the PROMIS methodology to develop and validate the Pediatric Global Health-7 measure (PGH-7), a 7-item PROM that assesses children's perceptions of their overall health.
      • Forrest C.B.
      • Bevans K.B.
      • Pratiwadi R.
      • et al.
      Development of the PROMIS® pediatric global health (PGH-7) measure.
      Qual Life Res. 2014; 23: 1221-1231
      The same research team is currently working with patients, caregivers, clinicians, the Dental Quality Alliance, and Medicaid leaders to generate child- and parent-report measures of pediatric oral health.
      PROMs can be used to evaluate pediatric health care quality in 2 contexts: at the point of care and in health care system performance evaluations. In both cases, PROMs provide information about health and the impact of health care that is unique and complementary to information derived from measures of clinical outcomes and health care processes. As described below, individual- and population-based PRO applications have unique functions, benefits, and practical and analytic challenges.

      Enhancing Health Care Quality at the Point of Care

      PRO measurement in clinical practice is increasing, but remains relatively uncommon, especially in pediatric care.
      • Broderick J.E.
      • Morgan-DeWitt E.
      • Rothrock N.
      • et al.
      Advances in patient reported outcomes: the NIH PROMIS measures.
      eGEMs. 2013; 1 (Article 12)
      , 

      Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

      PRO assessment informs patient-centered care by exposing child and family health concerns and treatment preferences, which often differ from those of clinicians.
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      This information can be used to engage patients and their families in medical decision making and treatment planning.

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      PROs tracked over time reveal changes in child health. When monitored in the context of intervention, PROs can be used to evaluate treatment effectiveness and inform treatment modifications.
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      PRO evaluations in clinical care may enhance patient–provider communication, patient activation, and treatment adherence,
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 
      • Detmar S.B.
      • Muller M.J.
      • Schornagel J.H.
      • et al.
      Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial.
      JAMA. 2002; 288: 3027-3034
      , 
      • Lohr K.N.
      • Zebrack B.J.
      Using patient-reported outcomes in clinical practice: challenges and opportunities.
      Qual Life Res. 2009; 18: 107
      which is particularly important during adolescence when compliance with treatment recommendations often declines. Given these promising trends, additional research is needed to evaluate other ways that PRO measurement impacts pediatric treatment planning and outcomes.
      • Broderick J.E.
      • Morgan-DeWitt E.
      • Rothrock N.
      • et al.
      Advances in patient reported outcomes: the NIH PROMIS measures.
      eGEMs. 2013; 1 (Article 12)
      , 
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      Two case examples are presented to illustrate the advantages and challenges of PRO use in clinical care.

       KLIK ePROfile: Emma Children's Hospital and Sint Lucas Andreas Hospital, Amsterdam

      KLIK is a Web-based application that supports the use of PROMs in clinical practice (http://www.hetklikt.nu).
      • Haverman L.
      • van Rossum M.A.J.
      • van Veenendaal M.
      • et al.
      Effectiveness of a Web-based application to monitor health-related quality of life.
      Pediatrics. 2013; 131: e533-e543
      , 
      • Haverman L.
      • Engelen V.
      • van Rossum M.A.J.
      • et al.
      Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
      BMC Pediatr. 2011; 11: 3
      KLIK utilization procedures were developed on the basis of many years of experience using PROs in diverse clinical contexts.
      • Haverman L.
      • Engelen V.
      • van Rossum M.A.J.
      • et al.
      Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
      BMC Pediatr. 2011; 11: 3
      , 
      • Engelen V.
      • Haverman L.
      • Koopman H.M.
      • et al.
      Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice.
      Patient Educ Couns. 2010; 81: 235-244
      Several days before a scheduled appointment, the KLIK Web site generates an automatic e-mail that invites children aged 8 to 18 years or parents of children aged 0 to 7 years to complete electronic PRO (ePRO) measures through a secure Web site. The measures include generic and condition-specific PROMs that assess multiple dimensions of health-related quality of life (HRQoL).
      Child or parent responses are used to generate ePROfile reports, which can be retrieved by the child's health care providers and used during the clinical encounter to identify, monitor, and discuss HRQoL problems.
      • Donabedian A.
      The quality of care: how can it be assessed?.
      JAMA. 1988; 260: 1743-1748
      , 

      Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

      On the basis of prior recommendations,
      • Engelen V.
      • Haverman L.
      • Koopman H.M.
      • et al.
      Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice.
      Patient Educ Couns. 2010; 81: 235-244
      , 
      • Greenhalgh J.
      The applications of PROs in clinical practice: what are they, do they work, and why?.
      Qual Life Res. 2009; 18: 115-123
      the ePROfile report consists of 2 parts: 1) PROM item-level responses that are color-coded such that “always” or “often” a problem are red; PROs that are “sometimes” a problem are orange; and PROs that are “never” or “almost never” a problem are green; and 2) graphs of longitudinally assessed PROM scores relative to healthy population norms. These graphs enable clinicians to inquire about specific HRQoL problems and to monitor the improvement or worsening of these outcomes over time.
      • Haverman L.
      • Engelen V.
      • van Rossum M.A.J.
      • et al.
      Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
      BMC Pediatr. 2011; 11: 3
      A multicenter control group study involving 176 children with juvenile idiopathic arthritis provides preliminary evidence of KLIK's impact on pediatric care. Although all participating child–parent dyads completed HRQoL measures using KLIK, only those in the intervention group (n = 109 dyads) reviewed their ePROfile report (PROM scores) with a physician during a routine appointment. This study demonstrated that physician access to the ePROfile led to more frequent discussions about children's emotional health and social functioning and greater physician satisfaction with the care they provided, particularly their capacity to provide emotional support.
      • Haverman L.
      • van Rossum M.A.J.
      • van Veenendaal M.
      • et al.
      Effectiveness of a Web-based application to monitor health-related quality of life.
      Pediatrics. 2013; 131: e533-e543
      Given its positive impact, the KLIK/ePROfile is now implemented in routine clinical care throughout Emma Children's Hospital and Sint Lucas Andreas Hospital.
      • Haverman L.
      • van Rossum M.A.J.
      • van Veenendaal M.
      • et al.
      Effectiveness of a Web-based application to monitor health-related quality of life.
      Pediatrics. 2013; 131: e533-e543

       Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio

      The Cincinnati Children's Hospital Medical Center (CCHMC) increased adoption of PROs in clinical practice as part of a hospitalwide strategic initiative to improve patient outcomes.

      Wu AW, Jensen RE, Salzberg C, Snyder C. Advances in the use of patient reported outcome measures in electronic health records including case studies. 2013. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHR-Landscape-Review-111913.pdf.

      PRO assessment takes place in many CCHMC clinics using a kiosk- or tablet-based PROM administration platform. Patients' PRO responses are combined with data derived from their electronic health records to identify, tailor, and gauge the effectiveness of health care interventions. Currently, physical functioning and pain interference PROs are monitored and reviewed during children's visits to the rheumatology clinic. In the heart institute, children with cardiomyopathy are routinely administered a PROM to ensure early detection of depression because children with chronic disease who experience depression have poorer long-term outcomes. The food allergy clinic uses PROMs to identify children who experience anxiety related to their allergic reactions.

      Wu AW, Jensen RE, Salzberg C, Snyder C. Advances in the use of patient reported outcome measures in electronic health records including case studies. 2013. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHR-Landscape-Review-111913.pdf.

      Researchers at CCHMC are currently partnering with patients and providers in the ImproveCareNow network to test the Personalized Learning System among adolescents with inflammatory bowel disease.
      • Kaplan H.C.
      • Adler J.
      • Saeed S.A.
      • et al.
      A personalized learning system for improving patient-physician collaboration.
      Harvard Business Rev. May 23, 2013;
      Using a Web-based interface, patients identify and track PROs that are important to them, usually on a daily basis. They also catalog routine changes in everyday life (eg, changes in diet, sleep, travel) and adherence with treatment recommendations (eg, medication use). Patients can submit data via text, e-mail, or Web-based forms. The data are used to generate real-time reports that may inform treatment decisions by providing valuable information about the effectiveness of specific lifestyle changes and health care strategies. The report contains graphs of longitudinally assessed PRO values and concurrent lifestyle of treatment events. The Figure displays the number of patient reported bowel movements per night over a 4-month period. The patient experienced a significant reduction in the number of nocturnal bowel movements when taking amoxicillin for a coincident sinus infection (shaded area). A similar trend was observed on another occasion when the patient was treated with a second course of antibiotics for recurrent sinusitis (data not shown). On the basis of these initial findings, an experiment was designed to test whether rifaximin (a nonabsorbable antibiotic) is an effective treatment for this patient's inflammatory bowel disease symptoms.
      Figure thumbnail gr1
      FigureExample data generated through the personalized learning system for an adolescent with inflammatory bowel disease.
      This figure is reproduced with permission of the Harvard Business Review.
      • Kaplan H.C.
      • Adler J.
      • Saeed S.A.
      • et al.
      A personalized learning system for improving patient-physician collaboration.
      Harvard Business Rev. May 23, 2013;
      As demonstrated by these case examples, PROMs can be used to gauge the effectiveness of health care services at the individual patient level. However, successful PRO assessment and monitoring requires attention to numerous methodological and practical factors.
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      First, PROMs administered in clinical settings should assess concepts that are significant and relevant to a patient's condition and its treatment.
      • Matza L.S.
      • Patrick D.L.
      • Riley A.W.
      • et al.
      Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force.
      Value Health. 2013; 16: 461-479
      , 
      • Patrick D.L.
      • Burke L.B.
      • Gwaltney C.J.
      • et al.
      Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
      Value Health. 2011; 14: 978-988
      The Personalized Learning System is used to identify and track patients' outcome priorities, which are often assessed using PROMs. Notably, the perceived importance of many PROs varies across the life course and as health states change. Thus, it is essential to periodically reevaluate PRO content to ensure that the outcomes are meaningful and important to the patient.
      Second, the psychometric properties of pediatric PROMs must be thoroughly evaluated among representatives of the target population (eg, children of a specific age). Child-report PROMs should be designed with sensitivity to children's understanding of health/illness concepts and overall cognitive capacities.
      • Bevans K.
      • Forrest C.B.
      The reliability and validity of children's self-reported health status.
      in: Ungar W. Economic Evaluation in Child Health. Oxford University Press, Oxford, UK2010: 33-54
      Several CCHMC clinics routinely administer PROMIS instruments, which were developed in accordance with rigorous psychometric standards that emphasize item understandability and the minimization of age-based measurement bias.
      • Cella D.
      • Yount S.
      • Rothrock N.
      • et al.
      The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
      Med Care. 2007; 45: S3-S11
      Third, PROMs should be selected according to patients' needs and treatment goals. Whereas generic PROMs enable comparisons across healthy and chronic disease populations, condition-specific measures may provide more sensitive measures of intervention effects.

      Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

      The KLIK developers administered a combination of generic and condition-specific HRQoL measures to children with juvenile idiopathic arthritis to ensure that the ePROfile provides information about all domains of the population's clinical care.
      • Haverman L.
      • van Rossum M.A.J.
      • van Veenendaal M.
      • et al.
      Effectiveness of a Web-based application to monitor health-related quality of life.
      Pediatrics. 2013; 131: e533-e543
      Finally, the frequency and timing of PROM administration depends on the goals of health care. A single PRO administration (screening) may reveal problems that would otherwise go undetected, but repeated administrations are needed to evaluate response to intervention.
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      Although KLIK and ePROfile were designed to track PROs over time, preliminary reports of the system's effectiveness focus on improving the initial characterization of children's HRQoL.
      • Haverman L.
      • van Rossum M.A.J.
      • van Veenendaal M.
      • et al.
      Effectiveness of a Web-based application to monitor health-related quality of life.
      Pediatrics. 2013; 131: e533-e543
      , 
      • Haverman L.
      • Engelen V.
      • van Rossum M.A.J.
      • et al.
      Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
      BMC Pediatr. 2011; 11: 3
      In contrast, the Personalized Learning System is designed for longitudinal data tracking that patients and providers can use to formulate hypotheses about factors that might improve the patients' health.
      • Kaplan H.C.
      • Adler J.
      • Saeed S.A.
      • et al.
      A personalized learning system for improving patient-physician collaboration.
      Harvard Business Rev. May 23, 2013;
      Technological advances reduce several logistical barriers to PRO assessment in clinical contexts. Internet-based data collection platforms such as the NIH Assessment Center enhance PROM administration feasibility, in part by minimizing administration burden. To reduce staff involvement in PROM administration, CCHMC is developing systems to support Web-based PRO measurement, which like KLIK, can be completed by patients and caregivers outside of the clinical appointment.

      Wu AW, Jensen RE, Salzberg C, Snyder C. Advances in the use of patient reported outcome measures in electronic health records including case studies. 2013. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHR-Landscape-Review-111913.pdf.

      The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 promotes the exchange and meaningful use of electronic health record (EHR) data including PROs collected through patient portals.
      • Wu A.W.
      • Kharrazi H.
      • Boulware L.E.
      • Snyder C.F.
      Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research.
      J Clin Epidemiol. 2013; 66: S12-S20
      A leading EHR developer (Epic, Verona, WI; www.epic.com) integrated a PRO application into its recent software release (Fall 2012). The software supports identification of “events” for each patient (eg, upcoming health care service, change in treatment, predefined interval), at which time the application sends the patient/parent an e-mail requesting completion of a PROM. Data are scored and stored in the EHR, and can be retrieved by health care providers for clinical use.
      • Broderick J.E.
      • Morgan-DeWitt E.
      • Rothrock N.
      • et al.
      Advances in patient reported outcomes: the NIH PROMIS measures.
      eGEMs. 2013; 1 (Article 12)
      Beyond logistical concerns, more resources are needed to support PRO use in clinical care.

      Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

      , 
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      These changes depend on a culture of patient-centered and collaborative care, which recognizes that high-quality health care is informed jointly by clinicians' technical expertise and patients' and/or caregivers' knowledge of their health and health care priorities. Clinician buy-in, which is crucial to the successful integration of PROMs in practice, may be enhanced by clinician training, incentives for PRO use, integration of PROs into practice guidelines, and the development of clinician-informed interventions to improve PROs.
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      To improve the use of PRO data in clinical practice, the KLIK/ePROfile developers train pediatricians on how to integrate PROfile results into their patient care.
      • Haverman L.
      • Engelen V.
      • van Rossum M.A.J.
      • et al.
      Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
      BMC Pediatr. 2011; 11: 3
      Patient buy-in is also crucial because the successful use of PROs in clinical care requires that children and parents provide data, often repeatedly.

      PROMs as Performance Measures

      Given the widespread variation in children's access to safe, effective, and affordable health care in the United States, there is a critical need to assess the performance of health care organizations with the goal of improving pediatric services, child and family outcomes, and population health.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      Organizational performance measures are numeric quantifications of health care quality for a designated accountable health care entity, such as a hospital, health plan, or clinic.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      Performance measures can take many forms but are most commonly percentage, mean, median, or ratio values.
      Performance measurement provides stakeholders with information they can use to improve health care quality. Patients or caregivers can use the information to make informed choices about the care they or their family members receive. For physicians and other health care providers (at the individual, clinic, medical group, and hospital levels), performance measurement provides information about the degree to which their services meet their patients' needs. Last, health care purchasers such as state Medicaid and Children's Health Insurance Programs, Centers for Medicare and Medicaid Services (CMS), as well as employers, can use the information to ensure they are getting the best value for their health care dollars. Performance measures can be used in quality improvement and accountability activities, including public reporting and performance-based payment.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      , 

      Goodrich K. Patient reported outcome measures in CMS programs. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-Presentation-Panel-II-111913.pdf.

      State Medicaid, CHIP leaders, and other key stakeholders recognize the need to integrate child health outcomes into assessments of pediatric health care quality.

      Frank R. State Children's Health Insurance Program—workshop on pediatric health and health care quality measurement and information needs. 2010. Available at: http://www.iom.edu/∼/media/Files/Activity Files/Quality/PediatricQualityMeasures/.

      In particular, stakeholders emphasize the need to gauge pediatric health care quality on the basis of children's functional capacities and wellbeing, which are commonly assessed using PROMs.

      Frank R. State Children's Health Insurance Program—workshop on pediatric health and health care quality measurement and information needs. 2010. Available at: http://www.iom.edu/∼/media/Files/Activity Files/Quality/PediatricQualityMeasures/.

      , 

      Wadhwa S. Measuring children's health and health care—workshop on pediatric health and health care quality measurement and information needs. 2010. Available at: http://www.iom.edu/∼/media/Files/Activity%20Files/Quality/PediatricQualityMeasures/Sandeep%20Wadhwa_3_23_10.ashx.

      PRO-performance measures (PRO-PMs) are based on PROM data aggregated for an entity deemed accountable for the quality of health care (eg, hospitals, physician practices, Medicaid or CHIP programs).
      Despite increased attention to the development of scientifically rigorous PRO-PMs,

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      , 

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      these measures remain underutilized, especially in pediatrics. A case example is presented to illustrate how a nonprofit organization applied PRO-PMs to assess health care quality for children with asthma.
      Minnesota Community Measurement (MNCM) is dedicated to improving Minnesota residents' health by publicly reporting information about health care costs, quality, and patient experiences (http://mncm.org/). The organization uses PRO-PMs to gauge the quality of health services offered by clinic, medical groups, and purchasers.

      MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

      Because asthma is a highly prevalent pediatric condition (7 million children in the United States) that has significant effects on child outcomes (eg, hospitalizations, school absenteeism),7 MNCM conducts annual evaluations of many health care conditions including asthma.

      MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

      Clinic performance is evaluated by the Optimal Asthma Care measure. This performance measure yields the percentage of patients aged 5 to 17 years with persistent asthma who meet 3 asthma control targets: 1) controlled asthma indicated by the patient's score on 1 of 3 validated PRO measure (Asthma Control Test,
      • Nathan R.A.
      • Sorkness C.A.
      • Kosinski M.
      • et al.
      Development of the Asthma Control Test: a survey for assessing asthma control.
      J Allergy Clin Immunol. 2004; 113: 59-65
      Asthma Control Questionnaire),
      • Juniper E.F.
      • O'Byrne P.M.
      • Guyatt G.H.
      • et al.
      Development and validation of a questionnaire to measure asthma control.
      Eur Respir J. 1999; 14: 902-907
      Asthma Therapy Assessment Questionnaire
      • Vollmer W.M.
      • Markson L.E.
      • O'Connor E.
      • et al.
      Association of asthma control with health care utilization and quality of life.
      Am J Respir Crit Care Med. 1999; 160: 1647-1652
      ; 2) no patient reported emergency department visits or hospitalizations due to asthma; and 3) patient educated about asthma self-management and received a written asthma management plan.

      MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

      Clinics abstract these data from their electronic health records or medical charts and submit them to MNCM through a secure data portal. The data are aggregated or stratified multiple ways to enable comparisons, such as across clinics, higher levels of aggregation such as medical groups, or between purchasers (eg, publicly financed programs vs private/other purchasers).
      MNCM reports the percentage of patients with optimal asthma care for individual clinics, which in 2013 ranged from 0% to 89%. Data aggregated by purchaser revealed a significant disparity in asthma control performance between publicly financed programs (eg, Medicaid, CHIP) and other purchasers (eg, private insurers). Optimal asthma care was offered to 32% of children with public insurance and 40% of children with nonpublic insurance.

      MN Community Measurement. 2012 Health Care Disparities Report for Minnesota Health Care Programs. Available at: http://mncm.org/wp-content/uploads/2013/04/Final2012HealthCareDisparitiesReport5.10.pdf.

      Comparisons such as these are important for publicly financed health insurance programs because purchasers like Medicaid and CHIP have an interest in establishing parity with or exceeding outcomes experienced by patients covered by other purchasers.
      The data are also used to track changes in asthma care quality over time. Between 2011 and 2012, clinics' rates of optimal asthma care increased to a greater degree than most other health care quality indicators.

      MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

      These improvements were largely attributable to an increase in the number of clinics that used validated asthma symptom PROMs, documented hospitalizations and emergency department visits, and developed asthma management plans for their patients.
      MNCM's evaluations demonstrate the impact that PRO-based performance measurement and reporting can have on the quality of health care delivery. Accordingly, the National Quality Forum, which reviews and endorses health care performance measures, has increased its focus on PRO-PMs. They developed an evaluative framework for PRO-PM endorsement that highlights the many factors that impact a measure's capacity to yield valid estimates of providers' performance.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      , 

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      First, to ensure that PRO-PMs' positively impact health care quality in meaningful ways, it is important to ensure that the measures reflect patients', caregivers', and providers' health care improvement priorities.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      , 

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      Priorities should reflect population health needs, particularly those for which there are subgroup disparities. The National Quality Forum and other leaders in PRO-based performance measurement specify that PRO-PMs should be grounded in empirically supported pathway models that delineate the impact of health care structure and process on the outcomes.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      , 

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      Others argue that even if PRO-PMs are not strongly linked to any particular process of care, they should be included as performance measures because improved patient health is the most important goal of accountable health care delivery and public health systems.
      Second, as with other PROM applications, the instruments that underlie PRO-PMs should provide reliable, valid, and meaningful estimates of a target population's health outcomes.
      • Matza L.S.
      • Patrick D.L.
      • Riley A.W.
      • et al.
      Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force.
      Value Health. 2013; 16: 461-479
      , 
      • Snyder C.F.
      • Aaronson N.K.
      Use of patient-reported outcomes in clinical practice.
      Lancet. 2009; 374: 369-370
      , 

      Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

      Psychometrically inadequate performance measurement may result in misspecification of providers' quality rankings, which may impact public reporting, perceptions of the providers' quality, and pay for performance.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      , 
      • Adams J.L.
      The Reliability of Provider Profiling: A Tutorial.
      Rand Corporation, Santa Monica, Calif2009
      http://www.rand.org/pubs/technical_reports/TR653.html
      Third, the PRO-PM data must be feasible to collect, store, transform, and report.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      PROMs that are overly burdensome to administer or complete yield lower response rates, which may bias results and reduce the likelihood that providers will achieve the sample sizes needed for a specific level of analysis (eg, clinic, hospital). General sample size cutoffs are commonly used to exclude providers from public reporting as a result of concerns about vulnerability to random error.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      For example, although MNCM strongly encouraged clinics to submit Optimal Asthma Care data for their total patient population, they accepted data that were systematically sampled from at least 60 patients.

      MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

      This approach may be problematic because reliability does not depend on sample size alone; it is also impacted by real differences between providers and within-provider variation.
      • Adams J.L.
      The Reliability of Provider Profiling: A Tutorial.
      Rand Corporation, Santa Monica, Calif2009
      http://www.rand.org/pubs/technical_reports/TR653.html
      One way to maximize sample sizes is to reduce the burden of PRO-PM assessment by repurposing scores on PROMs that are generated and used at the point-of-care. PRO scores used in clinical care can be aggregated into PRO-PM data; however, most health care providers do not yet have the capacity to collect, store, and transform PROM information.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      Epic, recently developed a PRO application that supports multipurpose use of PRO data. Further work is needed to support the integration of PRO data into EHRs and the transformation of those data into PRO-PM information that could be shared across health care systems to support performance assessment, quality improvement, research, and population surveillance.
      • Wu A.W.
      • Kharrazi H.
      • Boulware L.E.
      • Snyder C.F.
      Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research.
      J Clin Epidemiol. 2013; 66: S12-S20
      Given that PROMs have not been widely adopted in pediatric settings, raising awareness among health care providers about their benefits in clinical contexts may encourage their adoption, thereby generating data that can be repurposed for performance measurement and other applications.
      Fourth, there are number of analytic issues that must be addressed when generating PRO-PMs based on individual PROM data. PRO-PMs calculated at the provider level may be means, percentages, or ratio values. The best approach for generating a discriminating PRO-PM depends on the PROM and the distribution of the measured outcome in the population.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      Provider performance may be best expressed in terms of change in patient health status (eg, average amount of decreased symptoms from beginning to end of treatment) or a threshold achieved (eg, percentage of patients with moderate to severe symptoms).

      US Department of Health and Human Services; Food and Drug Administration. Guidance for industry patient report outcome measures: use in medical product development to support labeling claims. December 2009. Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. Accessed November 12, 2013.

      Risk adjustment techniques are often used to maximize the reliability and validity of PRO-PMs. Statistical risk adjustment allows for fair comparisons of outcomes by accounting for differences in patients' personal/demographic and clinical characteristics, thereby ensuring that observed differences are attributable to health care quality, not population differences.

      Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

      Notably, analyses that adjust for child and family sociodemographic characteristics associated with poorer health outcomes (eg, race, ethnicity, limited English proficiency) should be applied with caution because these procedures could mask disparities in health care quality.

      National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

      Last, the interpretation of PRO-PM scores should be informed by knowledge of what constitutes a PROM's minimal important difference, the smallest score change that is considered meaningful by patients and clinicians.
      • Revicki D.
      • Hays R.D.
      • Cella D.
      • Sloan J.
      Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes.
      J Clin Epidemiol. 2008; 61: 102-109
      Health care “responders” are patients whose scores change to a degree that equals or exceeds the amount of change demonstrated in the target population to have significant treatment benefit.
      • McLeod L.D.
      • Coon C.D.
      • Martoin S.A.
      • et al.
      Interpreting patient-reported outcomes results: US FDA guidance and emerging methods.
      Expert Rev Pharmacoecon Outcomes Res. 2011; 11: 163-169
      When such values are known, PRO-PMs may be calculated to reflect the percentage of patients who experience clinically meaningful changes in PROs within a predetermined time period. Although the magnitude of clinically meaningful change is unknown for most pediatric PROMs, the PROMIS pediatric instruments are currently undergoing clinical validation, which will result in the quantification of minimal important differences for these measures.

      Summary

      The use of PROs in pediatric health care and performance measurement facilitates identification of interventions and programs that result in meaningful and clinically important improvements in children's health. When appropriate PROMs are used, the effects of health care interventions can be assessed in developmentally, cognitively, culturally, and clinically diverse populations. The expanding culture of patient-centered care provides a context in which PROs are increasingly recognized as valuable indicators of health care quality. Furthermore, recent methodological and technical innovations, such as the integration of psychometrically robust and adaptive PROMs into electronic health records reduce many barriers to PROM usage. Despite these advances, additional work is needed to educate health care providers about the value of using PROs in clinical care, develop innovative and scalable approaches for personalizing PROs at the individual patient level, establish clinically meaningful values and target thresholds for pediatric PRO-PMs, and enhance the feasibility of PRO-based performance assessment by repurposing PRO data for multiple uses.

      Acknowledgment

      This work was funded by a Pediatric Quality Measures Program U18HS20408 from the Agency for Healthcare Research and Quality .

      References

        • Institute of Medicine; National Research Council
        Child and Adolescent Health and Health Care Quality: Measuring What Matters Committee on Pediatric Health and Health Care Quality Measures.
        National Academies Press, Washington, DC2011
        View in Article
        • Broderick J.E.
        • Morgan-DeWitt E.
        • Rothrock N.
        • et al.
        Advances in patient reported outcomes: the NIH PROMIS measures.
        eGEMs. 2013; 1 (Article 12)
        View in Article

      3. US Department of Health and Human Services; Agency for Healthcare Research and Quality. Pediatric Quality Measures Program (PQMP) centers of excellence grant awards. March 2012. Available at: http://www.ahrq.gov/legacy/chipra/pqmpfact.htm. Accessed November 12, 2013.

        View in Article
        • Donabedian A.
        The quality of care: how can it be assessed?.
        JAMA. 1988; 260: 1743-1748
        View in Article

      5. Cella D, Hahn EA, Jensen SE, et al. Methodological issues in the selection, administration and use of patient-reported outcomes in performance measurement in health care settings. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2.

        View in Article
        • Bevans K.
        • Forrest C.B.
        The reliability and validity of children's self-reported health status.
        in: Ungar W. Economic Evaluation in Child Health. Oxford University Press, Oxford, UK2010: 33-54
        View in Article
        • Patrick D.L.
        • Burke L.B.
        • Powers J.H.
        • et al.
        Patient-reported outcomes to support medical product labeling claims: FDA perspective.
        Value Health. 2007; 10: S125-S137
        View in Article

      8. US Department of Health and Human Services; Food and Drug Administration. Guidance for industry patient report outcome measures: use in medical product development to support labeling claims. December 2009. Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf. Accessed November 12, 2013.

        View in Article
        • Cella D.
        • Yount S.
        • Rothrock N.
        • et al.
        The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
        Med Care. 2007; 45: S3-S11
        View in Article

      10. National Institutes of Health. PROMIS® instrument development and psychometric evaluation scientific standards. Available at: http://www.nihpromis.org/Documents/PROMIS_Standards_050212.pdf. Accessed October 26, 2013.

        View in Article
        • Matza L.S.
        • Patrick D.L.
        • Riley A.W.
        • et al.
        Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPOR PRO good research practices for the assessment of children and adolescents task force.
        Value Health. 2013; 16: 461-479
        View in Article
        • Forrest C.B.
        • Bevans K.B.
        • Pratiwadi R.
        • et al.
        Development of the PROMIS® pediatric global health (PGH-7) measure.
        Qual Life Res. 2014; 23: 1221-1231
        View in Article
        • Forrest C.B.
        • Bevans K.B.
        • Tucker C.
        • et al.
        The Patient Reported Outcome Measurement Information System (PROMIS®) for children and youth: application to pediatric psychology.
        J Pediatr Psychol. 2012; 37 ([commentary]): 614-621
        View in Article
        • Bevans K.B.
        • Gardner W.
        • Pajer K.
        • et al.
        Qualitative development of the PROMIS® pediatric stress response item banks.
        J Pediatr Psychol. 2013; 38: 173-191
        View in Article
        • Irwin D.E.
        • Stucky B.
        • Langer M.M.
        • et al.
        An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales.
        Qual Life Res. 2010; 19: 595-607
        View in Article
        • Snyder C.F.
        • Aaronson N.K.
        Use of patient-reported outcomes in clinical practice.
        Lancet. 2009; 374: 369-370
        View in Article

      17. Aaronson N, Choucair A, Elliott T, et al. User's guide to implementing patient-reported outcomes assessment in clinical practice. 2011. Available at: http://www.isoqol.org/UserFiles/file/UsersGuide.pdf.

        View in Article
        • Detmar S.B.
        • Muller M.J.
        • Schornagel J.H.
        • et al.
        Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial.
        JAMA. 2002; 288: 3027-3034
        View in Article
        • Lohr K.N.
        • Zebrack B.J.
        Using patient-reported outcomes in clinical practice: challenges and opportunities.
        Qual Life Res. 2009; 18: 107
        View in Article
        • Haverman L.
        • van Rossum M.A.J.
        • van Veenendaal M.
        • et al.
        Effectiveness of a Web-based application to monitor health-related quality of life.
        Pediatrics. 2013; 131: e533-e543
        View in Article
        • Haverman L.
        • Engelen V.
        • van Rossum M.A.J.
        • et al.
        Monitoring health-related quality of life in paediatric pratice: development of an innovative Web-based application.
        BMC Pediatr. 2011; 11: 3
        View in Article
        • Engelen V.
        • Haverman L.
        • Koopman H.M.
        • et al.
        Development and implementation of a patient reported outcome intervention (QLIC-ON PROfile) in clinical paediatric oncology practice.
        Patient Educ Couns. 2010; 81: 235-244
        View in Article
        • Greenhalgh J.
        The applications of PROs in clinical practice: what are they, do they work, and why?.
        Qual Life Res. 2009; 18: 115-123
        View in Article

      24. Wu AW, Jensen RE, Salzberg C, Snyder C. Advances in the use of patient reported outcome measures in electronic health records including case studies. 2013. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-EHR-Landscape-Review-111913.pdf.

        View in Article
        • Kaplan H.C.
        • Adler J.
        • Saeed S.A.
        • et al.
        A personalized learning system for improving patient-physician collaboration.
        Harvard Business Rev. May 23, 2013;
        View in Article
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
        Value Health. 2011; 14: 978-988
        View in Article
        • Wu A.W.
        • Kharrazi H.
        • Boulware L.E.
        • Snyder C.F.
        Measure once, cut twice—adding patient-reported outcome measures to the electronic health record for comparative effectiveness research.
        J Clin Epidemiol. 2013; 66: S12-S20
        View in Article

      28. National Quality Forum. Patient reported outcomes (PROs) in performance measurement. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx.

        View in Article

      29. Deutsch A, Smith L, Gage B, et al. Patient-reported outcomes in performance measurement: commissioned paper on PRO-based performance measures for healthcare accountable entities. Available at: http://www.qualityforum.org/Projects/n-r/Patient-Reported_Outcomes/Patient-Reported_Outcomes.aspx#t=2&s=&p=2|.

        View in Article

      30. Goodrich K. Patient reported outcome measures in CMS programs. Available at: http://www.pcori.org/assets/2013/11/PCORI-PRO-Workshop-Presentation-Panel-II-111913.pdf.

        View in Article

      31. Frank R. State Children's Health Insurance Program—workshop on pediatric health and health care quality measurement and information needs. 2010. Available at: http://www.iom.edu/∼/media/Files/Activity Files/Quality/PediatricQualityMeasures/.

        View in Article

      32. Wadhwa S. Measuring children's health and health care—workshop on pediatric health and health care quality measurement and information needs. 2010. Available at: http://www.iom.edu/∼/media/Files/Activity%20Files/Quality/PediatricQualityMeasures/Sandeep%20Wadhwa_3_23_10.ashx.

        View in Article

      33. MN Community Measurement. 2013 Health Care Quality Report. Available at: http://mncm.org/wp-content/uploads/2014/02/2013-HCQR-Final-2.4.2014.pdf.

        View in Article
        • Nathan R.A.
        • Sorkness C.A.
        • Kosinski M.
        • et al.
        Development of the Asthma Control Test: a survey for assessing asthma control.
        J Allergy Clin Immunol. 2004; 113: 59-65
        View in Article
        • Juniper E.F.
        • O'Byrne P.M.
        • Guyatt G.H.
        • et al.
        Development and validation of a questionnaire to measure asthma control.
        Eur Respir J. 1999; 14: 902-907
        View in Article
        • Vollmer W.M.
        • Markson L.E.
        • O'Connor E.
        • et al.
        Association of asthma control with health care utilization and quality of life.
        Am J Respir Crit Care Med. 1999; 160: 1647-1652
        View in Article

      37. MN Community Measurement. 2012 Health Care Disparities Report for Minnesota Health Care Programs. Available at: http://mncm.org/wp-content/uploads/2013/04/Final2012HealthCareDisparitiesReport5.10.pdf.

        View in Article
        • Adams J.L.
        The Reliability of Provider Profiling: A Tutorial.
        Rand Corporation, Santa Monica, Calif2009 (Available at:)
        http://www.rand.org/pubs/technical_reports/TR653.html
        View in Article
        • Revicki D.
        • Hays R.D.
        • Cella D.
        • Sloan J.
        Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes.
        J Clin Epidemiol. 2008; 61: 102-109
        View in Article
        • McLeod L.D.
        • Coon C.D.
        • Martoin S.A.
        • et al.
        Interpreting patient-reported outcomes results: US FDA guidance and emerging methods.
        Expert Rev Pharmacoecon Outcomes Res. 2011; 11: 163-169
        View in Article

      Article Info

      Publication History

      Accepted: June 5, 2014
      Received: November 13, 2013

      Footnotes

      The authors declare that they have no conflict of interest.

      Publication of this article was supported by the US Department of Health and Human Services and the Agency for Healthcare Research and Quality.

      The views expressed in this article are those of the authors and do not necessarily represent the views of the US Department of Health and Human Services or the Agency for Healthcare Research and Quality.

      Identification

      DOI: https://doi.org/10.1016/j.acap.2014.06.002

      Copyright

      © 2014 Academic Pediatric Association. Published by Elsevier Inc. All rights reserved.

      ScienceDirect

      Access this article on ScienceDirect
      We use cookies to help provide and enhance our service and tailor content. To update your cookie settings, please visit the for this site.
      Copyright © 2022 Elsevier Inc. except certain content provided by third parties. The content on this site is intended for healthcare professionals.


      RELX