If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to L. Charles Bailey, MD, PhD, Division of Oncology/CTRB 10407, Children's Hospital of Philadelphia, 34th St and Civic Center Blvd, Philadelphia, PA 19104.
Electronic health records (EHR) and registries play a central role in health care and provide access to detailed clinical information at the individual, institutional, and population level. Use of these data for clinical quality/performance improvement and cost management has been a focus of policy initiatives over the past decade. The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA)-mandated Pediatric Quality Measurement Program supports development and testing of quality measures for children on the basis of electronic clinical information, including de novo measures and respecification of existing measures designed for other data sources. Drawing on the experience of Centers of Excellence, we review both structural and pragmatic considerations in e-measurement. The presence of primary observations in EHR-derived data make it possible to measure outcomes in ways that are difficult with administrative data alone. However, relevant information may be located in narrative text, making it difficult to interpret. EHR systems are collecting more discrete data, but the structure, semantics, and adoption of data elements vary across vendors and sites. EHR systems also differ in ability to incorporate pediatric concepts such as variable dosing and growth percentiles. This variability complicates quality measurement, as do limitations in established measure formats, such as the Quality Data Model, to e-measurement. Addressing these challenges will require investment by vendors, researchers, and clinicians alike in developing better pediatric content for standard terminologies and data models, encouraging wider adoption of technical standards that support reliable quality measurement, better harmonizing data collection with clinical work flow in EHRs, and better understanding the behavior and potential of e-measures.
Interest in reliably measuring the quality of health care in the United States has risen rapidly in response to 2 factors: greater recognition of burdens from medical errors
The Institute of Medicine recommended that the federal government lead by example and enhance the quality of health care it delivers through its programs.
US Institute of Medicine Committee on Enhancing Federal Healthcare Quality Programs. Leadership by Example: Coordinating Government Roles in Improving Health Care Quality.
A number of large-scale federal initiatives (Table 1) have tried to make quality improvement, including use of electronic clinical data, a central aspect of delivering and paying for health care.
Table 1Examples of Key Federal Health IT Quality Measurement and Improvement Initiatives
Legislation
Quality-Related Focus Areas
ARRA—American Recovery and Reinvestment Act of 2009
Supported research into more effective use of electronic clinical information.
•
Agency for Healthcare Research and Quality PROSPECT, SHARP, and Enhanced Registries programs.
•
National Institutes of Health Research and Research Infrastructure grants.
HITECH—Health Information Technology for Economic and Clinical Health Act of 2009
CMS Medicare and Medicaid EHR Incentive Programs.
•
Providers to receive financial incentives to adopt and “meaningfully use” certified EHR technology.
CHIPRA—Children’s Health Insurance Program Reauthorization Act of 2009
Model Children's EHR format.
•
Minimum set of data elements and applicable data standards that can be used as a blueprint for EHR developers.
CHIPRA Quality Demonstration Grant Program.
•
Evaluation of state-level efforts in implementing health IT to improve children's quality of care.
CHIPRA Pediatric Quality Measures Program.
•
Development of new e-measures by the PQMP Centers of Excellence.
PPACA—Patient Protection and Affordable Care Act of 2010
Value-based Purchasing Program and the Physician Quality Reporting System.
•
Integrates existing Physician Quality Reporting System and Medicare/Medicaid EHR Incentive Program into VBP.
•
Payments to providers will be adjusted on the basis of performance on selected EHR-enabled quality measures.
Patient-Centered Outcomes Research Institute
•
Focus on investigation of clinical outcomes with important impact on patients' health.
Early use focused on business transactions; hence, administrative data sets are currently the most well-developed and widely accessible sources of electronic data, and they have been widely used for quality measurement.
Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium.
in: Henriksen K. Battles J.B. Keyes M.A. Grady M.L. Advances in Patient Safety: New Directions and Alternative Approaches. Vol. 1, Assessment. Agency for Healthcare Research and Quality,
Rockville, Md2008
Prognostic models based on administrative data alone inadequately predict the survival outcomes for critically ill patients at 180 days post-hospital discharge.
Profiling individual physicians using administrative data from a single insurer: variance components, reliability, and implications for performance improvement efforts.
The Office of the National Coordinator for Health Information Technology (ONC) is charged with developing a national health information technology (IT) infrastructure, and the HITECH Act provides financial incentives for use of certified electronic health record (EHR) technology, following regulations released in a 3-stage process spanning 2011 to 2016.
These threads converge around the need to more effectively measure quality using electronic clinical information. To this end, ONC has sponsored Beacon Communities to demonstrate effective use of health IT for quality improvement in practice.
outlines ways to incorporate health IT into quality and value assessment, and requires identification of and response to gaps in available quality measures.
Against this public policy background, the medical and research communities have undertaken major efforts to better utilize clinical information systems. The Learning Health System model articulated by the Institute of Medicine
expresses the ethical imperative that health systems uphold patients' trust by committing to continuous improvement through data integration, quality measurement, and clinical decision support. From an economic perspective, the shift of risk to health systems through tying of reimbursement to achieving quality goals provides additional concrete incentive to use available resources to improve care. Accountable Care Organizations (ACO) are a step further down this path, as capitation increases the need for health systems to provide efficient and effective care.
These models imply a fundamental shift in strategy, from reactive minimization of complications to proactive maximization of outcomes. Both the Agency for Healthcare Research and Quality (AHRQ)
support applied research to improve our capacity to make this transition.
Consistent with overall health care utilization, early work has focused on adults, while pediatric quality measurement has evolved more slowly. This reflects not only a limited ability to reuse adult concepts such as mortality-based measures or acute readmissions in pediatrics
Measures for avoidance of unneeded care, more than reduction of morbidity from chronic conditions, play a larger part in pediatric quality assessment. Well-child care addresses preventive measures and normal growth and development, yielding long-term gains by reducing later morbidity. These factors, among others, have sharply limited the utility of claims data for pediatric quality measurement. EHRs' ability to manage pediatric information has also been limited.
For example, conditions such as obesity and hypertension are based on growth-specific thresholds in children; some EHRs cannot use percentiles rather than absolute values. Pediatric concepts are also underrepresented in standard terminologies that underlie electronic clinical information. This has led pediatric groups to create domain-specific vocabularies and has catalyzed formation of the Pediatric Terminologies Project
to systematically review core areas of child health, develop harmonized terminologies, and integrate the results back into community-wide standard terminologies. Nonetheless, defining and measuring quality of care for children has proven difficult.
Responding to these needs, the Children's Health Insurance Program Reauthorization Act (CHIPRA) of 2009 also includes health IT provisions. It calls for development of a “model children's EHR format,” containing child-specific data elements and functionality to support modules tailored to children's health. CHIPRA also established the Quality Demonstration Grant Program, a pediatric state-level analog to ONC's Beacon Communities effort. Finally, it mandated creation of a Pediatric Quality Measures Program (PQMP), whose Centers of Excellence are charged with creating innovative measures of pediatric care.
The Children’s Health Insurance Program Reauthorization Act quality measures initiatives: moving forward to improve measurement, care, and child and adolescent outcomes.
Developing Pediatric Measures Using Electronic Clinical Information
The expansion of machine-readable health information provides new opportunities for quality measurement (Table 2). By increasing computable data, it permits better detection of clinically important population-level variations, through deeper sampling or even direct measurement of entire populations.
Greater ability to collect and share data also enables more timely measure evaluation. The ideal is near-real-time assessment of outcomes, such as disease outbreaks or increased risk of treatment complications associated with other data (eg, pharmacogenomics testing) known about a patient, with results fed back to the point of care to quickly respond to population changes or patient-specific needs. Clinical data sets also allow direct assessment of primary data, such as physiologic or laboratory results, that are absent or only indirectly reflected in administrative data, supporting better outcome- rather than process-based measures.
for better longitudinal measurement of quality. EHRs are not the sole source of such information; clinical registries can provide detailed data focused on a particular condition,
New data domains available in ECI bring new challenges to quality measurement. Chief among these are difficulties in representing clinical concepts. For a measure to scale well, the facts it evaluates must be in computable form. This is best accomplished using structured data with a known value set, such as discrete physical measurements or coded diagnoses. However, EHRs are tailored primarily to clinical work flows; information is often captured as narrative documentation. Differences in writing style, as well as desire to accurately record clinical details or degree of confidence in observations, lead to wide variations in clinicians' expression of a given fact, making automated interpretation difficult. Text mining, via pattern recognition or other natural language processing (NLP), has been both resource-intensive and hard to perform reliably. However, increasing experience and available computing power have yielded better health care–specific NLP systems, such as cTAKES
Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
especially when interpreting parts of the medical record with structural cues and limited variability (eg, diagnostic reports). Although site-specific practice
remain significant barriers, decision support systems incorporating NLP have performed similarly to human review in identifying fractures from CT reports
Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
The alternative to NLP is the shifting of critical facts into structured data. Although the conceptual advantages are clear, several problems arise in practice. Besides clinicians' cultural reliance on narrative, structured documentation can be inhibited by the EHR's user interface
: extensive use of drop-down lists or check boxes can make documentation inefficient and imprecise, creating pressure to develop workarounds. Where structured data entry is deployed, the priority is often adapting the EHR to existing local practice, resulting in vendor- or site-specific data models and vocabularies. This has limited impact on local quality measurement, but it makes it difficult to specify measures that can be shared across sites. It is possible to map local terms to standardized terminologies; although this can be resource intensive and prone to local variations, costs are incurred only once when constructing the mapping, and transparent mapping allows for better understanding and refinement of results. Over time, however, local variations are best addressed by consensus-driven harmonization and adoption of community-wide standard terminologies
Local data models also create direct barriers to cross-site quality measurement. Different systems may capture a particular observation in different data elements, making it difficult to specify measure components. Additionally, an EHR may contain a discrete data element well adapted to the measure's requirements, but in practice, user interface design or local conventions lead to recording of data elsewhere. Addressing these problems will require consideration of measurement as well as clinical needs; although improving documentation templates to be consistent with pediatric guidelines improves clinical flow, if key data are unstructured, automated review remains difficult. Conversely, quality measures based on ECI will need to incorporate some ability to adapt to local circumstances as a trade-off for more detailed clinical information, recognizing that when the 2 uses collide, clinical work flow is likely to outweigh population management in EHR design for the foreseeable future. This is not new; what differs for e-measures is that specifications must be amenable to automated evaluation, whereas traditional measures have been able to rely on the abstractionist to scan the record broadly and provide a level of interpretation between it and the measure.
Once appropriate data elements, value sets, and population criteria have been identified, a formal specification must be constructed to permit reproducible evaluation and validation (Fig. 1). The intent of the National Quality Forum Quality Data Model (QDM) is to support valid, reliable quality comparisons across organizations. It has been refined extensively over time against administrative and abstraction measures, but the breadth of opportunities for new measurement using ECI have exceeded the expressive capacity of the QDM.
Unambiguously expressing some aspects of ECI (eg, distinguishing between primary measurement and clinical interpretation) may be harder to represent using the QDM, while some core QDM concepts (eg, diagnosis active at a particular time) may be harder to detect given the variety of potential expressions the concept may have in the EHR. The QDM's focus on unambiguous expression of a concept may also create tension with unstructured data elements in ECI. Recent revisions to the QDM have increased its capacity to express e-measures, and this remains a focus of great interest.
Figure 1Development cycle for e-measures. MU indicates CMS EHR Incentive Program (Meaningful Use); QDM, National Quality Forum Quality Data Model for specification of data elements, constraints, and measure logic; HQMF, HL7 Consortium Health Quality Measurement Format; and MAT, CMS Measure Authoring Tool, which supports definitions conforming to the QDM, and produces either HQMF or human-readable measure specifications.
In implementing the CHIPRA-mandated Pediatric Quality Measurement Program, Centers for Medicare & Medicaid Services and AHRQ identified e-measure development as a priority. Although the current state of EHR systems constrains development of robust e-measures, current specifications inform ongoing EHR and measure development, as well as seeding e-measure repositories and reporting requirements. PQMP Centers of Excellence have accumulated experience in developing and testing de novo e-measures or respecifications of existing measures designed for chart review, several aspects of which we summarize here.
ADHD Measurement at the Pediatric Measurement Center of Excellence (PMCoE)
Attention-deficit/hyperactivity disorder (ADHD) is a prevalent neurobehavioral condition with demonstrated performance and measurement gaps. The 2011 American Academy of Pediatrics (AAP) ADHD Guidelines
ADHD: clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents.
define a new standard of care for children with ADHD. These provided guidance for measure concepts and specifications developed by a PMCoE Expert Workgroup with broad stakeholder involvement, followed by public review and comment. The quality measure “Accurate Diagnosis of ADHD” was specified as an e-measure for testing by Chicago Pediatric Quality and Safety Consortium (CPQSC) institutions.
A Data Element Table (DET; Table 3) was used to assess the EHR data elements needed for the measure, which was analyzed for feasibility, integrity, and face validity. In 2 settings, paper records are still used for mental health care, including ADHD, so the e-measure was not feasible. Another 2 sites (including a public Cook County institution) had EHRs currently able to discretely capture all data elements needed; another site would be able to implement the e-measure with minor work flow modifications.
Whether or not the data element is captured in the EHR, the data source application, primary user interface data location, data type, coding system, unit of measure, frequency of collection, and calculability within the measure context.
Measure integrity information
An assessment by the testing site as to the degree the measure, as specified, retains the originally stated intention of the measure.
Measure validity information
An assessment by the testing site as to the degree the scores obtained from the measure, as specified, will accurately differentiate quality performance across providers.
Overall feasibility assessment
•
“Feasible. Can do today.”
•
“Feasible with work flow modification/changes to EHR.”
We found these results encouraging, particularly since the publication of the AAP guideline was so recent. The greatest challenge was variability in documentation of ADHD care (Fig. 2A). This suggests future measures may benefit from structured elements for care of a child with ADHD, such as an “ADHD Assessment Method” element with a menu of options, because this information is often in progress notes now.
ADHD: Key Successes and Challenges
•
Integration with clinical care: Adoption of the AAP 2011 guidelines for ADHD in practices using EHRs has been rapid.
•
Data structure and Measure Specification: Challenges centered on variable structured representation of data elements addressing behavioral health, complicating measure specification.
Figure 2Examples of e-measurement. (A) Feasibility of e-measures addressing pediatric ADHD at CPSQC institutions. (B) Location of required data for adolescent preventive health quality measures in the EHR of 1 test site. “Discrete data” indicates search of structured data elements; “templated text,” pattern matching in clinical notes; and “risk factor addressed,” relevant data present in the EHR, whether captured or not by the measure definition. (C) Discordance between primary measurement and diagnosis of obesity in EHRs from 6 pediatric networks.
Adolescent Preventive Health Measurement at the National Collaborative for Innovation in Quality Measurement (NCINQ) Center of Excellence
In 2012, the NCINQ developed and tested quality measures for annual adolescent well care (AWC) visits, such as developmental screening and preventive health services (eg, immunization, chlamydia screening, and tobacco avoidance). During field testing, sites provided information about whether data elements identified by NCINQ as necessary to calculate the measures were captured routinely during clinical care, as well as the feasibility of automatic extraction. We also assessed the validity of measure results by comparing automated with manual record abstraction. We found that EHR systems contain the structured fields needed for most measures but that variations in clinical documentation affect whether or not these structured fields are used (Fig. 2B). In particular, discrete data elements associated with clinical procedures, such as laboratory testing or medication use, were more likely to be present, while information associated with psychosocial screening or counseling was more likely to be embedded in notes or absent entirely. NCINQ's initial work in adolescent depression management has revealed similar findings. NCINQ is following up these findings to better elucidate the impact of work flow on e-measurement and to identify opportunities to revise work flow and EHR systems to achieve improvements in quality.
Adolescent Well Care: Key Successes and Challenges
•
Data types: Availability of standardized behavioral instrument results is limited
•
Data structure: Structured data elements available for most AWC measures
•
Data capture: Use of available structured elements is variable by site and type
Assessing Common Pediatric Conditions at the Children's Hospital of Philadelphia (CHOP) Center of Excellence
Because many pediatric conditions use relatively few health care resources per patient but have significant system-wide impact as a result of high prevalence, the CHOP CoE is examining use of ECI to provide more accurate populations for quality measurement. In a study examining data from 6 institutions' EHRs, anthropometric measurements were highly reliable across a cohort of 528,000 children and yielded accurate population estimates of obesity.
However, there was marked discordance between high body mass index percentile and diagnosis of obesity (Fig. 2C), highlighting the value of primary outcome data in e-measures. The study also demonstrated the feasibility of correlating different clinical outcomes in ECI.
To develop better measures of appropriate treatment for otitis media (OM), we are building on prior EHR-based quality improvement interventions.
Validation against clinical documentation again highlighted the risks of exclusive reliance on diagnostic coding: over 20% of visits with noted ear problems include a normal ear examination, suggesting use of diagnostic codes for “resolved” or “rule out” evaluations. We also note that detailed ECI supports better specificity of avoidance measures, based on better attribution of suspect interventions to alternate, appropriate indications (eg, antibiotic prescribed not for nonacute OM but for pharyngitis). It also allows construction of OM-related episodes of care using several source data types, and consequent measurement across a disease episode rather than a single encounter. The trade-off for these more accurate measures is more complex specifications; current work centers on producing measures using both administrative and clinical data to appropriately balance these considerations. Until improvements in sharing of ECI permit routinely replicating cross-institution measurement from clinical data, we expect that use of complementary measures will be necessary for large-scale assessment of health care delivery.
Common Health Conditions: Key Successes and Challenges
•
Data types: Primary observation of BMI and ear findings are more accurate than diagnoses of these common conditions
•
Population scale: EHR dataset allows detection of rare interactions with obesity
•
Data structure: Significant OM-related assessment information in narrative notes
•
Measure specification: Episode-based measures may be better for measuring aspects of care for chronic conditions
Although we focus here on the PQMP's valuable role as incubator for e-measures against rapidly evolving infrastructure and policy requirements, it is important to note similar activity in other contexts. In addition to work de novo in Beacon Communities, ONC is supporting respecification of a number of pediatric preventive-care measures for EHRs. Regional cooperative groups, such as the Community Care of North Carolina (CCNC) network, which includes more than 95% of that state's primary care practices, provide another example. Using administrative data currently, and wider clinical information as the state HIE develops, CCNC Pediatrics is developing e-measures in priority areas that with elements of the CHIPRA model pediatric EHR format and are consistent with CMS Meaningful Use principles. Their goal is to support real-time review for quality improvement on their complete population, regardless of payer.
Requirements for e-Measurement Optimization
Early experience with e-measurement highlights the potential of new opportunities based on population scale, accuracy, rich clinical data, adaptability to new clinical practices, support for longitudinal evaluation, and close connection to the point of care. However, it has also identified critical limitations in current practice, as vendors, providers, and researchers alike adjust to new systems and learn how to use them to assess quality (Table 4).
Table 4Requirements for Improving Child Health e-Measurement Capacity
Informatics research
•
Refinement of standard terminologies to reflect clinical requirements.
•
More accurate concept recognition in narrative and semistructured data (eg, natural language processing).
•
Better understanding of usability and work flow factors that determine what is and is not well documented in electronic records.
•
Improved translation of quality measurement into practice improvement and decision support.
Health IT policies and vendors
•
Broader interoperability of clinical data.
•
Greater capacity to incorporate pediatric practice into EHRs (eg, variable dosing, growth measurement).
•
Improved usability of EHRs, reducing fragmented and duplicative work flows.
•
More efficient use of structured data elements, especially for developmental, psychosocial, and clinical assessments.
•
Ability to interact with external systems for quality measurement and decision support.
Quality measurement science
•
Identification of measures with greatest potential to improve patients' health.
•
Deeper understanding of relationship between ECI data elements and clinical practice.
•
Expansion of quality measure specifications to incorporate ECI data elements and e-measurement algorithms.
•
Robust integration of data elements and processes for quality measurement into clinical and EHR work flows.
Clinical practice
•
Better integration of decision support tools into patient care.
•
Improved capture of data to support quality measurement.
•
Increasing focus on rapid-cycle change to improve outcomes.
•
Closer linkage between system- and population-level measurement and patient care.
IT indicates information technology; EHR, electronic health record; and ECI, electronic clinical information.
The specific changes needed to make a given measure viable will vary; understanding them will require iterative testing of candidate e-measures and revision of underlying data sources. However, important themes have clearly emerged. Improving our understanding of the semantics, or clinical meaning, and operating characteristics, or use in practice, of ECI must remain a focus of research efforts. These will include construction of better pediatric terminologies, development of new data capture methods, including NLP, and better identification of measures that address outcomes most important for patients' health, rather than those most amenable now to measurement. In parallel, EHR vendors and standards organizations need to make systems better able to manage information needed for pediatric quality measurement. This will require better incorporation of core pediatric practices ranging from variable medication dosing to developmental screening. Improvements in usability are also needed to support more efficient entry and retrieval of structured data; these must be compatible with clinical work flow to enable adoption. In turn, clinicians and health systems must continue to incorporate practices that support and respond to accurate measurement of outcomes without compromising delivery of care. Their input will also be critical to guide the transition to using clinical information systems as knowledge engines rather than simple data repositories.
The mixture of early successes and roadblocks also creates significant challenges for policy makers as well, given the pressing need to improve both quality and cost-effectiveness of the health care enterprise. In the short term, we must focus on balancing integration of EHRs into clinical operations with developing capacity for quality measurement. This will involve making informed choices between meaningful and feasible targets for early measurement, remaining alert to both too-intense focus on current tools, at the risk of suppressing innovation, and too-rapid adoption of new technology, at the risk of creating unintended sources of error. We must also determine where resources should be committed to advance our long-term capability for e-measurement and what standards must be adopted to foster interoperability of data. Evolution of ECI systems will be incremental; therefore, realizing the potential of e-measures will also be a gradual process. The constant goal remains construction of accurate measures capable of regular evaluation against ECI to provide consistent feedback to providers and programs relative to their own previous practice, as well as relative to best practices in the field and for public reporting.
Conflict of Interest
The Children's Hospital of Philadelphia reserves the right to assert commercial rights to software developed as part of prior work by CB for quality improvement related to OM; no current licensing exists, and CB receives no payment derived from this work. The opinions expressed here are those of the authors and do not represent the views of their institutions or the Agency for Healthcare Research and Quality. The authors declare that they have no conflict of interest.
Acknowledgment
The authors are supported by Pediatric Quality Measurement Program Center of Excellence grants from the Agency for Healthcare Research and Quality to the Children's Hospital of Philadelphia (C.B.), the National Collaborative for Innovation in Quality Measurement (A.T.), and the Pediatric Measurement Center of Excellence (M.R., K.H., K.C., M.J, D.W.).
US Institute of Medicine Committee on Enhancing Federal Healthcare Quality Programs. Leadership by Example: Coordinating Government Roles in Improving Health Care Quality.
Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium.
in: Henriksen K. Battles J.B. Keyes M.A. Grady M.L. Advances in Patient Safety: New Directions and Alternative Approaches. Vol. 1, Assessment. Agency for Healthcare Research and Quality,
Rockville, Md2008
Prognostic models based on administrative data alone inadequately predict the survival outcomes for critically ill patients at 180 days post-hospital discharge.
Profiling individual physicians using administrative data from a single insurer: variance components, reliability, and implications for performance improvement efforts.
The Children’s Health Insurance Program Reauthorization Act quality measures initiatives: moving forward to improve measurement, care, and child and adolescent outcomes.
Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
ADHD: clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents.
The authors declare that they have no conflict of interest.
Publication of this article was supported by the US Department of Health and Human Services and the Agency for Healthcare Research and Quality.
The views expressed in this article are those of the authors and do not necessarily represent the views of the US Department of Health and Human Services or the Agency for Healthcare Research and Quality.
00215-0&id=gr1.jpg){kind=link}
00215-0&id=gr2.jpg){kind=link}