Advertisement

Addressing Electronic Clinical Information in the Construction of Quality Measures

      Abstract

      Electronic health records (EHR) and registries play a central role in health care and provide access to detailed clinical information at the individual, institutional, and population level. Use of these data for clinical quality/performance improvement and cost management has been a focus of policy initiatives over the past decade. The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA)-mandated Pediatric Quality Measurement Program supports development and testing of quality measures for children on the basis of electronic clinical information, including de novo measures and respecification of existing measures designed for other data sources. Drawing on the experience of Centers of Excellence, we review both structural and pragmatic considerations in e-measurement. The presence of primary observations in EHR-derived data make it possible to measure outcomes in ways that are difficult with administrative data alone. However, relevant information may be located in narrative text, making it difficult to interpret. EHR systems are collecting more discrete data, but the structure, semantics, and adoption of data elements vary across vendors and sites. EHR systems also differ in ability to incorporate pediatric concepts such as variable dosing and growth percentiles. This variability complicates quality measurement, as do limitations in established measure formats, such as the Quality Data Model, to e-measurement. Addressing these challenges will require investment by vendors, researchers, and clinicians alike in developing better pediatric content for standard terminologies and data models, encouraging wider adoption of technical standards that support reliable quality measurement, better harmonizing data collection with clinical work flow in EHRs, and better understanding the behavior and potential of e-measures.

      Keywords

      Interest in reliably measuring the quality of health care in the United States has risen rapidly in response to 2 factors: greater recognition of burdens from medical errors
      • Kohn L.T.
      • Corrigan J.
      • Donaldson M.S.
      To Err Is Human: Building a Safer Health System.
      and increasing cost of health care.
      • Hillestad R.
      • Bigelow J.
      • Bower A.
      • et al.
      Can electronic medical record systems transform health care? Potential health benefits, savings, and costs.
      The Institute of Medicine recommended that the federal government lead by example and enhance the quality of health care it delivers through its programs.
      • Corrigan J.
      • Eden J.
      • Smith B.M.
      US Institute of Medicine Committee on Enhancing Federal Healthcare Quality Programs. Leadership by Example: Coordinating Government Roles in Improving Health Care Quality.
      A number of large-scale federal initiatives (Table 1) have tried to make quality improvement, including use of electronic clinical data, a central aspect of delivering and paying for health care.
      Table 1Examples of Key Federal Health IT Quality Measurement and Improvement Initiatives
      LegislationQuality-Related Focus Areas
      ARRA—American Recovery and Reinvestment Act of 2009
      • Supported research into more effective use of electronic clinical information.
      • Agency for Healthcare Research and Quality PROSPECT, SHARP, and Enhanced Registries programs.
      • National Institutes of Health Research and Research Infrastructure grants.
      HITECH—Health Information Technology for Economic and Clinical Health Act of 2009
      • CMS Medicare and Medicaid EHR Incentive Programs.
      • Providers to receive financial incentives to adopt and “meaningfully use” certified EHR technology.
      CHIPRA—Children’s Health Insurance Program Reauthorization Act of 2009
      • Model Children's EHR format.
      • Minimum set of data elements and applicable data standards that can be used as a blueprint for EHR developers.
      • CHIPRA Quality Demonstration Grant Program.
      • Evaluation of state-level efforts in implementing health IT to improve children's quality of care.
      • CHIPRA Pediatric Quality Measures Program.
      • Development of new e-measures by the PQMP Centers of Excellence.
      PPACA—Patient Protection and Affordable Care Act of 2010
      • Value-based Purchasing Program and the Physician Quality Reporting System.
      • Integrates existing Physician Quality Reporting System and Medicare/Medicaid EHR Incentive Program into VBP.
      • Payments to providers will be adjusted on the basis of performance on selected EHR-enabled quality measures.
      • Patient-Centered Outcomes Research Institute
      • Focus on investigation of clinical outcomes with important impact on patients' health.
      Similar factors have driven adoption of health information technology, yielding a rapid shift of information into electronic form.
      • Blumenthal D.
      • Glaser J.P.
      Information technology comes to medicine.
      • Conn J.
      Tipping point. Nation’s EHR use tops 50%, stage set for next steps.
      Early use focused on business transactions; hence, administrative data sets are currently the most well-developed and widely accessible sources of electronic data, and they have been widely used for quality measurement.
      • McDonald K.M.
      • Romano P.S.
      • Geppert J.
      • et al.
      Measures of Patient Safety Based on Hospital Administrative Data—The Patient Safety Indicators.
      • Quan H.
      • Drosler S.
      • Sundararajan V.
      • et al.
      Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium.
      • MacTaggart P.
      • Foster A.
      • Markus A.
      Medicaid Statistical Information System (MSIS): a data source for quality reporting for Medicaid and the Children’s Health Insurance Program (CHIP).

      CMS Research Data Assistance Center. Available at: http://www.resdac.org. Accessed July 10, 2014.

      However, limitations of administrative data have been clearly recognized,
      • Bohensky M.A.
      • Jolley D.
      • Pilcher D.V.
      • et al.
      Prognostic models based on administrative data alone inadequately predict the survival outcomes for critically ill patients at 180 days post-hospital discharge.
      • Eijkenaar F.
      • van Vliet R.C.
      Profiling individual physicians using administrative data from a single insurer: variance components, reliability, and implications for performance improvement efforts.
      • Urquhart R.
      • Rayson D.
      • Porter G.A.
      • Grunfeld E.
      Quantifying limitations in chemotherapy data in administrative health databases: implications for measuring the quality of colorectal cancer care.
      • Gidengil C.
      • Mangione-Smith R.
      • Bailey L.C.
      • et al.
      Enhancing Medicaid and CHIP claims data to support pediatric quality measurement: lessons from three centers of excellence in measure development.
      and broader use of health information technologies has been a key goal of government and private initiatives.
      • Blumenthal D.
      • Tavenner M.
      The “meaningful use” regulation for electronic health records.
      The Office of the National Coordinator for Health Information Technology (ONC) is charged with developing a national health information technology (IT) infrastructure, and the HITECH Act provides financial incentives for use of certified electronic health record (EHR) technology, following regulations released in a 3-stage process spanning 2011 to 2016.
      • Blumenthal D.
      • Tavenner M.
      The “meaningful use” regulation for electronic health records.
      These threads converge around the need to more effectively measure quality using electronic clinical information. To this end, ONC has sponsored Beacon Communities to demonstrate effective use of health IT for quality improvement in practice.
      • McKethan A.
      • Brammer C.
      • Fatemi P.
      • et al.
      An early status report on the Beacon Communities’ plans for transformation via health information technology.
      The Patient Protection and Affordable Care Act (PPACA)

      Patient Protection and Affordable Care Act. Public Law 111–148. Available at: http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/html/PLAW-111publ148.htm. Accessed June 23, 2014.

      outlines ways to incorporate health IT into quality and value assessment, and requires identification of and response to gaps in available quality measures.
      Against this public policy background, the medical and research communities have undertaken major efforts to better utilize clinical information systems. The Learning Health System model articulated by the Institute of Medicine
      • Institute of Medicine of the National Academies
      Roundtable on Value and Science-Driven Health Care.
      expresses the ethical imperative that health systems uphold patients' trust by committing to continuous improvement through data integration, quality measurement, and clinical decision support. From an economic perspective, the shift of risk to health systems through tying of reimbursement to achieving quality goals provides additional concrete incentive to use available resources to improve care. Accountable Care Organizations (ACO) are a step further down this path, as capitation increases the need for health systems to provide efficient and effective care.
      • Lewis V.A.
      • Colla C.H.
      • Carluzzo K.L.
      • et al.
      Accountable care organizations in the United States: market and demographic factors associated with formation.
      These models imply a fundamental shift in strategy, from reactive minimization of complications to proactive maximization of outcomes. Both the Agency for Healthcare Research and Quality (AHRQ)
      • Holve E.
      • Segal C.
      • Hamilton Lopez M.
      Opportunities and challenges for comparative effectiveness research (CER) with electronic clinical data: a perspective from the EDM forum.
      and the Patient-Centered Outcomes Research Institute
      • Selby J.V.
      • Beal A.C.
      • Frank L.
      The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.
      support applied research to improve our capacity to make this transition.
      Consistent with overall health care utilization, early work has focused on adults, while pediatric quality measurement has evolved more slowly. This reflects not only a limited ability to reuse adult concepts such as mortality-based measures or acute readmissions in pediatrics
      • Bardach N.S.
      • Vittinghoff E.
      • Asteria-Penaloza R.
      • et al.
      Measuring hospital quality using pediatric readmission and revisit rates.
      (though these remain important measures in specific populations), but also significant differences in quality goals.
      • Mangione-Smith R.
      • Schiff J.
      • Dougherty D.
      Identifying children’s health care quality measures for Medicaid and CHIP: an evidence-informed, publicly transparent expert process.
      Measures for avoidance of unneeded care, more than reduction of morbidity from chronic conditions, play a larger part in pediatric quality assessment. Well-child care addresses preventive measures and normal growth and development, yielding long-term gains by reducing later morbidity. These factors, among others, have sharply limited the utility of claims data for pediatric quality measurement. EHRs' ability to manage pediatric information has also been limited.
      • Gracy D.
      • Weisman J.
      • Grant R.
      • et al.
      Content barriers to pediatric uptake of electronic health records.
      • Leu M.G.
      • O’Connor K.G.
      • Marshall R.
      • et al.
      Pediatricians’ use of health information technology: a national survey.
      For example, conditions such as obesity and hypertension are based on growth-specific thresholds in children; some EHRs cannot use percentiles rather than absolute values. Pediatric concepts are also underrepresented in standard terminologies that underlie electronic clinical information. This has led pediatric groups to create domain-specific vocabularies and has catalyzed formation of the Pediatric Terminologies Project
      • Kahn M.G.
      • Bailey L.C.
      • Forrest C.B.
      • et al.
      Building a common pediatric research terminology for accelerating child health research.
      to systematically review core areas of child health, develop harmonized terminologies, and integrate the results back into community-wide standard terminologies. Nonetheless, defining and measuring quality of care for children has proven difficult.
      Responding to these needs, the Children's Health Insurance Program Reauthorization Act (CHIPRA) of 2009 also includes health IT provisions. It calls for development of a “model children's EHR format,” containing child-specific data elements and functionality to support modules tailored to children's health. CHIPRA also established the Quality Demonstration Grant Program, a pediatric state-level analog to ONC's Beacon Communities effort. Finally, it mandated creation of a Pediatric Quality Measures Program (PQMP), whose Centers of Excellence are charged with creating innovative measures of pediatric care.
      • Dougherty D.
      • Schiff J.
      • Mangione-Smith R.
      The Children’s Health Insurance Program Reauthorization Act quality measures initiatives: moving forward to improve measurement, care, and child and adolescent outcomes.

      Developing Pediatric Measures Using Electronic Clinical Information

      The expansion of machine-readable health information provides new opportunities for quality measurement (Table 2). By increasing computable data, it permits better detection of clinically important population-level variations, through deeper sampling or even direct measurement of entire populations.
      • Bailey L.C.
      • Milov D.E.
      • Kelleher K.
      • et al.
      Multi-institutional sharing of electronic health record data to assess childhood obesity.
      Greater ability to collect and share data also enables more timely measure evaluation. The ideal is near-real-time assessment of outcomes, such as disease outbreaks or increased risk of treatment complications associated with other data (eg, pharmacogenomics testing) known about a patient, with results fed back to the point of care to quickly respond to population changes or patient-specific needs. Clinical data sets also allow direct assessment of primary data, such as physiologic or laboratory results, that are absent or only indirectly reflected in administrative data, supporting better outcome- rather than process-based measures.
      • Klomp H.
      • Dyck R.F.
      • Sidhu N.
      • et al.
      Measuring quality of diabetes care by linking health care system administrative databases with laboratory data.
      Fine-grained clinical information supports more accurate construction of episodes spanning related encounters,
      • Cars T.
      • Wettermark B.
      • Malmstrom R.E.
      • et al.
      Extraction of electronic health record data in a hospital setting: comparison of automatic and semi-automatic methods using anti-TNF therapy as model.
      for better longitudinal measurement of quality. EHRs are not the sole source of such information; clinical registries can provide detailed data focused on a particular condition,
      • Billett A.L.
      • Colletti R.B.
      • Mandel K.E.
      • et al.
      Exemplar pediatric collaborative improvement networks: achieving results.
      while patient-reported outcome systems yield direct measurements of functional status and quality of life.
      • Irwin D.E.
      • Stucky B.D.
      • Thissen D.
      • et al.
      Sampling plan and patient characteristics of the PROMIS pediatrics large-scale survey.
      For convenience in discussion, we refer collectively to information derived from these types of sources as electronic clinical information (ECI).
      Table 2Key Factors Affecting Quality Measurement Based on Electronic Clinical Information
      FactorOpportunitiesChallenges
      Population scale
      • Larger population size than chart abstraction.
      • Broad representation of health system population allows detection of rare variations.
      • Robust automated specifications required.
      • Measures must account for population variation.
      Data types
      • Direct clinical observations (eg, blood pressure).
      • Clinical reasoning directly available.
      • Patient-reported data.
      • Standards less developed for primary clinical observations and patient-reported data.
      • Limited pediatric content of standardized terminologies.
      • Pediatric criteria may be age or growth dependent (eg, dosing, growth percentiles).
      • Semantics of data may differ across data sets.
      Data structure
      • Many structured observations available.
      • Distinction between clinical and billing data apparent.
      • Significant portion of record is unstructured (eg, narrative text).
      • Vendor- and site-specific data models affect generality and comparison of measures.
      Data capture
      • Comprehensive clinical record allows better identification of measure exclusions.
      • Rapid data extraction and exchange possible.
      • Timely reevaluation of measures supports population surveillance or change assessment.
      • Use of structured data elements varies across vendors and sites.
      • Gaps in out-of-network care.
      • EHR user interface design limits opportunities to capture structured data.
      Integration with clinical care
      • Data recorded at point of care.
      • Timeliness and data model provide shorter path to clinical work flows and QI.
      • Potential exists for point-of-care decision support.
      • Clinical workflows reliant on narrative documentation.
      • Limited user capacity to provide separate information for quality measurement.
      Measure specification
      • Potential to take advantage of richer data types and scope than administrative data.
      • Mapping of some constructs in standard measure formats (eg, QDM) to EHR data model unclear.
      • Specification formats undergoing rapid evolution.
      • Limited experience with operating characteristics of e-measures.
      EHR indicates electronic health record; QI, quality improvement; and QDM, National Quality Forum Quality Data Model.
      New data domains available in ECI bring new challenges to quality measurement. Chief among these are difficulties in representing clinical concepts. For a measure to scale well, the facts it evaluates must be in computable form. This is best accomplished using structured data with a known value set, such as discrete physical measurements or coded diagnoses. However, EHRs are tailored primarily to clinical work flows; information is often captured as narrative documentation. Differences in writing style, as well as desire to accurately record clinical details or degree of confidence in observations, lead to wide variations in clinicians' expression of a given fact, making automated interpretation difficult. Text mining, via pattern recognition or other natural language processing (NLP), has been both resource-intensive and hard to perform reliably. However, increasing experience and available computing power have yielded better health care–specific NLP systems, such as cTAKES
      • Savova G.K.
      • Masanz J.J.
      • Ogren P.V.
      • et al.
      Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications.
      and MedLEE,
      • Chiang J.H.
      • Lin J.W.
      • Yang C.W.
      Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
      especially when interpreting parts of the medical record with structural cues and limited variability (eg, diagnostic reports). Although site-specific practice
      • Liu M.
      • Shah A.
      • Jiang M.
      • et al.
      A study of transportability of an existing smoking status detection module across institutions.
      and abbreviation
      • Wu Y.
      • Denny J.C.
      • Rosenbloom S.T.
      • et al.
      A comparative study of current Clinical Natural Language Processing systems on handling abbreviations in discharge summaries.
      remain significant barriers, decision support systems incorporating NLP have performed similarly to human review in identifying fractures from CT reports
      • Yadav K.
      • Sarioglu E.
      • Smith M.
      • Choi H.A.
      Automated outcome classification of emergency department computed tomography imaging reports.
      and asthma in young children,
      • Wu S.T.
      • Sohn S.
      • Ravikumar K.E.
      • et al.
      Automated chart review for asthma cohort identification using natural language processing: an exploratory study.
      and pilot quality measures based on NLP have shown promise.
      • Chiang J.H.
      • Lin J.W.
      • Yang C.W.
      Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
      • Pathak J.
      • Bailey K.R.
      • Beebe C.E.
      • et al.
      Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium.
      The alternative to NLP is the shifting of critical facts into structured data. Although the conceptual advantages are clear, several problems arise in practice. Besides clinicians' cultural reliance on narrative, structured documentation can be inhibited by the EHR's user interface
      • Ahmed A.
      • Chandra S.
      • Herasevich V.
      • et al.
      The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance.
      : extensive use of drop-down lists or check boxes can make documentation inefficient and imprecise, creating pressure to develop workarounds. Where structured data entry is deployed, the priority is often adapting the EHR to existing local practice, resulting in vendor- or site-specific data models and vocabularies. This has limited impact on local quality measurement, but it makes it difficult to specify measures that can be shared across sites. It is possible to map local terms to standardized terminologies; although this can be resource intensive and prone to local variations, costs are incurred only once when constructing the mapping, and transparent mapping allows for better understanding and refinement of results. Over time, however, local variations are best addressed by consensus-driven harmonization and adoption of community-wide standard terminologies
      • Kahn M.G.
      • Bailey L.C.
      • Forrest C.B.
      • et al.
      Building a common pediatric research terminology for accelerating child health research.
      and common data elements.
      Local data models also create direct barriers to cross-site quality measurement. Different systems may capture a particular observation in different data elements, making it difficult to specify measure components. Additionally, an EHR may contain a discrete data element well adapted to the measure's requirements, but in practice, user interface design or local conventions lead to recording of data elsewhere. Addressing these problems will require consideration of measurement as well as clinical needs; although improving documentation templates to be consistent with pediatric guidelines improves clinical flow, if key data are unstructured, automated review remains difficult. Conversely, quality measures based on ECI will need to incorporate some ability to adapt to local circumstances as a trade-off for more detailed clinical information, recognizing that when the 2 uses collide, clinical work flow is likely to outweigh population management in EHR design for the foreseeable future. This is not new; what differs for e-measures is that specifications must be amenable to automated evaluation, whereas traditional measures have been able to rely on the abstractionist to scan the record broadly and provide a level of interpretation between it and the measure.
      Once appropriate data elements, value sets, and population criteria have been identified, a formal specification must be constructed to permit reproducible evaluation and validation (Fig. 1). The intent of the National Quality Forum Quality Data Model (QDM) is to support valid, reliable quality comparisons across organizations. It has been refined extensively over time against administrative and abstraction measures, but the breadth of opportunities for new measurement using ECI have exceeded the expressive capacity of the QDM.
      • Casciato A.
      • Angier H.
      • Milano C.
      • et al.
      Are pediatric quality care measures too stringent?.
      Unambiguously expressing some aspects of ECI (eg, distinguishing between primary measurement and clinical interpretation) may be harder to represent using the QDM, while some core QDM concepts (eg, diagnosis active at a particular time) may be harder to detect given the variety of potential expressions the concept may have in the EHR. The QDM's focus on unambiguous expression of a concept may also create tension with unstructured data elements in ECI. Recent revisions to the QDM have increased its capacity to express e-measures, and this remains a focus of great interest.
      Figure thumbnail gr1
      Figure 1Development cycle for e-measures. MU indicates CMS EHR Incentive Program (Meaningful Use); QDM, National Quality Forum Quality Data Model for specification of data elements, constraints, and measure logic; HQMF, HL7 Consortium Health Quality Measurement Format; and MAT, CMS Measure Authoring Tool, which supports definitions conforming to the QDM, and produces either HQMF or human-readable measure specifications.

      Early Experience With e-Measures in PQMP

      In implementing the CHIPRA-mandated Pediatric Quality Measurement Program, Centers for Medicare & Medicaid Services and AHRQ identified e-measure development as a priority. Although the current state of EHR systems constrains development of robust e-measures, current specifications inform ongoing EHR and measure development, as well as seeding e-measure repositories and reporting requirements. PQMP Centers of Excellence have accumulated experience in developing and testing de novo e-measures or respecifications of existing measures designed for chart review, several aspects of which we summarize here.

       ADHD Measurement at the Pediatric Measurement Center of Excellence (PMCoE)

      Attention-deficit/hyperactivity disorder (ADHD) is a prevalent neurobehavioral condition with demonstrated performance and measurement gaps. The 2011 American Academy of Pediatrics (AAP) ADHD Guidelines
      • Wolraich M.
      • Brown L.
      • Brown R.T.
      • et al.
      ADHD: clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents.
      define a new standard of care for children with ADHD. These provided guidance for measure concepts and specifications developed by a PMCoE Expert Workgroup with broad stakeholder involvement, followed by public review and comment. The quality measure “Accurate Diagnosis of ADHD” was specified as an e-measure for testing by Chicago Pediatric Quality and Safety Consortium (CPQSC) institutions.
      • Woods D.M.
      • Holl J.L.
      • Angst D.B.
      • et al.
      Gaps in pediatric clinician communication and opportunities for improvement.
      A Data Element Table (DET; Table 3) was used to assess the EHR data elements needed for the measure, which was analyzed for feasibility, integrity, and face validity. In 2 settings, paper records are still used for mental health care, including ADHD, so the e-measure was not feasible. Another 2 sites (including a public Cook County institution) had EHRs currently able to discretely capture all data elements needed; another site would be able to implement the e-measure with minor work flow modifications.
      Table 3Data Element Table (DET) Structure
      Content based on Data Element Table Tool, © 2012 American Medical Association.
      MeasureDescription
      Data element informationWhether or not the data element is captured in the EHR, the data source application, primary user interface data location, data type, coding system, unit of measure, frequency of collection, and calculability within the measure context.
      Measure integrity informationAn assessment by the testing site as to the degree the measure, as specified, retains the originally stated intention of the measure.
      Measure validity informationAn assessment by the testing site as to the degree the scores obtained from the measure, as specified, will accurately differentiate quality performance across providers.
      Overall feasibility assessment
      • “Feasible. Can do today.”
      • “Feasible with work flow modification/changes to EHR.”
      • “Not feasible. Unable to do today.”
      EHR indicates electronic health record.
      Content based on Data Element Table Tool, © 2012 American Medical Association.
      We found these results encouraging, particularly since the publication of the AAP guideline was so recent. The greatest challenge was variability in documentation of ADHD care (Fig. 2A). This suggests future measures may benefit from structured elements for care of a child with ADHD, such as an “ADHD Assessment Method” element with a menu of options, because this information is often in progress notes now.
      ADHD: Key Successes and Challenges
      • Integration with clinical care: Adoption of the AAP 2011 guidelines for ADHD in practices using EHRs has been rapid.
      • Data structure and Measure Specification: Challenges centered on variable structured representation of data elements addressing behavioral health, complicating measure specification.
      Figure thumbnail gr2
      Figure 2Examples of e-measurement. (A) Feasibility of e-measures addressing pediatric ADHD at CPSQC institutions. (B) Location of required data for adolescent preventive health quality measures in the EHR of 1 test site. “Discrete data” indicates search of structured data elements; “templated text,” pattern matching in clinical notes; and “risk factor addressed,” relevant data present in the EHR, whether captured or not by the measure definition. (C) Discordance between primary measurement and diagnosis of obesity in EHRs from 6 pediatric networks.

       Adolescent Preventive Health Measurement at the National Collaborative for Innovation in Quality Measurement (NCINQ) Center of Excellence

      In 2012, the NCINQ developed and tested quality measures for annual adolescent well care (AWC) visits, such as developmental screening and preventive health services (eg, immunization, chlamydia screening, and tobacco avoidance). During field testing, sites provided information about whether data elements identified by NCINQ as necessary to calculate the measures were captured routinely during clinical care, as well as the feasibility of automatic extraction. We also assessed the validity of measure results by comparing automated with manual record abstraction. We found that EHR systems contain the structured fields needed for most measures but that variations in clinical documentation affect whether or not these structured fields are used (Fig. 2B). In particular, discrete data elements associated with clinical procedures, such as laboratory testing or medication use, were more likely to be present, while information associated with psychosocial screening or counseling was more likely to be embedded in notes or absent entirely. NCINQ's initial work in adolescent depression management has revealed similar findings. NCINQ is following up these findings to better elucidate the impact of work flow on e-measurement and to identify opportunities to revise work flow and EHR systems to achieve improvements in quality.
      Adolescent Well Care: Key Successes and Challenges
      • Data types: Availability of standardized behavioral instrument results is limited
      • Data structure: Structured data elements available for most AWC measures
      • Data capture: Use of available structured elements is variable by site and type

       Assessing Common Pediatric Conditions at the Children's Hospital of Philadelphia (CHOP) Center of Excellence

      Because many pediatric conditions use relatively few health care resources per patient but have significant system-wide impact as a result of high prevalence, the CHOP CoE is examining use of ECI to provide more accurate populations for quality measurement. In a study examining data from 6 institutions' EHRs, anthropometric measurements were highly reliable across a cohort of 528,000 children and yielded accurate population estimates of obesity.
      • Bailey L.C.
      • Milov D.E.
      • Kelleher K.
      • et al.
      Multi-institutional sharing of electronic health record data to assess childhood obesity.
      However, there was marked discordance between high body mass index percentile and diagnosis of obesity (Fig. 2C), highlighting the value of primary outcome data in e-measures. The study also demonstrated the feasibility of correlating different clinical outcomes in ECI.
      To develop better measures of appropriate treatment for otitis media (OM), we are building on prior EHR-based quality improvement interventions.
      • Forrest C.B.
      • Fiks A.G.
      • Bailey L.C.
      • et al.
      Improving adherence to otitis media guidelines with clinical decision support and physician feedback.
      Validation against clinical documentation again highlighted the risks of exclusive reliance on diagnostic coding: over 20% of visits with noted ear problems include a normal ear examination, suggesting use of diagnostic codes for “resolved” or “rule out” evaluations. We also note that detailed ECI supports better specificity of avoidance measures, based on better attribution of suspect interventions to alternate, appropriate indications (eg, antibiotic prescribed not for nonacute OM but for pharyngitis). It also allows construction of OM-related episodes of care using several source data types, and consequent measurement across a disease episode rather than a single encounter. The trade-off for these more accurate measures is more complex specifications; current work centers on producing measures using both administrative and clinical data to appropriately balance these considerations. Until improvements in sharing of ECI permit routinely replicating cross-institution measurement from clinical data, we expect that use of complementary measures will be necessary for large-scale assessment of health care delivery.
      Common Health Conditions: Key Successes and Challenges
      • Data types: Primary observation of BMI and ear findings are more accurate than diagnoses of these common conditions
      • Population scale: EHR dataset allows detection of rare interactions with obesity
      • Data structure: Significant OM-related assessment information in narrative notes
      • Measure specification: Episode-based measures may be better for measuring aspects of care for chronic conditions
      Although we focus here on the PQMP's valuable role as incubator for e-measures against rapidly evolving infrastructure and policy requirements, it is important to note similar activity in other contexts. In addition to work de novo in Beacon Communities, ONC is supporting respecification of a number of pediatric preventive-care measures for EHRs. Regional cooperative groups, such as the Community Care of North Carolina (CCNC) network, which includes more than 95% of that state's primary care practices, provide another example. Using administrative data currently, and wider clinical information as the state HIE develops, CCNC Pediatrics is developing e-measures in priority areas that with elements of the CHIPRA model pediatric EHR format and are consistent with CMS Meaningful Use principles. Their goal is to support real-time review for quality improvement on their complete population, regardless of payer.

      Requirements for e-Measurement Optimization

      Early experience with e-measurement highlights the potential of new opportunities based on population scale, accuracy, rich clinical data, adaptability to new clinical practices, support for longitudinal evaluation, and close connection to the point of care. However, it has also identified critical limitations in current practice, as vendors, providers, and researchers alike adjust to new systems and learn how to use them to assess quality (Table 4).
      Table 4Requirements for Improving Child Health e-Measurement Capacity
      • Informatics research
        • Refinement of standard terminologies to reflect clinical requirements.
        • More accurate concept recognition in narrative and semistructured data (eg, natural language processing).
        • Better understanding of usability and work flow factors that determine what is and is not well documented in electronic records.
        • Improved translation of quality measurement into practice improvement and decision support.
      • Health IT policies and vendors
        • Broader interoperability of clinical data.
        • Greater capacity to incorporate pediatric practice into EHRs (eg, variable dosing, growth measurement).
        • Improved usability of EHRs, reducing fragmented and duplicative work flows.
        • More efficient use of structured data elements, especially for developmental, psychosocial, and clinical assessments.
        • Ability to interact with external systems for quality measurement and decision support.
      • Quality measurement science
        • Identification of measures with greatest potential to improve patients' health.
        • Deeper understanding of relationship between ECI data elements and clinical practice.
        • Expansion of quality measure specifications to incorporate ECI data elements and e-measurement algorithms.
        • Robust integration of data elements and processes for quality measurement into clinical and EHR work flows.
      • Clinical practice
        • Better integration of decision support tools into patient care.
        • Improved capture of data to support quality measurement.
        • Increasing focus on rapid-cycle change to improve outcomes.
        • Closer linkage between system- and population-level measurement and patient care.
      IT indicates information technology; EHR, electronic health record; and ECI, electronic clinical information.
      The specific changes needed to make a given measure viable will vary; understanding them will require iterative testing of candidate e-measures and revision of underlying data sources. However, important themes have clearly emerged. Improving our understanding of the semantics, or clinical meaning, and operating characteristics, or use in practice, of ECI must remain a focus of research efforts. These will include construction of better pediatric terminologies, development of new data capture methods, including NLP, and better identification of measures that address outcomes most important for patients' health, rather than those most amenable now to measurement. In parallel, EHR vendors and standards organizations need to make systems better able to manage information needed for pediatric quality measurement. This will require better incorporation of core pediatric practices ranging from variable medication dosing to developmental screening. Improvements in usability are also needed to support more efficient entry and retrieval of structured data; these must be compatible with clinical work flow to enable adoption. In turn, clinicians and health systems must continue to incorporate practices that support and respond to accurate measurement of outcomes without compromising delivery of care. Their input will also be critical to guide the transition to using clinical information systems as knowledge engines rather than simple data repositories.
      The mixture of early successes and roadblocks also creates significant challenges for policy makers as well, given the pressing need to improve both quality and cost-effectiveness of the health care enterprise. In the short term, we must focus on balancing integration of EHRs into clinical operations with developing capacity for quality measurement. This will involve making informed choices between meaningful and feasible targets for early measurement, remaining alert to both too-intense focus on current tools, at the risk of suppressing innovation, and too-rapid adoption of new technology, at the risk of creating unintended sources of error. We must also determine where resources should be committed to advance our long-term capability for e-measurement and what standards must be adopted to foster interoperability of data. Evolution of ECI systems will be incremental; therefore, realizing the potential of e-measures will also be a gradual process. The constant goal remains construction of accurate measures capable of regular evaluation against ECI to provide consistent feedback to providers and programs relative to their own previous practice, as well as relative to best practices in the field and for public reporting.

      Conflict of Interest

      The Children's Hospital of Philadelphia reserves the right to assert commercial rights to software developed as part of prior work by CB for quality improvement related to OM; no current licensing exists, and CB receives no payment derived from this work. The opinions expressed here are those of the authors and do not represent the views of their institutions or the Agency for Healthcare Research and Quality. The authors declare that they have no conflict of interest.

      Acknowledgment

      The authors are supported by Pediatric Quality Measurement Program Center of Excellence grants from the Agency for Healthcare Research and Quality to the Children's Hospital of Philadelphia (C.B.), the National Collaborative for Innovation in Quality Measurement (A.T.), and the Pediatric Measurement Center of Excellence (M.R., K.H., K.C., M.J, D.W.).

      References

        • Kohn L.T.
        • Corrigan J.
        • Donaldson M.S.
        To Err Is Human: Building a Safer Health System.
        National Academies Press, Washington, DC2000
        • Hillestad R.
        • Bigelow J.
        • Bower A.
        • et al.
        Can electronic medical record systems transform health care? Potential health benefits, savings, and costs.
        Health Aff (Millwood). 2005; 24: 1103-1117
        • Corrigan J.
        • Eden J.
        • Smith B.M.
        US Institute of Medicine Committee on Enhancing Federal Healthcare Quality Programs. Leadership by Example: Coordinating Government Roles in Improving Health Care Quality.
        National Academies Press, Washington, DC2003
        • Blumenthal D.
        • Glaser J.P.
        Information technology comes to medicine.
        N Engl J Med. 2007; 356: 2527-2534
        • Conn J.
        Tipping point. Nation’s EHR use tops 50%, stage set for next steps.
        Mod Healthc. 2013; 43: 14-15
        • McDonald K.M.
        • Romano P.S.
        • Geppert J.
        • et al.
        Measures of Patient Safety Based on Hospital Administrative Data—The Patient Safety Indicators.
        Agency for Healthcare Research and Quality, Rockville, Md2002
        • Quan H.
        • Drosler S.
        • Sundararajan V.
        • et al.
        Adaptation of AHRQ patient safety indicators for use in ICD-10 administrative data by an international consortium.
        in: Henriksen K. Battles J.B. Keyes M.A. Grady M.L. Advances in Patient Safety: New Directions and Alternative Approaches. Vol. 1, Assessment. Agency for Healthcare Research and Quality, Rockville, Md2008
        • MacTaggart P.
        • Foster A.
        • Markus A.
        Medicaid Statistical Information System (MSIS): a data source for quality reporting for Medicaid and the Children’s Health Insurance Program (CHIP).
        Perspect Health Inf Manag. 2011; 8: 1d
      1. CMS Research Data Assistance Center. Available at: http://www.resdac.org. Accessed July 10, 2014.

        • Bohensky M.A.
        • Jolley D.
        • Pilcher D.V.
        • et al.
        Prognostic models based on administrative data alone inadequately predict the survival outcomes for critically ill patients at 180 days post-hospital discharge.
        J Crit Care. 2012; 27: 422.e411-422.e421
        • Eijkenaar F.
        • van Vliet R.C.
        Profiling individual physicians using administrative data from a single insurer: variance components, reliability, and implications for performance improvement efforts.
        Med Care. 2013; 51: 731-739
        • Urquhart R.
        • Rayson D.
        • Porter G.A.
        • Grunfeld E.
        Quantifying limitations in chemotherapy data in administrative health databases: implications for measuring the quality of colorectal cancer care.
        Healthc Policy. 2011; 7: 32-40
        • Gidengil C.
        • Mangione-Smith R.
        • Bailey L.C.
        • et al.
        Enhancing Medicaid and CHIP claims data to support pediatric quality measurement: lessons from three centers of excellence in measure development.
        Acad Pediatr. 2014; 14: S76-S81
        • Blumenthal D.
        • Tavenner M.
        The “meaningful use” regulation for electronic health records.
        N Engl J Med. 2010; 363: 501-504
        • McKethan A.
        • Brammer C.
        • Fatemi P.
        • et al.
        An early status report on the Beacon Communities’ plans for transformation via health information technology.
        Health Aff (Millwood). 2011; 30: 782-788
      2. Patient Protection and Affordable Care Act. Public Law 111–148. Available at: http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/html/PLAW-111publ148.htm. Accessed June 23, 2014.

        • Institute of Medicine of the National Academies
        Roundtable on Value and Science-Driven Health Care.
        The Learning Health System and Its Innovation Collaboratives: Update Report. 2011; (Available at:) (Accessed June 23, 2014)
        • Lewis V.A.
        • Colla C.H.
        • Carluzzo K.L.
        • et al.
        Accountable care organizations in the United States: market and demographic factors associated with formation.
        Health Serv Res. 2013; 48: 1840-1858
        • Holve E.
        • Segal C.
        • Hamilton Lopez M.
        Opportunities and challenges for comparative effectiveness research (CER) with electronic clinical data: a perspective from the EDM forum.
        Med Care. 2012; 50: S11-S18
        • Selby J.V.
        • Beal A.C.
        • Frank L.
        The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda.
        JAMA. 2012; 307: 1583-1584
        • Bardach N.S.
        • Vittinghoff E.
        • Asteria-Penaloza R.
        • et al.
        Measuring hospital quality using pediatric readmission and revisit rates.
        Pediatrics. 2013; 132: 429-436
        • Mangione-Smith R.
        • Schiff J.
        • Dougherty D.
        Identifying children’s health care quality measures for Medicaid and CHIP: an evidence-informed, publicly transparent expert process.
        Acad Pediatr. 2011; 11: S11-S21
        • Gracy D.
        • Weisman J.
        • Grant R.
        • et al.
        Content barriers to pediatric uptake of electronic health records.
        Adv Pediatr. 2012; 59: 159-181
        • Leu M.G.
        • O’Connor K.G.
        • Marshall R.
        • et al.
        Pediatricians’ use of health information technology: a national survey.
        Pediatrics. 2012; 130: e1441-e1446
        • Kahn M.G.
        • Bailey L.C.
        • Forrest C.B.
        • et al.
        Building a common pediatric research terminology for accelerating child health research.
        Pediatrics. 2014; 133: 516-525
        • Dougherty D.
        • Schiff J.
        • Mangione-Smith R.
        The Children’s Health Insurance Program Reauthorization Act quality measures initiatives: moving forward to improve measurement, care, and child and adolescent outcomes.
        Acad Pediatr. 2011; 11: S1-S10
        • Bailey L.C.
        • Milov D.E.
        • Kelleher K.
        • et al.
        Multi-institutional sharing of electronic health record data to assess childhood obesity.
        PLoS One. 2013; 8: e66192
        • Klomp H.
        • Dyck R.F.
        • Sidhu N.
        • et al.
        Measuring quality of diabetes care by linking health care system administrative databases with laboratory data.
        BMC Res Notes. 2010; 3: 233
        • Cars T.
        • Wettermark B.
        • Malmstrom R.E.
        • et al.
        Extraction of electronic health record data in a hospital setting: comparison of automatic and semi-automatic methods using anti-TNF therapy as model.
        Basic Clin Pharmacol Toxicol. 2013; 112: 392-400
        • Billett A.L.
        • Colletti R.B.
        • Mandel K.E.
        • et al.
        Exemplar pediatric collaborative improvement networks: achieving results.
        Pediatrics. 2013; 131: S196-S203
        • Irwin D.E.
        • Stucky B.D.
        • Thissen D.
        • et al.
        Sampling plan and patient characteristics of the PROMIS pediatrics large-scale survey.
        Qual Life Res. 2010; 19: 585-594
        • Savova G.K.
        • Masanz J.J.
        • Ogren P.V.
        • et al.
        Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications.
        J Am Med Inform Assoc. 2010; 17: 507-513
        • Chiang J.H.
        • Lin J.W.
        • Yang C.W.
        Automated evaluation of electronic discharge notes to assess quality of care for cardiovascular diseases using Medical Language Extraction and Encoding System (MedLEE).
        J Am Med Inform Assoc. 2010; 17: 245-252
        • Liu M.
        • Shah A.
        • Jiang M.
        • et al.
        A study of transportability of an existing smoking status detection module across institutions.
        AMIA Annu Symp Proc. 2012; 2012: 577-586
        • Wu Y.
        • Denny J.C.
        • Rosenbloom S.T.
        • et al.
        A comparative study of current Clinical Natural Language Processing systems on handling abbreviations in discharge summaries.
        AMIA Annu Symp Proc. 2012; 2012: 997-1003
        • Yadav K.
        • Sarioglu E.
        • Smith M.
        • Choi H.A.
        Automated outcome classification of emergency department computed tomography imaging reports.
        Acad Emerg Med. 2013; 20: 848-854
        • Wu S.T.
        • Sohn S.
        • Ravikumar K.E.
        • et al.
        Automated chart review for asthma cohort identification using natural language processing: an exploratory study.
        Ann Allergy Asthma Immunol. 2013; 111: 364-369
        • Pathak J.
        • Bailey K.R.
        • Beebe C.E.
        • et al.
        Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium.
        J Am Med Inform Assoc. 2013; 20: e341-e348
        • Ahmed A.
        • Chandra S.
        • Herasevich V.
        • et al.
        The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance.
        Crit Care Med. 2011; 39: 1626-1634
        • Casciato A.
        • Angier H.
        • Milano C.
        • et al.
        Are pediatric quality care measures too stringent?.
        J Am Board Fam Med. 2012; 25: 686-693
        • Wolraich M.
        • Brown L.
        • Brown R.T.
        • et al.
        ADHD: clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents.
        Pediatrics. 2011; 128: 1007-1022
        • Woods D.M.
        • Holl J.L.
        • Angst D.B.
        • et al.
        Gaps in pediatric clinician communication and opportunities for improvement.
        J Healthc Qual. 2008; 30: 43-54
        • Forrest C.B.
        • Fiks A.G.
        • Bailey L.C.
        • et al.
        Improving adherence to otitis media guidelines with clinical decision support and physician feedback.
        Pediatrics. 2013; 131: e1071-e1081