Advertisement

Adjusting for Social Risk Factors in Pediatric Quality Measures: Adding to the Evidence Base

      Abstract

      Background

      Outcome and utilization quality measures are adjusted for patient case-mix including demographic characteristics and comorbid conditions to allow for comparisons between hospitals and health plans. However, controversy exists around whether and how to adjust for social risk factors.

      Objective

      To assess an approach to incorporating social risk variables into a pediatric measure of utilization from the Pediatric Quality Measures Program (PQMP).

      Methods

      We used data from California Medicaid claims (2015–16) and Massachusetts All Payer Claims Database (2014–2015) to calculate health plan performance using measure specifications from the Pediatric Asthma Emergency Department Use measure. Health plan performance categories were assessed using mixed effect negative binomial models with and without adjustment for social risk factors, with both models adjusting for age, gender and chronic condition category. Mixed effects linear models were then used to compare patient social risk for health plans that changed performance categories to patient social risk for health plans that did not.

      Results

      Of 133 health plans, serving 404,649 pediatric patients with asthma, 7% to 13% changed performance categories after social risk adjustment. Health plans that moved to higher performance categories cared for lower socioeconomic status (SES) patients whereas those that moved to lower performance categories cared for higher SES patients.

      Conclusions

      Adjustment for social risk factors changed performance rankings on the PQMP Pediatric Asthma Emergency Department Use measure for a substantial number of health plans. Some health plans caring for higher risk patients performed more poorly when social risk factors were not included in risk adjustment models. In light of this, social risk factors are incorporated into the National Quality Forum-endorsed measure; whether to incorporate social risk factors into pediatric quality measures will differ depending on the use case.

      Keywords

      What's New
      Little is known about including social risk factors in case-mix adjustment models for pediatric quality measures. Using the Asthma Emergency Department Use measure as an example, we discuss the merits, feasibility, and challenges of incorporating these factors into pediatric quality assessment.
      With the shift toward value-based care, quality measurement has become an essential component of American healthcare. Since 2011, over 160 quality measures have been developed through the Pediatric Quality Measures Program (PQMP) by the Centers of Excellence, many of which have been endorsed by the National Quality Forum (NQF).

      Pediatric Quality Measures Program. Agency for Healthcare Research and Quality. 2018. Available at: https://www.ahrq.gov/pqmp/index.html Accessed February 23, 2021.

      ,
      • Adirim T
      • Meade K
      • Mistry K.
      A new era in quality measurement: the development and application of quality measures.
      To facilitate unbiased performance comparisons, quality measures assessing outcomes (eg, mortality, readmissions) and utilization commonly risk-adjust for patient demographic and clinical characteristics such as age, sex, comorbidities, and severity of underlying illness. Historically, measure developers have avoided including social risk factors in risk-adjustment approaches to avoid potentially setting lower quality standards for hospitals or providers that disproportionately care for poorer patients.
      • Krumholz HM
      • Bernheim SM.
      Considering the role of socioeconomic status in hospital outcomes measures.
      However, more recent studies have raised concerns that patients cared for by these hospitals may be more likely to have poorer outcomes due to social risk factors rather than poor-quality healthcare. Hence, there has been an active public debate about whether, how, and when to adjust for social risk.
      This debate has been most prominent surrounding the Medicare Hospital Readmission Reduction Program in adults.

      Hospital Readmissions Reduction Program (HRRP). Centers for Medicare and Medicaid services. 2020. Available at: https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program Accessed February 23, 2021.

      Proponents of social risk adjustment have argued that financial penalties could lead to fewer resources allocated to hospitals that disproportionately treat patients with high social risk, thereby exacerbating existing disparities and discouraging providers from caring for these patients.
      • Joynt KE
      • Jha AK.
      Thirty-day readmissions—truth and consequences.
      • Hernandez AF
      • Curtis LH.
      Minding the gap between efforts to reduce readmissions and disparities.
      • Bhalla R
      • Kalkut G.
      Could Medicare readmission policy exacerbate health care system inequity?.
      • Gu Q
      • Koenig L
      • Faerberg J
      • et al.
      The Medicare Hospital Readmissions Reduction Program: potential unintended consequences for hospitals serving vulnerable populations.
      These concerns prompted legislative proposals to change Medicare pay-for-performance programs to include social risk factors in risk-adjustment models,

      Improving Medicare post-acute care transformation act of 2014. Public Law No. 113-185. 2014. Available at: https://www.govinfo.gov/content/pkg/PLAW-113publ185/html/PLAW-113publ185.htm. Accessed February 4, 2020.

      and in 2014, an NQF Expert Panel concluded that lack of adjustment for social risk factors might worsen health disparities.
      Risk Adjustment for Socioeconomic Status or Other Socioeconomic Factors.
      Since then, the NQF has published a framework for determining when and how to adjust for social risk more broadly in quality measurement,
      Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
      and several studies in adults have investigated how hospital or health plan performance change with social risk adjustment.
      • Bernheim SM
      • Parzynski CS
      • Horwitz L
      • et al.
      Accounting for patients' socioeconomic status does not change hospital readmission rates.
      • Joynt Maddox KE
      • Reidhead M
      • Hu J
      • et al.
      Adjusting for social risk factors impacts performance and penalties in the Hospital Readmissions Reduction Program.
      • Gaskin DJ
      • Zare H
      • Vazin R
      • et al.
      Racial and ethnic composition of hospitals’ service areas and the likelihood of being penalized for excess readmissions by the Medicare program.
      • Durfey SNM
      • Kind AJH
      • Gutman R
      • et al.
      Impact of risk adjustment for socioeconomic status on Medicare Advantage plan quality rankings.
      • Glance LG
      • Kellermann AL
      • Osler TM
      • et al.
      Impact of risk adjustment for socioeconomic status on risk-adjusted surgical readmission rates.
      • Sandhu AT
      • Bhattacharya J
      • Lam J
      • et al.
      Adjustment for social risk factors does not meaningfully affect performance on Medicare's MIPS clinician cost measures.
      • Nguyen CA
      • Gilstrap LG
      • Chernew ME
      • et al.
      Social risk adjustment of quality measures for diabetes and cardiovascular disease in a commercially insured US population.
      Relatively fewer studies have studied the implications of social risk adjustment in pediatric quality measures.
      • Sills MR
      • Hall M
      • Colvin JD
      • et al.
      Association of social determinants with children's hospitals’ preventable readmissions performance.
      ,
      • Sills MR
      • Hall M
      • Cutler GJ
      • et al.
      Adding social determinant data changes children's hospitals’ readmissions performance.
      Key differences between pediatric and adult patients are worthy of attention. Nearly 21% of children live in poverty,
      • Betson DM
      • Michael RT.
      Why so many children are poor.
      leading to a high proportion of children insured by Medicaid/CHIP nationally. However, the percentage of Medicaid-insured children varies widely across states (15%–61%),

      Health insurance coverage of children 0-18. Kaiser Family Foundation. 2019. Available at:https://www.kff.org/other/state-indicator/children-0-18/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D. Accessed February 23, 2021.

      in part due to federal regulations granting substantial discretion to states regarding enrollment criteria. State differences in pediatric Medicaid enrollment may lead to variable validity across states on measures that include social risk factor adjustment. In addition, because children have a social safety net in the form of their parents, parental protection and prioritization of their child's health may blunt the effects of social risk on health outcomes.
      This paper explores the utility of incorporating social risk factors into risk adjustment for the Pediatric Asthma Emergency Department (ED) Use measure.

      IMPLEMENT for Child Health. The regents of the University of California. 2011. Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed on March 27, 2021.

      ,

      Rate of emergency department visit use for children managed for persistent asthma: detailed measure specifications. Collaboration for the advancement of pediatric quality measures (CAPQuaM). 2020. Available at: http://www.capquam.org/. Accessed June 14, 2021.

      Asthma is the most common pediatric chronic disease, affecting 9% of school-aged children, and the third most common of pediatric hospitalizations in the United States.

      Centers for Disease Control and Prevention. Most recent asthma data. 2019. Available at: https://www.cdc.gov/asthma/most_recent_data.htm. Accessed June 14, 2021.

      ,
      • Moorman JE
      • Akinbami LJ
      • Bailey CM
      • et al.
      National surveillance of asthma: United States, 2001–2010.
      In addition, pediatric asthma exerts a tremendous burden on patients and their families in the form of days lost from school, interference with physical exercise, and time off of work to care for sick children. The Pediatric Asthma ED Use measure was created to evaluate the quality of outpatient care for asthma patients at the health plan level.
      In this paper, we assess whether health plans change performance categories on the Pediatric Asthma ED Use measure with the addition of socioeconomic status (SES) into risk adjustment. Specifically, we evaluate the effects of social risk adjustment on health plan performance using two different sets of social risk factors for risk adjustment. We then compare social risk factors between patients served by health plans that changed performance categories to those that did not. We hypothesized that social risk factor inclusion would change health plan performance category, improving the measured performance of those health plans serving patients with higher social risk.

      Methods

       Rationale for Including Social Risk Factors in a Pediatric Asthma Utilization Measure

      Children living below the federal poverty line are disproportionately affected by asthma with a prevalence close to 12%, and they are 40% more likely to be hospitalized annually than children from wealthier families.
      • Halfon N
      • Newacheck PW.
      Childhood asthma and poverty: differential impacts and utilization of health services.
      Asthma self-management is complex, requiring health literacy, time for and transportation to preventive visits, and capacity to acquire and administer asthma controller medications. Patients and caregivers from lower SES groups may have less capacity for asthma management, leading to more uncontrolled disease and increased ED utilization.
      • Ortega AN
      • Belanger KD
      • Paltiel AD
      • et al.
      Use of health services by insurance status among children with asthma.
      Previous reports have shown that children insured by Medicaid are nearly 70% more likely to use the ED for asthma services than privately insured children, even after adjustment for frequency of primary care visits, asthma specialty care, symptomatology, and medication use.
      • Halfon N
      • Newacheck PW.
      Childhood asthma and poverty: differential impacts and utilization of health services.
      ,
      • Ortega AN
      • Belanger KD
      • Paltiel AD
      • et al.
      Use of health services by insurance status among children with asthma.
      Failure to account for social risk in asthma quality of care measures may lead to unfair comparisons across health plans.
      Key social risk factors from the NQF report include education (related to health literacy and taking medication), employment, and income (both related to financial stability allowing for attending visits and acquiring medication). In addition, Andrist et al found an increased association between high child poverty index and pediatric intensive care use.
      • Andrist E
      • Riley CL
      • Brokamp C
      • et al.
      Neighborhood poverty and pediatric intensive care use.
      Given that there is no standard set of social risk factors for quality measure risk adjustment, we explored the effects of including multiple social risk factors (education, employment, and income) on risk adjustment as well as a single risk factor (child poverty).

       Data Sources

      We used data from the Massachusetts (MA) all-payer claims database (2014–2015) and the California Medicaid claims database (2015–2016) to examine how health plan performance categories changed when social risk was included in the risk-adjustment models for asthma-related ED utilization. The MA all-payer claims database contains claims from most insurers (commercial, Medicaid, and Medicare) but does not include individuals covered by TRICARE, Veteran's Affairs, the Federal Employees Health Benefits Program, or some commercial self-pay insurance plans.

      Overview of the Massachusetts all-payer claims database. 2016. Available at: http://www.chiamass.gov/. Accessed June 14, 2021.

      Data are submitted by health insurance payers to the MA Center for Health Information and Analysis and include patient demographics, service details, and inpatient and outpatient medical and pharmacy claims.

       Study Population

      Consistent with the IMPLEMENT PQMP Pediatric Asthma ED Use measure denominator specifications,

      IMPLEMENT for Child Health. The regents of the University of California. 2011. Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed on March 27, 2021.

      ,

      Rate of emergency department visit use for children managed for persistent asthma: detailed measure specifications. Collaboration for the advancement of pediatric quality measures (CAPQuaM). 2020. Available at: http://www.capquam.org/. Accessed June 14, 2021.

      we included patients aged 3 to 21 years-old in the measurement years (2014–2015 in MA and 2015–2016 in CA) with a diagnosis of identifiable asthma. Briefly, eligible patients had 1) at least 3 months of consecutive enrollment in the same insurance plan (the measurement month and the 2 months before), 2) evidence of claims with a primary or secondary diagnosis of asthma within the measurement month, and 3) prior documentation of asthma in the measurement year or year prior (2013 or 2014).
      • Harder VS
      • Shaw JS
      • McCulloch CE
      • et al.
      Statewide asthma learning collaborative participation and asthma-related emergency department use.
      Prior documentation of asthma was defined as a prior hospitalization for asthma, one or more ambulatory visits with a diagnosis of asthma or bronchitis, and/or at least one asthma-related prescription. The numbers of ambulatory visits and asthma-related prescriptions required for inclusion varied by age (complete details have been described prior
      • Harder VS
      • Shaw JS
      • McCulloch CE
      • et al.
      Statewide asthma learning collaborative participation and asthma-related emergency department use.
      and are in the publicly available technical specifications

      IMPLEMENT for Child Health. The regents of the University of California. 2011. Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed on March 27, 2021.

      ,

      Rate of emergency department visit use for children managed for persistent asthma: detailed measure specifications. Collaboration for the advancement of pediatric quality measures (CAPQuaM). 2020. Available at: http://www.capquam.org/. Accessed June 14, 2021.

      ). Patients with a diagnosis of cystic fibrosis or emphysema were excluded.

       Social Risk Factor Variable Definitions

      Patient 5-digit zip codes from claims records were linked to zip code-level social risk factors in the US American Community Survey (5-year file), as suggested in the NQF report.
      Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
      Social risk factors included the percentage of adults with less than high school education, the percentage of unemployed males aged 25–60 years (MA) or percentage of unemployed adults aged ≥16 years (CA), the percentage of households below the federal poverty line, and median household income. These specific variables have been validated as measures of social risk in adults and found to be associated with increased healthcare utilization and readmissions in pediatrics.
      • Bird CE
      • Seeman T
      • Escarce JJ
      • et al.
      Neighbourhood socioeconomic status and biological “wear and tear” in a nationally representative sample of US adults.
      • Beck AF
      • Huang B
      • Wheeler K
      • et al.
      The Child Opportunity Index and disparities in pediatric asthma hospitalizations across one Ohio metropolitan area, 2011-2013.
      • Baltrus P
      • Xu J
      • Immergluck L
      • et al.
      Individual and county level predictors of asthma related emergency department visits among children on Medicaid: a multilevel approach.
      • Auger KA
      • Kahn RS
      • Simmons JM
      • et al.
      Using address information to identify hardships reported by families of children hospitalized with asthma.
      In addition, following the approach from Andrist et al, we repeated the analyses using a single SES variable: zip code-level percentage of children aged <18 years living in poverty.
      • Andrist E
      • Riley CL
      • Brokamp C
      • et al.
      Neighborhood poverty and pediatric intensive care use.

       Outcome Definition

      We defined the outcome, asthma-related ED visit rate (ED visits/100 child-years), using the definition from the PQMP Pediatric Asthma ED Use measure specifications.

      IMPLEMENT for Child Health. The regents of the University of California. 2011. Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed on March 27, 2021.

      ,

      Rate of emergency department visit use for children managed for persistent asthma: detailed measure specifications. Collaboration for the advancement of pediatric quality measures (CAPQuaM). 2020. Available at: http://www.capquam.org/. Accessed June 14, 2021.

      ED visits were identified from claims records as having a primary or secondary diagnosis of asthma. The primary diagnosis was often a related symptom (eg, fever or wheezing) or a known asthma trigger (eg, upper respiratory tract illness). ED visits were excluded from the numerator if a hospitalization occurred within the same or the next calendar day to allow for ED visits that crossed midnight.

       Statistical Analyses

      We used negative binomial regression with random effects for payer to estimate health plan performance on asthma-related ED visit rates. Negative binomial regression is a generalization of Poisson regression which allows for over-dispersion of the outcome variable when the conditional variance exceeds the mean. The predicted effects (ED visit rates per member-month) and standard errors for each plan were generated in a postestimation “margins” command.
      • Norton EC
      • Miller MM
      • Kleinman LC.
      Computing adjusted risk ratios and risk differences in Stata.
      This command calculates a marginal (or average) ED visit rate by averaging predicted values for each predictor category, which can be interpreted as the per member-month rate for each predictor category. We then calculated the rate per 100 child-years for each health plan by multiplying the marginal rates by 1200 (12 months in calendar year * 100 children). Standardized z-scores were used to identify high, medium, and low performers consistent with the Centers for Medicare & Medicaid Services approach.

      Overall hospital quality star rating on care compare methodology report (v4.0). Yale-New Haven Health Services Corporation/Center for Outcomes Research and Evaluation. 2021. Available at:https://qualitynet.cms.gov/files/603966dda413b400224ddf50?filename=Star_Rtngs_CompMthdlgy_v4.1.pdf. Accessed February 3, 2021.

      Plans with a Z-statistic > 1.96 were considered poor performing outliers, those with < −1.96 were considered high performing outliers and those in between were considered no different from average.
      The baseline risk-adjustment model included fixed effects for age, gender, and mutually exclusive Pediatric Medical Complexity Algorithm chronic condition indicators.
      • Simon TD
      • Cawthon ML
      • Stanford S
      • et al.
      Pediatric medical complexity algorithm: a new method to stratify children by medical complexity.
      Z-statistics from this model were then compared to those of a second model that adjusted for the same variables plus the social risk factors specified above except child poverty level. Health plans that changed performance categories before and after risk adjustment for neighborhood social risk factors were identified. The distribution of social risk variables was compared across health plans by whether they changed rankings, using mixed effects linear models, clustered by health plan. As a secondary analysis, we repeated the same calculations using only child poverty level as a social risk factor in risk adjustment. All analyses were performed with SAS version 9.4 (SAS Institute, Inc, Cary, NC) and STATA 16 (STATA Corp, College Station, Tex).

      Results

      The sample included 83,577 patients across 29 health plans in MA and 321,072 patients across 104 health plans in CA. The overall rate of asthma-related ED visits was 18.0/100 child-years (95% confidence interval (CI) 17.6–18.3) in MA and 26.1/100 child-years (95% CI 25.9–26.3) in CA (Table 1); however, there was significant variability in ED visit rates across plans (Table 2). Using the performance categories defined above, 5 (17%) health plans in MA and 27 (26%) health plans in CA were labeled as high-performing and 6 (21%) health plans in MA and 33 (32%) health plans in CA were labeled as low-performing.
      Table 1Eligible Patient Characteristics and Asthma-Related ED Utilization Rates
      Eligible Patients, N (%)Asthma ED Rates (per 100 Child-Years, 95% CI)
      Total404,649 (100%)24.5 (24.4–24.7)
      Age group, years
       3–559,198 (14.6%)39.5 (38.9–40.1)
       6–11177,795 (43.9)21.9 (21.6–22.1)
       12–17118,886 (29.4)19.8 (19.5–20.1)
       18–2148,770 (12.1)29.7 (29.1–30.3)
      Gender
       Male226,726 (56.0%)24.5 (24.3–24.8)
       Female177,923 (44.0)24.6 (24.3–24.8)
      Insurance type
       Medicaid managed care340,616 (77.7%)25.6 (25.4–25.8)
       Medicaid FFS59,470 (13.6)28.9 (28.2–29.6)
       Commercial (MA only)30,941 (7.1)8.8 (8.4–9.2)
       Other (MA only)7,617 (1.7)10.9 (9.9–11.9)
      PMCA
       None (nonchronic)253,138 (62.6%)22.3 (22.1–22.6)
       Chronic, noncomplex89,366 (22.1)25.3 (24.9–25.7)
       Complex chronic62,145 (15.4)32.2 (31.6–32.8)
      Health plans
       Number of Health plans, N133
       Number of eligible patients per health plan, mean (SD), median (IQR)Mean: 3161 (6357)

      Median: 922 (25%ile: 121, 75%ile: 3337)
      State
       California321,07226.1 (25.9–26.3)
       Massachusetts83,57718.0 (17.6–18.3)
      ED indicates emergency department; SD, standard deviation; CI, confidence interval; PMCA, Pediatric Medical Complexity Algorithm; MA, Massachusetts; IQR, interquartile range; and FFS, fee for services.
      Table 2Performance and Performance Categories for Massachusetts and California Health Plans, Baseline Risk Adjusted Without Social Risk Factor Adjustment
      High Performing
      For asthma ED use, a lower quantity on the measure is considered higher performance. Performance categories are from the baseline risk adjustment model (adjusting for age, gender, chronic condition indicator). Rates across the 3 groups before and after risk adjustment including SES factors were similar.
      No Different Than AverageLow Performing
      For asthma ED use, a lower quantity on the measure is considered higher performance. Performance categories are from the baseline risk adjustment model (adjusting for age, gender, chronic condition indicator). Rates across the 3 groups before and after risk adjustment including SES factors were similar.
      Measure
       Massachusetts
       Health plan performance category baseline risk adjustment, n (%)
      n is the number of health plans.
      5 (17.2%)18 (62.1%)6 (20.7%)
       Rate asthma ED visits per 100 child-years
      Mean rates across patients in each performance group.
      8.9 (8.5–9.4)10.9 (9.7–12.0)23.9 (23.4–24.5)
      California
       Health plan performance category baseline risk adjustment, n (%)
      n is the number of health plans.
      27 (26.0%)44 (42.3%)33 (31.7%)
       Rate asthma ED visits per 100 child-years
      Mean rates across patients in each performance group.
      17.6 (17.3–17.9)23.1 (22.4–23.8)30.4 (30.1–30.7)
      ED indicates emergency department.
      low asterisk For asthma ED use, a lower quantity on the measure is considered higher performance. Performance categories are from the baseline risk adjustment model (adjusting for age, gender, chronic condition indicator). Rates across the 3 groups before and after risk adjustment including SES factors were similar.
      Mean rates across patients in each performance group.
      n is the number of health plans.
      In both MA and CA, performance varied with the addition of social risk variables to the baseline risk adjustment model. In MA, 2 (7%) health plans moved to a higher performance ranking and 27 (93%) did not move when the risk adjustment model included the social risk variables; no health plans moved to a lower performance ranking. In CA, 7 (7%) health plans moved to a higher performance category, 91 (88%) did not move, and 6 (6%) moved to a lower performance category after adjustment for social risk. Health plans in both states that moved to higher performance categories had more patients on average residing in higher social risk zip codes as evidenced by lower median incomes, higher percentage of limited education, higher percentage of poverty, and higher unemployment percentage. In contrast, health plans that moved to a lower performance category in CA had fewer patients residing in higher social risk zip codes than those that did not move performance categories (Table 3). These differences were statistically significant for poverty level and unemployment in CA. When only using child poverty rates for social risk factor assessment, findings for Massachusetts were the same (same number of health plans changing performance ranking and the same health plans moved); California findings were similar with both approaches, but with fewer health plans changing performance category (2 moved higher; 7 moved lower) (Table 4). Differences in social risk factors by change in performance category were statistically significant in education and poverty measures.
      Table 3Distribution of Social Risk Measures for Patients in Health Plans That Changed Performance Ranking After Social Risk Factor Adjustment Using Income, Education, Employment Measures
      Median Household Income in Zip Code, Mean (95% CI)% Population With <High School Education in Zip Code, Mean (95% CI)% Households Below Federal Poverty Level in Zip Code, Mean (95% CI)% Unemployed in Zip Code, Mean (95% CI)
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.
      Massachusetts
       Patients in health plans that did not change (n = 77,304)
      Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      $72,696

      ($69,207–$76,360)
      5.2%

      (4.8%–5.5%)
      10.3%

      (9.4%–11.1%)
      7.6%

      (7.3%–7.9%)
       Patients in health plans moved to higher performance ranking (n = 6273)
      Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      $64,888

      ($55,568–$75,771)
      5.6%

      (4.4%–6.8%)
      11.1%

      (8.3%–13.8%)
      8.1%

      (7.2%–8.9%)
      California
       Patients in health plans that did not change (n = 276,938)
      Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      $52,362

      ($50,200–54,618)
      19.7%

      (17.9%–21.5%)
      13.5%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (12.4%–14.6%)
      5.8%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (5.5%–6.1%)
       Patients in health plans that moved to higher performance ranking (n = 18,182)
      Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      $45,208

      ($38,981–52,430)
      22.3%

      (17.9%–21.5%)
      19.5%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (15.6%–23.4%)
      7.3%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (6.2%–8.3%)
       Patients in health plans that moved to lower performance ranking (n = 25,952)
      Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      $72,181

      ($61,489–84,734)
      17.1%

      (10.2%–24.0%)
      8.1%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (4.0%–12.4%)
      5.0%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (3.9%–6.1%)
      CI indicates confidence interval.
      low asterisk Health plans that did not change: MA n = 27; CA n = 91. Health plans that improved: MA n = 2; CA n = 7. Health plans with lower performance: MA n = 0; CA n = 6.
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.
      Table 4Distribution of Social Risk Measures for Patients in Health Plans That Changed Performance Ranking After Social Risk Factor Adjustment Only Using Percent of Child Poverty
      Median Household Income in Zip Code, Mean (95% CI)% Population With <High School Education in Zip Code, Mean (95% CI)% Households Below Federal Poverty Level in Zip Code, Mean (95% CI)% Unemployed in Zip Code, Mean (95% CI)
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.
      California
       Patients in health plans that did not change (n = 307,666)
      Health plans that did not change: CA n = 95. Health plans that improved: CA n = 2. Health plans with lower performance: CA n = 7. Performance effects in Massachusetts health plans were the same as the model in Table 3.
      $52,239

      ($50,044–54,532)
      20.0%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (18.4%–21.7%)
      13.9%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (12.8%–14.9%)
      5.9%

      (5.6%–6.2%)
       Patients in health plans that moved to higher performance ranking (n = 3388)
      Health plans that did not change: CA n = 95. Health plans that improved: CA n = 2. Health plans with lower performance: CA n = 7. Performance effects in Massachusetts health plans were the same as the model in Table 3.
      $42,511

      ($31,838–56,762)
      34.4%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (23.0%–45.6%)
      22.7%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (15.5%–30.0%)
      6.1%

      (4.1%–8.2%)
       Patients in health plans that moved to lower performance ranking (n = 10,018)
      Health plans that did not change: CA n = 95. Health plans that improved: CA n = 2. Health plans with lower performance: CA n = 7. Performance effects in Massachusetts health plans were the same as the model in Table 3.
      $65,248

      ($55,892–76,172)
      12.2%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (6.1%–18.2%)
      8.1%
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.


      (4.2%–12.0%)
      5.0%

      (3.9%–6.1%)
      CI indicates confidence interval.
      low asterisk Health plans that did not change: CA n = 95. Health plans that improved: CA n = 2. Health plans with lower performance: CA n = 7. Performance effects in Massachusetts health plans were the same as the model in Table 3.
      P value < .01 for comparisons across patients in health plans according to health plan change in performance ranking. Otherwise, these P values were >.05.

      Discussion

      Using the Pediatric Asthma ED Use measure, we found that incorporation of social risk factors into risk-adjustment had a measurable effect on health plan performance rankings. In both MA and CA, 7% to 13% of health plans changed rankings after adjustment for social risk. Inclusion of neighborhood education, unemployment and income in the case-mix adjusted models appeared to improve performance for health plans caring for more patients from areas with higher social adversity but decreased performance for health plans caring for patients with lower adversity. These findings were similar across both approaches to social risk factor assessment and suggest that some of the variation in performance for lower-performing health plans was explained by higher percentages of patient with social risk.
      Numerous studies have documented a strong relationship between asthma ED visit rates and social risk factors.
      • Beck AF
      • Huang B
      • Wheeler K
      • et al.
      The Child Opportunity Index and disparities in pediatric asthma hospitalizations across one Ohio metropolitan area, 2011-2013.
      ,
      • Baltrus P
      • Xu J
      • Immergluck L
      • et al.
      Individual and county level predictors of asthma related emergency department visits among children on Medicaid: a multilevel approach.
      ,
      • Malhotra K
      • Baltrus P
      • Zhang S
      • et al.
      Geographic and racial variation in asthma prevalence and emergency department use among Medicaid-enrolled children in 14 southern states.
      ,
      • Zhang Q
      • Lamichhane R
      • Diggs LA
      Disparities in emergency department visits in American children with asthma: 2006–2010.
      Factors such as obesity, healthcare literacy, access to preventive and specialist care, and ability to afford controller medications are believed to contribute to this relationship.
      • Malhotra K
      • Baltrus P
      • Zhang S
      • et al.
      Geographic and racial variation in asthma prevalence and emergency department use among Medicaid-enrolled children in 14 southern states.
      While some of these factors may be addressable with proper asthma management and education, others are more challenging to address. As such, even with excellent outpatient care, health plans that care for a disproportionate number of patients with high social risk may be unable to effectively improve their asthma ED utilization rates.
      Our findings demonstrate that it is feasible to incorporate social risk into risk-adjustment models at the health plan level using all-payer claims data as well as payer-specific Medicaid claims; however, questions remain as to which social risk metrics to use. Social risk is a complex attribute that encompasses a host of factors with both neighborhood and individual factors interacting to influence health outcomes.
      Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
      ,

      National Quality Forum. Evaluation of the NQF trial period for risk adjustment for social risk factors. 2017. Available from: NQF: NQF Initiative to Adjust Measures for Social Risk Factors Highlights Successes, Opportunities (qualityforum.org) Accessed August 23, 2021.

      ,

      National Academies of Sciences, Engineering, and Medicine. Integrating social care into the delivery of health care. 2019. Available from: Integrating Social Needs Care into the Delivery of Health Care to Improve the Nations Health | National Academies Accessed August 23, 2021.

      With the multitude of social risk variables available, there is no consensus on the best approach for incorporating social risk factors in risk adjustment.
      Case-mix adjustment models using administrative claims data have gravitated toward neighborhood measures (zip code or census tract) of social risk because they are readily available, easily linked, and have been shown to be associated with health outcomes.

      National Quality Forum. Risk adjustment for socioeconomic status or other sociodemographic factors. 2014. Available from: NQF: Risk Adjustment for Socioeconomic Status or Other Sociodemographic Factors (qualityforum.org) Accessed August 23, 2021.

      While appealing, these variables may not accurately reflect the socioeconomic composition of a health plan population if there is significant heterogeneity within a neighborhood. As such, neighborhood risk factors can be poor proxies for individual social risk and may not appropriately represent the socioeconomic composition of a health plan. However, there is also some evidence that neighborhood alone, independent of individual risk, is associated with health outcomes.
      • Diez Roux AV
      Investigating neighborhood and area effects on health.
      ,
      • McClure R
      • Kegler S
      • Davey T
      • et al.
      Contextual determinants of childhood injury: a systematic review of studies with multilevel analytic methods.
      As noted in the NQF report, the choice of social risk metrics and their source is an evolving science, leading to the recommendation to assess the effects of social risk adjustments without dictating exact methods for doing so.
      Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
      Research aimed at understanding the importance of various social risk metrics (education, income, occupation, health literacy, environmental factors) in risk-adjustment as well as the interaction between individual and neighborhood social risk is greatly needed.
      The NQF report to Congress in 2020 published a framework for determining when and how to adjust for social risk in quality measurement.
      Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
      They concluded that structure and process measures should never be adjusted for social risk or other case-mix variables, whereas other measures such as resource use measures should always be adjusted for social risk to ensure that socially at-risk patients receive adequate resources. For outcome, patient experience, and utilization measures, the decision whether to adjust for social risk was more challenging, and the NQF advised against a blanket social risk-adjustment policy.
      In the case of the Pediatric Asthma ED utilization measure, disparities in asthma care and outcomes are well-documented and adjustment for social risk changed performance rankings for ∼10% of health plans. Our findings suggest that though child poverty has validity as a stand-alone indicator of social risk, the inclusion of additional measures of education and employment status, per the conceptual model suggested by the NQF and others, capture social risk more completely.
      • Andrist E
      • Riley CL
      • Brokamp C
      • et al.
      Neighborhood poverty and pediatric intensive care use.
      ,

      National Quality Forum. Risk adjustment for socioeconomic status or other sociodemographic factors. 2014. Available from: NQF: Risk Adjustment for Socioeconomic Status or Other Sociodemographic Factors (qualityforum.org) Accessed August 23, 2021.

      Based on the findings in this study in the context of broader discussions, the PQMP measure includes the fuller set of social risk variables in the NQF-endorsed risk-adjustment model for the measure.

      National Quality Forum. Measure 3599: Pediatric Asthma Emergency Department Use. 2021. Available from: NQF: Measures, Reports & Tools (qualityforum.org) Accessed August 23, 2021.

      Although this approach has been adopted for the pediatric asthma measure, the decision whether to risk adjust for social risk will differ depending on the measure and other approaches to risk adjustment may be considered. For example, the Medicare Payment Advisory Commission ultimately concluded that adult 30-day hospital readmission measures should not be risk-adjusted for social risk but instead proposed stratifying performance comparisons to evaluate safety-net hospitals relative to one another rather than to the broader group.

      Report to the Congress: Medicare and the health care delivery system. Medicare Payment Advisory Commission. Washington, DC: 2013. Available at: http://www.medpac.gov/docs/default-source/reports/jun13_entirereport.pdf?sfvrsn=0. Accessed June 14, 2021.

      While this approach has worked for hospital measures and led to fewer penalties levied on safety-net hospitals, there are no comparable safety-net designations for health plans or outpatient providers. The overarching goal of quality measurement is to optimize the care of pediatric patients, and the discussion about social risk highlights the opportunity in pediatrics to address social risk early, leveraging the current efforts towards social risk screening in pediatric primary care.
      • Sokol R
      • Austin A
      • Chandler C
      • et al.
      Screening children for social determinants of health: a systematic review.
      ,
      • Duffee J
      • Szilagyi M
      • Forkey H
      • et al.
      Trauma-informed care in child health systems.
      There are a few notable limitations to this study. First, we adjusted for zip-code level social risk factors because we did not have access to census block data, though this can be a limitation also for those using quality measures. ZIP codes have more heterogeneity than census blocks or tracts, making census blocks potentially preferable for risk adjustment. Second, we used Medicaid data from MA and CA which historically have had lower income eligibility criteria than other states; thus, it is not clear how these analyses would generalize more broadly to other state Medicaid programs.
      In summary, we found that adjusting for social risk changed performance rankings on the Pediatric Asthma ED Use measure for a substantial number of health plans. This example and others in the literature show that careful consideration should be given to each measure and its application when deciding whether to risk adjust for social risk. In particular, insight into the role for both individual and neighborhood level social risk as well as the intersection between these metrics is needed. Approaches to social risk adjustment should be guided by future research into which variables to include and the implications for risk adjustment on hospital or health plan performance.

      Acknowledgment

      Financial statement: NSB and CEM were supported by grant number U18HS025297 from the Agency for Healthcare Research and Quality (AHRQ). ST was supported by grant number U18 HS025299 from AHRQ.
      The views expressed in this article are those of the authors, and no official endorsement by the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare and Medicaid Services (CMS), or the Department of Health and Human Services (DHHS) is intended or should be inferred.
      This article is published as part of a supplement sponsored by the US Department of Health and Human Services, the Centers for Medicare and Medicaid Services, and the Agency for Healthcare Research and Quality.

      References

      1. Pediatric Quality Measures Program. Agency for Healthcare Research and Quality. 2018. Available at: https://www.ahrq.gov/pqmp/index.html Accessed February 23, 2021.

        • Adirim T
        • Meade K
        • Mistry K.
        A new era in quality measurement: the development and application of quality measures.
        Pediatrics. 2017; 139e201623442
        • Krumholz HM
        • Bernheim SM.
        Considering the role of socioeconomic status in hospital outcomes measures.
        Ann Intern Med. 2014; 161: 833-834https://doi.org/10.7326/M14-2308
      2. Hospital Readmissions Reduction Program (HRRP). Centers for Medicare and Medicaid services. 2020. Available at: https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program Accessed February 23, 2021.

        • Joynt KE
        • Jha AK.
        Thirty-day readmissions—truth and consequences.
        N Engl J Med. 2012; 366: 1366-1369https://doi.org/10.1056/NEJMp1201598
        • Hernandez AF
        • Curtis LH.
        Minding the gap between efforts to reduce readmissions and disparities.
        JAMA. 2011; 305: 715-716https://doi.org/10.1001/jama.2011.167
        • Bhalla R
        • Kalkut G.
        Could Medicare readmission policy exacerbate health care system inequity?.
        Ann Intern Med. 2010; 152: 114-117https://doi.org/10.7326/0003-4819-152-2-201001190-00185
        • Gu Q
        • Koenig L
        • Faerberg J
        • et al.
        The Medicare Hospital Readmissions Reduction Program: potential unintended consequences for hospitals serving vulnerable populations.
        Health Serv Res. 2014; 49: 818-837https://doi.org/10.1111/1475-6773.12150
      3. Improving Medicare post-acute care transformation act of 2014. Public Law No. 113-185. 2014. Available at: https://www.govinfo.gov/content/pkg/PLAW-113publ185/html/PLAW-113publ185.htm. Accessed February 4, 2020.

      4. Risk Adjustment for Socioeconomic Status or Other Socioeconomic Factors.
        National Quality Forum, Washington, DC2014 (Available at:) (Accessed November 16, 2021)
      5. Second Report to Congress on Social Risk Factors and Performance in Medicare's Value-Based Purchasing Program.
        U.S. Department of Health and Human Services, Washington, DC2020 (Available at: https://aspe.hhs.gov/pdf-report/second-impact-report-to-congress#:~:text=As%20required%20by%20the%20IMPACT,on%20providers%20serving%20socially%20at%2D. Accessed November 16, 2021)
        • Bernheim SM
        • Parzynski CS
        • Horwitz L
        • et al.
        Accounting for patients' socioeconomic status does not change hospital readmission rates.
        Health Aff (Milwood). 2016; 35: 1461-1470
        • Joynt Maddox KE
        • Reidhead M
        • Hu J
        • et al.
        Adjusting for social risk factors impacts performance and penalties in the Hospital Readmissions Reduction Program.
        Health Serv Res. 2019; 54: 327-336
        • Gaskin DJ
        • Zare H
        • Vazin R
        • et al.
        Racial and ethnic composition of hospitals’ service areas and the likelihood of being penalized for excess readmissions by the Medicare program.
        Med Care. 2018; 56: 934-943
        • Durfey SNM
        • Kind AJH
        • Gutman R
        • et al.
        Impact of risk adjustment for socioeconomic status on Medicare Advantage plan quality rankings.
        Health Aff (Milwood). 2018; 37: 1065-1072
        • Glance LG
        • Kellermann AL
        • Osler TM
        • et al.
        Impact of risk adjustment for socioeconomic status on risk-adjusted surgical readmission rates.
        Ann Surg. 2016; 263: 698-704
        • Sandhu AT
        • Bhattacharya J
        • Lam J
        • et al.
        Adjustment for social risk factors does not meaningfully affect performance on Medicare's MIPS clinician cost measures.
        Health Aff (Milwood). 2020; 39: 1495-1503
        • Nguyen CA
        • Gilstrap LG
        • Chernew ME
        • et al.
        Social risk adjustment of quality measures for diabetes and cardiovascular disease in a commercially insured US population.
        JAMA Netw Open. 2019; 2e190838
        • Sills MR
        • Hall M
        • Colvin JD
        • et al.
        Association of social determinants with children's hospitals’ preventable readmissions performance.
        JAMA Pediatr. 2016; 170: 350-358
        • Sills MR
        • Hall M
        • Cutler GJ
        • et al.
        Adding social determinant data changes children's hospitals’ readmissions performance.
        J Pediatr. 2017; 186: 150-157
        • Betson DM
        • Michael RT.
        Why so many children are poor.
        Future Child. 1997; 7: 25-39
      6. Health insurance coverage of children 0-18. Kaiser Family Foundation. 2019. Available at:https://www.kff.org/other/state-indicator/children-0-18/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D. Accessed February 23, 2021.

      7. IMPLEMENT for Child Health. The regents of the University of California. 2011. Available at: https://chipper.ucsf.edu/studies/implement/documents. Accessed on March 27, 2021.

      8. Rate of emergency department visit use for children managed for persistent asthma: detailed measure specifications. Collaboration for the advancement of pediatric quality measures (CAPQuaM). 2020. Available at: http://www.capquam.org/. Accessed June 14, 2021.

      9. Centers for Disease Control and Prevention. Most recent asthma data. 2019. Available at: https://www.cdc.gov/asthma/most_recent_data.htm. Accessed June 14, 2021.

        • Moorman JE
        • Akinbami LJ
        • Bailey CM
        • et al.
        National surveillance of asthma: United States, 2001–2010.
        Vital Health Stat 3. 2012; 35: 1-58
        • Halfon N
        • Newacheck PW.
        Childhood asthma and poverty: differential impacts and utilization of health services.
        Pediatrics. 1993; 91: 56-61
        • Ortega AN
        • Belanger KD
        • Paltiel AD
        • et al.
        Use of health services by insurance status among children with asthma.
        Med Care. 2001; 39: 1065-1074
        • Andrist E
        • Riley CL
        • Brokamp C
        • et al.
        Neighborhood poverty and pediatric intensive care use.
        Pediatrics. 2019; 144e20190748
      10. Overview of the Massachusetts all-payer claims database. 2016. Available at: http://www.chiamass.gov/. Accessed June 14, 2021.

        • Harder VS
        • Shaw JS
        • McCulloch CE
        • et al.
        Statewide asthma learning collaborative participation and asthma-related emergency department use.
        Pediatrics. 2020; 146e20200213
        • Bird CE
        • Seeman T
        • Escarce JJ
        • et al.
        Neighbourhood socioeconomic status and biological “wear and tear” in a nationally representative sample of US adults.
        J Epidemiol Community Health. 2010; 64: 860-865
        • Beck AF
        • Huang B
        • Wheeler K
        • et al.
        The Child Opportunity Index and disparities in pediatric asthma hospitalizations across one Ohio metropolitan area, 2011-2013.
        J Pediatr. 2017; 190: 200-206
        • Baltrus P
        • Xu J
        • Immergluck L
        • et al.
        Individual and county level predictors of asthma related emergency department visits among children on Medicaid: a multilevel approach.
        J Asthma. 2017; 54: 53-61
        • Auger KA
        • Kahn RS
        • Simmons JM
        • et al.
        Using address information to identify hardships reported by families of children hospitalized with asthma.
        Acad Pediatr. 2017; 17: 79-87
        • Norton EC
        • Miller MM
        • Kleinman LC.
        Computing adjusted risk ratios and risk differences in Stata.
        Stata J Promot Commun Stat Stata. 2013; 13: 492-509
      11. Overall hospital quality star rating on care compare methodology report (v4.0). Yale-New Haven Health Services Corporation/Center for Outcomes Research and Evaluation. 2021. Available at:https://qualitynet.cms.gov/files/603966dda413b400224ddf50?filename=Star_Rtngs_CompMthdlgy_v4.1.pdf. Accessed February 3, 2021.

        • Simon TD
        • Cawthon ML
        • Stanford S
        • et al.
        Pediatric medical complexity algorithm: a new method to stratify children by medical complexity.
        Pediatrics. 2014; 113: e1647-e1654
        • Malhotra K
        • Baltrus P
        • Zhang S
        • et al.
        Geographic and racial variation in asthma prevalence and emergency department use among Medicaid-enrolled children in 14 southern states.
        J Asthma. 2014; 51: 913-921
        • Zhang Q
        • Lamichhane R
        • Diggs LA
        Disparities in emergency department visits in American children with asthma: 2006–2010.
        J Asthma. 2017; 54: 679-686
      12. National Quality Forum. Evaluation of the NQF trial period for risk adjustment for social risk factors. 2017. Available from: NQF: NQF Initiative to Adjust Measures for Social Risk Factors Highlights Successes, Opportunities (qualityforum.org) Accessed August 23, 2021.

      13. National Academies of Sciences, Engineering, and Medicine. Integrating social care into the delivery of health care. 2019. Available from: Integrating Social Needs Care into the Delivery of Health Care to Improve the Nations Health | National Academies Accessed August 23, 2021.

      14. National Quality Forum. Risk adjustment for socioeconomic status or other sociodemographic factors. 2014. Available from: NQF: Risk Adjustment for Socioeconomic Status or Other Sociodemographic Factors (qualityforum.org) Accessed August 23, 2021.

        • Diez Roux AV
        Investigating neighborhood and area effects on health.
        Am J Public Health. 2001; 91: 1783-1789
        • McClure R
        • Kegler S
        • Davey T
        • et al.
        Contextual determinants of childhood injury: a systematic review of studies with multilevel analytic methods.
        Am J Public Health. 2015; 105: e37-e43
      15. National Quality Forum. Measure 3599: Pediatric Asthma Emergency Department Use. 2021. Available from: NQF: Measures, Reports & Tools (qualityforum.org) Accessed August 23, 2021.

      16. Report to the Congress: Medicare and the health care delivery system. Medicare Payment Advisory Commission. Washington, DC: 2013. Available at: http://www.medpac.gov/docs/default-source/reports/jun13_entirereport.pdf?sfvrsn=0. Accessed June 14, 2021.

        • Sokol R
        • Austin A
        • Chandler C
        • et al.
        Screening children for social determinants of health: a systematic review.
        Pediatrics. 2019; 144e20191622
        • Duffee J
        • Szilagyi M
        • Forkey H
        • et al.
        Trauma-informed care in child health systems.
        Pediatrics. 2021; 148e2021052579