Abstract
Objective
To develop and test a new quality measure assessing timeliness of follow-up mental health care for youth presenting to the emergency department (ED) with suicidal ideation or self-harm.
Methods
Based on a conceptual framework, evidence review, and a modified Delphi process, we developed a quality measure assessing whether youth 5 to 17 years old evaluated for suicidal ideation or self-harm in the ED and discharged to home had a follow-up mental health care visit within 7 days. The measure was tested in 4 geographically dispersed states (California, Pennsylvania, South Carolina, Tennessee) using Medicaid administrative data. We examined measure feasibility of implementation, variation, reliability, and validity. To test validity, adjusted regression models examined associations between quality measure scores and subsequent all-cause and same-cause hospital readmissions/ED return visits.
Results
Overall, there were 16,486 eligible ED visits between September 1, 2014 and July 31, 2016; 53.5% of eligible ED visits had an associated mental health care follow-up visit within 7 days. Measure scores varied by state, ranging from 26.3% to 66.5%, and by youth characteristics: visits by youth who were non-White, male, and living in an urban area were significantly less likely to be associated with a follow-up visit within 7 days. Better quality measure performance was not associated with decreased reutilization.
Conclusions
This new ED quality measure may be useful for monitoring and improving the quality of care for this vulnerable population; however, future work is needed to establish the measure's predictive validity using more prevalent outcomes such as recurrence of suicidal ideation or deliberate self-harm.
Keywords
What's New
We developed and tested a new quality measure to assess timeliness of follow-up mental health care for youth presenting to the emergency department with suicidal ideation/self-harm. Results demonstrate poor measure scores overall with substantial variation by state and patient characteristics.
Suicidal ideation and deliberate self-harm is a major health concern in the pediatric population. Suicide is the second-leading cause of death among children, adolescents, and young adults aged 10 to 24.
1
In 2019, almost 19% of United States high school students reported seriously considering suicide, 16% made a plan to kill themselves, and 3% made a suicide attempt that required medical attention.CDC. Ten Leading Causes of Death by Age Group, United States –2018. Available at: https://www.cdc.gov/injury/images/lc-charts/leading_causes_of_death_by_age_group_2018_1100w850h.jpg. Accessed June 1, 2020.
2
It is estimated that 1.12 million emergency department (ED) visits by children and adolescents 5 to 17 years old (referred to as youth from here forward) in 2015 were due to suicide attempts and suicidal ideation, almost double the estimated number of such visits in 2007.CDC. Youth Risk Behavior Survey Data Summary and Trends Report 2009-2019. Available at:https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBSDataSummaryTrendsReport2019-508.pdf. Accessed February 2, 2021.
3
In light of the high prevalence of suicidal ideation and self-harm and the associated burdens of morbidity and mortality, recent federal health policies have made quality improvement and optimizing care for pediatric mental health conditions a national goal.
4
,5
One potential target for quality improvement has been timely follow-up after discharge from the ED. The days immediately following discharge from the ED for youth with suicidal ideation or self-harm are high-risk. Yet rates of follow-up in this population are variable, ranging from 39% to 87%, with a higher likelihood of follow-up among those with an existing psychiatric diagnosis.6
, 7
, 8
, 9
, 10
In order to develop and evaluate the validity of pediatric quality measures related to mental health care, the Pediatric Quality Measures Program (PQMP) funded Center of Excellence on Quality of Care Measures for Children with Complex Needs was charged with developing quality measures to assess hospital-based mental health services for youth. In this paper, we describe the development and testing of a new quality measure that assesses whether youth presenting to the ED for suicidal ideation or self-harm received timely follow-up mental health care.
Specifically, the measure assesses whether a follow-up visit with a mental health care provider occurred within 7 days of the index ED visit. Evidence supporting this measure includes previous findings that timely follow-up after an ED visit for suicidal ideation/self-harm is associated with a longer period of time to the next ED visit, and lower rates of subsequent suicidal ideation/self-harm.
7
,11
, 12
, 13
, 14
, 15
, 16
, 17
In addition, prior work has shown that active engagement strategies based on cognitive analytic or behavioral therapies, such as Therapeutic Assessment and Family Intervention for Suicide Prevention, can be effective in facilitating mental health treatment access and ongoing participation.18
, 19
, 20
, 21
While there is no common standard for how soon after discharge the appointment should occur, within one week parallels other recommendations and published studies and was judged to be a reasonable amount of time for the appointment to occur by a Delphi panel (see Method) that assessed the face validity of this measure.9
,22
, 23
, 24
Method
Measure Development
Measure development began with the creation of a conceptual framework
25
and a thorough review of existing clinical practice guidelines26
,27
and existing literature on hospital-based care for youth with suicidal ideation/self-harm; details of this literature review are described elsewhere.28
Based on this framework and evidence review, we developed draft quality measures to assess hospital-based care for youth with suicidal ideation or self-harm. Using the RAND–University of California, Los Angeles, modified Delphi method, a multistake holder panel evaluated and scored each measure on validity and feasibility. This approach was used to identify measures for further development and testing.28
This method resulted in 21 measures endorsed for potential field-testing: 4 measures to be assessed using a caregiver survey25
, 16 measures to be assessed using medical record abstraction28
, and 1 measure to be assessed using administrative (claims) data.In this paper, we focus on the development and testing of the one administrative data quality measure endorsed by the panel, which concerned timely follow-up care for youth presenting to the ED with suicidal ideation/self-harm and discharged to home (development and testing of the remaining 20 measures has been described elsewhere
25
,28
).Measure Definition
Denominator: Eligible ED visits attended by youth 5 to 17 years old with a diagnosis (not necessarily primary) of suicidal ideation or self-harm who were discharged to home. For the visit to be eligible, the youth had to be enrolled in the health plan (in this case Medicaid) at the time of the ED visit and for at least one month after the ED visit. ED visits followed by an inpatient admission within 7 days were not eligible because these patients may have had an appointment scheduled with an outpatient mental health provider within the 7-day window but were admitted and so could not attend the appointment. ED visit eligibility was determined using administrative billing codes and an algorithm that indicated the presence of Evaluation & Management Current Procedural Terminology (CPT) codes and a Place of Service code 23 (emergency room-hospital), or a revenue center code of 0450-0459 (emergency room) or 0981 (professional fees-emergency room). Visits for suicidal ideation or self-harm were identified using International Classification of Diseases, Ninth Revision (ICD-9) codes of E950-E959, V62.84, or 300.9 or ICD-10 codes; see Appendix for ICD-10 codes.
29
, 30
, 31
Numerator: Eligible ED visits that were followed by a mental health care visit (including tele-health) within 7 days of discharge. CPT codes associated with mental health care service provision were used to identify follow-up visits that fulfilled this measure definition (see Appendix for further details).
The quality measure was dichotomously scored: if a youth having an ED visit for suicidal ideation or self-harm subsequently had a follow-up mental health care visit (including tele-health) within 7 days of ED discharge, the measure score was one, otherwise the measure score was zero. At the State Medicaid health plan level, the measure score was calculated as the percentage of scores that were equal to one, with higher percentages indicating better quality of care.
Field Test Sample
We tested the measure in 4 State Medicaid insurance programs: South Carolina (SC), California (CA), Tennessee (TN), Pennsylvania (PA). Specifically, we assessed the feasibility of using existing administrative data to score this quality measure, examine variation in measure performance, and evaluate the predictive validity of the measure. In state Medicaid administrative data, we identified all eligible ED visits (defined above) between September 1, 2014 and July 31, 2016. Although the measure specifications require continuous enrollment for only one month following the index ED visit, here we required the youth to be continuously enrolled in Medicaid for 2 months immediately following the ED visit to allow for validation metric availability, described below. For youth with multiple eligible ED visits between September 1, 2014 and July 31, 2016, only the first eligible ED visit was included (that is, a youth may only contribute to the measure denominator one time). As described above, to pass the quality measure and be counted in the numerator, a youth with an eligible ED visit had to have an associated administrative claim indicating a mental health care follow-up visit occurred within 7 days of ED discharge.
Validation Metrics
We used 4 validation metrics to examine the measure's predictive validity: 1) all-cause ED return visits within 8 to 30 days of the index ED visit; 2) all-cause inpatient admissions within 8 to 30 days of the index ED visit; 3) same-cause ED return visits within 8 to 30 days of the index ED visit; and 4) same-cause inpatient admissions within 8 to 30 days of the index ED visit. Codes used to identify same-cause visits/admissions were the same codes used to assess eligibility for the measure (see Appendix). ED visits within the first 7 days of the index ED visit were not counted as return visits (N = 1,675; 84% of these were among youth in CA, 39% were among Hispanic youth, 37% were among White youth) because it is possible that an ED return visit occurring within this time-frame might have occurred prior to a scheduled mental health care follow-up visit (which we hypothesized may prevent such reutilization). Based on previous work showing that engagement in treatment can reduce future suicidal ideation,
12
, 13
, 14
we hypothesized that better performance on the quality measure would be associated with decreased 30-day ED return visits and hospital admissions.Measure Testing
Measure feasibility was evaluated by examining the eligibility counts by state. Variation in measure scores by youth characteristics was assessed using multivariate logistic regression. We examined the independent association between performance on the quality measure and age, gender, race/ethnicity, and residence in a metropolitan statistical area (MSA).
Since this measure was developed primarily for implementation at the health plan level, we assessed health plan-level (managed care organization [MCO]) and hospital-level (1) variability and (2) reliability using an intraclass correlation coefficient (ICC) which assesses the extent to which the quality measure can differentiate between hospitals/health plans in terms of performance. The ICC was used to calculate the number of eligible visits per hospital/health plan needed to achieve excellent reliability using the Spearman-Brown formula. This assessment of reliability parallels the National Quality Forum measure evaluation criteria which states that a measure must be well-defined and precisely specified so it can be implemented consistently within and across organizations and allow for comparability.
33
Health plan information was not available for SC and thus, SC was only included in the hospital-level reliability assessment.National Quality Forum. Measure Evaluation Criteria and Guidance for Evaluating Measures for Endorsement. Available at:http://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=86084. Accessed June 6, 2018.
Predictive validity was examined using logistic regression models (pooling all states) adjusted for state, race/ethnicity, age, and gender where the outcomes were the validation metrics described above. Study procedures and use of data were approved by each state's Medicaid agency and the institutional review board of the RAND Corporation.
Results
Feasibility
Overall, there were 16,486 eligible ED visits: 1,161 in South Carolina, 9930 in California, 2178 in Tennessee, and 3217 in Pennsylvania. These eligible ED visits occurred in a total of 2288 hospitals.
Over one-third of eligible visits were among youth 16 to 17 years old, more than half were among females, 43% were among White youth, and the majority were among youth living in an MSA (Table 1). Distributions of all characteristics significantly differed across states. As expected, differences in the race/ethnicity distribution by state reflect the different composition of the states’ populations;
P < .001.MSA indicates metropolitan statistical area.
34
for example, CA had a much higher percentage of visits by Hispanic youth (45.9% in CA vs 2.9% in SC, 3.1% in TN, 11.6% in PA).U.S. Census Bureau, 2010 Census. American FactFinder, Race and Hispanic or Latino Origin: 2010. Available at:https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=CF. Accessed July 14, 2017.
Table 1Characteristics of Youth with an Eligible ED Visit for Suicidal Ideation/Self-Harm
| Overall N = 16,486 | South Carolina N = 1,161 | California N = 9,930 | Tennessee N = 2,178 | Pennsylvania N = 3,217 | |
| N (%) | N (%) | N (%) | N (%) | N (%) | |
| Youth Characteristics | |||||
| Age | |||||
| 5–12 years | 2842 (17.2%) | 301 (25.9%) | 1452 (14.6%) | 397 (18.2%) | 692 (21.5%) |
| 13–15 years | 7364 (44.7%) | 499 (43.0%) | 4530 (45.6%) | 955 (43.9%) | 1380 (42.9%) |
| 16–17 years | 6280 (38.1%) | 361 (31.1%) | 3948 (39.8%) | 826 (37.9%) | 1145 (35.6%) |
| Gender | |||||
| Male | 5980 (36.3%) | 457 (39.4%) | 3337 (33.6%) | 873 (40.1%) | 1313 (40.8%) |
| Female | 10506 (63.7%) | 704 (60.6%) | 6593 (66.4%) | 1305 (59.9%) | 1904 (59.2%) |
| Race/ethnicity | |||||
| Hispanic | 5033 (30.5%) | 34 (2.9%) | 4559 (45.9%) | 67 (3.1%) | 373 (11.6%) |
| White | 7090 (43.0%) | 526 (45.3%) | 3061 (30.8%) | 1312 (60.2%) | 2191 (68.1%) |
| African American | 2111 (12.8%) | 357 (30.8%) | 932 (9.4%) | 284 (13.0%) | 538 (16.7%) |
| Asian/Pacific Islander | 319 (1.9%) | 3 (0.3%) | 289 (2.9%) | 9 (0.4%) | 18 (0.6%) |
| Other/Unknown | 1933 (11.7%) | 241 (20.8%) | 1089 (11.0%) | 506 (23.2%) | 97 (3%) |
| Lives in MSA | |||||
| No | 2297 (13.9%) | 315 (27.2%) | 574 (5.8%) | 765 (35.1%) | 643 (20.0%) |
| Yes | 14182 (86.1%) | 843 (72.8%) | 9353 (94.2%) | 1412 (64.9%) | 2574 (80.0%) |
Notes: Significance indicates differences in characteristics between states;
Measure Performance, Variation, and Reliability
Overall performance was 53.5% across all eligible ED visits in all states. The measure scores differed dramatically by state: from 26.3% in Tennessee to 66.5% in California (Table 2, P < .0001). Visits by non-White youth were significantly less likely to result in a follow-up visit within 7 days compared to White youth, whereas females and those not living in an MSA were more likely to have a follow-up visit within 7 days compared to males and those living in an MSA, respectively.
Table 2Measure Scores Overall, by State, and by Youth Characteristics
| Measure Score (0–100%) | ||
| Overall | 53.5% | |
| State*** | Measure Score (0–100%) | |
| South Carolina | 32.3% | |
| California | 66.5% | |
| Tennessee | 26.3% | |
| Pennsylvania | 39.2% | |
| Youth Characteristics | Measure Score (0–100%) | Adjusted Odds Ratio (95% Confidence Interval) ‡ Odds ratio and significance logistic regression results examining differences in measure scores by youth characteristic, adjusted for state and all other youth characteristics in this table; *P < .05. **P < .01. ***P < .001. ref indicates the reference group when testing for differences in measure scores by characteristic. |
| Youth age | ||
| 5–12 years (ref) | 51.5% | 1.00 |
| 13–15 years | 54.6% | 0.95 (0.86, 1.04) |
| 16–17 years | 53.1% | 0.88 (0.80, 0.97)** |
| Youth gender | ||
| Male (ref) | 50.7% | 1.00 |
| Female | 55.0% | 1.10 (1.03, 1.18)** |
| Youth race/ethnicity | ||
| Hispanic | 62.9% | 0.84 (0.77, 0.92)*** |
| White (ref) | 51.0% | 1.00 |
| African American | 43.6% | 0.73 (0.65, 0.81)*** |
| Asian/Pacific Islander | 57.7% | 0.69 (0.54, 0.87)** |
| Other/unknown | 48.1% | 0.79 (0.71, 0.88)*** |
| Lives in MSA | ||
| No | 47.5% | 1.43 (1.29, 1.58)*** |
| Yes (ref) | 54.4% | 1.00 |
Notes: Measure is percentage of eligible ED visits where a follow-up visit with a mental health care provider occurred within 7 days of the ED visit, MSA indicates metropolitan statistical area;
† Significance indicates differences in measure scores between states;
‡ Odds ratio and significance logistic regression results examining differences in measure scores by youth characteristic, adjusted for state and all other youth characteristics in this table;*P < .05.**P < .01.***P < .001.ref indicates the reference group when testing for differences in measure scores by characteristic.
With respect to variability, health-plan level measure scores ranged from 0% to 100% with a median of 67.9%, a mean of 64.4%, and an interquartile range (IQR) of 43.4% to 88.8%. Hospital level scores ranged from 0% to 100% with a median, mean, and IQR of 46.2%, 46.6%, and 0% to 100%, respectively. With respect to reliability, the health-plan level and hospital-level ICCs were 0.30 and 0.36, respectively. Based on the Spearman-Brown formula, this means that 21 eligible visits per health plan or 16 eligible visits per hospital would be needed to achieve a reliability of .90, respectively.
Validation
Overall, 1436 (8.7%) of the eligible ED visits were followed by a return ED visit (for any reason) within 8 to 30 days and 414 (2.5%) were followed by an inpatient admission (for any reason) within 8 to 30 days; 121 (0.7%) were followed by a same-cause return ED visit within 8 to 30 days and 72 (0.4%) were followed by a same-cause inpatient admission within 8 to 30 days. Having a mental health care follow-up visit within 7 days after the index ED visit was not significantly associated with either 30-day ED return visits (odds ratio = 1.08; 95% confidence interval = 0.96, 1.21; P > .05), 30-day inpatient admissions (odds ratio = 1.07; 95% confidence interval = 0.87, 1.31; P > .05), 30-day same-cause ED return visits (odds ratio = 1.08; 95% confidence interval = 0.74, 1.58; P > .05), or 30-day same-cause inpatient admissions (odds ratio = 0.91; 95% confidence interval = 0.55, 1.50; P > .05).
Discussion
This paper describes the development and testing of a new quality measure that assesses mental health follow-up care for youth aged 5 to 17 years presenting to the ED with suicidal ideation or self-harm. Given the high prevalence of depression in this age group,
35
,CDC Injury Prevention & Control: Division of Violence Prevention. Available at:http://www.cdc.gov/violenceprevention/suicide/youth_suicide.html. Accessed August 4, 2016.
36
and the associated risk of suicide, the relatively low scores observed in our study should be of concern to clinicians and policymakers. Overall, only 53.5% of ED visits for suicidal ideation or self-harm resulting in a discharge to home were followed by a mental health care follow-up visit within 7 days, and this percentage was even lower (<40%) in 3 of the 4 states where it was assessed.Our results are relatively consistent with previous work which has shown moderate rates of follow-up in this population ranging from 39% to 87%.
6
, 7
, 8
,24
From a provider perspective, it is important to consider the barriers to accessing follow-up care that these youth and their caregivers may face. Caregiver concerns about lengthy evaluations, long waiting periods, billing procedures, provider availability, and time off work/school may contribute to lack of follow-up.37
,38
In addition, caregiver perceptions regarding the severity of their child's mental health conditions may hinder follow-up.39
Mental health care provider capacity may in part explain the wide range of measure performance across states. For example, the lowest measure score was observed in TN at 26.3% and TN, of the 4 states, also has the lowest mental health provider capacity (eg, 41 counties in TN versus less than 10 in each of the other 3 states have less than one psychologist per 10,000 children aged 0–17 years).40
,Rates of Mental and Behavioral Health Service Providers by County, 2015. Centers for Disease Control and Prevention. Available at:https://www.cdc.gov/childrensmentalhealth/stateprofiles-providers.html. Accessed May 4, 2021.
41
Certainly, lack of mental health provider capacity would limit the ability of providers and caregivers to ensure timely follow-up.Mental Health Care Health Professional Shortage Areas (HPSAs). Kaiser Family Foundation. Available at:https://www.kff.org/other/state-indicator/mental-health-care-health-professional-shortage-areas-hpsas/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Percent%20of%20Need%20Met%22,%22sort%22:%22desc%22%7D. Accessed May 4, 2021
Our examination of reliability at the health-plan and hospital-level demonstrates that excellent reliability can be achieved with a relatively moderate sample size ie, 21 per health plan or 16 per hospital, thus implying that this measure can usefully distinguish between health plans/hospitals with respect to performance. Future use of measure scores may be helpful in identifying high vs low measure performance and thus, help with targeting quality improvement interventions as needed.
Based on our conceptual framework and prior evidence showing decreased recurrence of suicidal ideation and suicide attempts,
11
,16
,17
we hypothesized that a mental health care follow-up visit within 7 days would be associated with lower odds of both 30-day ED return visits and inpatient admissions; however, we did not find an association between quality measure scores and these utilization outcomes (all-cause or same-case). There are a few possible explanations for this observed lack of association. First, while a notably high percentage (8.7%) of visits were followed by a return ED visit within 30 days, the utilization outcomes were relatively uncommon events eg, 2.5% for 30-day inpatient admissions, <1% for same-cause outcomes, making small effect sizes difficult to detect without a larger sample size. Second, it may be necessary to consider the use of additional outcome validation measures more closely linked to the potential benefits associated with meeting this quality metric, such as future suicidal ideation,9
longer-term ongoing mental health follow-up (eg, 90 or 180 days), improvement in Patient Health questionnaire (PHQ-9)42
scores, time to return to school, or school performance, though importantly, these are not available in claims data. Additional work is needed to better understand the relationship between receipt of appropriate follow-up care and outcomes including those described above and future hospital-based utilization in this population of youth.Our results demonstrating a lower likelihood of follow-up within 7 days among non-White youth compared to White youth are concerning and add to existing work showing that minority youth are less likely to receive appropriate mental health care.
43
, 44
, 45
, 46
Such racial/ethnic disparities in mental health care may be driven by a multitude of factors including differences in past experiences with the quality of care received from mental health providers, access to care, stigma, distrust in the mental healthcare system, and/or provider behavior.U.S. Department of Health and Human Services
Mental Health: Culture, Race, and Ethnicity—A Supplement to Mental Health: A Report of the Surgeon General.
Mental Health: Culture, Race, and Ethnicity—A Supplement to Mental Health: A Report of the Surgeon General.
U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services,
Rockville, MD2001
47
, 48
, 49
These findings indicate that it is important for providers to consider the development and implementation of interventions to increase access to services and outreach for these youth and their families.Interestingly, adjusted analyses highlighted significantly higher quality scores among youth living outside an MSA. Previous work examining access to and utilization of mental health care services has demonstrated the opposite, with individuals living in more urban settings having more access and utilization.
50
, 51
, 52
One possible explanation for our finding is the growing availability of telehealth services for mental health care.53
,54
If youth living outside an MSA are more likely to engage in telehealth services, that may explain these findings. Importantly, state laws and Medicaid policies regarding telemedicine vary dramatically from state to state. While the utilization of telehealth has substantially increased during the COVID-19 pandemic, we note that this study was conducted in 2014–2016. According to a 2018 report, CA was identified as having progressive telemedicine Medicaid program policies, while TN was identified as moderate, and both PA and SC were identified as having restrictive telemedicine policies.55
It is possible that differences in the availability of telehealth programs, telehealth utilization, and telehealth policies may partially explain our observed differences by state and/or our MSA findings. Unfortunately, it was not possible to differentiate between in-person and telehealth follow-up visits in our study, so future research in this area is warranted. If in fact youth in rural areas are more likely than those in urban areas to engage in telemedicine mental health services, this may lead to reductions in suicide rates in rural areas which historically tend to be higher than in urban areas.State Telehealth Laws and Medicaid Policies: 50-State Survey Findings (2018). Manatt Health. Available at:https://www.manatt.com/insights/newsletters/manatt-on-health/state-policy-levers-for-telehealth-50-state-surve. Accessed February 3, 2021.
56
,57
The development and testing of this new quality measure is an important first step toward addressing the dearth of quality measures available to assess pediatric hospital-based mental health care. Attempting suicide is a leading predictor of subsequent suicide attempts.
58
, 59
, 60
, 61
Thus, implementing best practices for youth presenting to the ED with suicidal ideation or self-harm has the potential to improve the care of a large proportion of those at risk for future suicide attempts. Previously established guidelines have explicitly recommended mental health follow-up care for this population.19
,62
This study has some limitations. First, since this measure was only tested in 4 states among youth with Medicaid coverage, our understanding of the implementation and performance on this measure for youth in other states, with other types of insurance, and with different resources and demographic compositions is limited. Second, we may not have identified all eligible youth or all follow-up visits due to imperfect sensitivity/specificity of the codes used, inconsistent or incorrect use of codes, or potential errors or missingness in the administrative data. Though we parallel previous work in our selection of diagnosis codes to identify youth with suicidal ideation or self-harm, there are limitations to relying on diagnosis codes alone (as opposed to in combination with medical record abstraction and/or natural language processing).
29
,30
Similarly, it is possible that there is incorrect or inconsistent (eg, across states) use of codes to identify mental health follow-up visits. Third, we acknowledge that the content and quality of follow-up visits likely varies, and the administrative data measure we developed does not allow us to assess this variation in care or how it may influence outcomes. Fourth, though we adjust for individual characteristics including race/ethnicity, age, and gender in our validation analysis, there may be important unmeasured confounders such as severity of illness not available in administrative data. Lastly, as noted earlier, given the low rates for some of the utilization outcomes (eg, <1% for same-cause outcomes), our study may have been underpowered to detect small effect sizes. In addition, quality measures calculated using administrative claims data are difficult to monitor in real-time since claims may take weeks or months after the index visit to be finalized and made available for quality measurement and improvement purposes. However, while more readily available in real-time, quality measures requiring medical record abstraction are comparatively more resource intensive and burdensome.In conclusion, this new quality measure evaluates a key aspect of ED care for youth with suicidal ideation or self-harm and may facilitate assessing and improving quality of care for this vulnerable population. The development and testing of this measure is an important first step toward understanding the quality of care received by youth with suicidal ideation or self-harm in the ED setting. This measure could be used in the future to examine and monitor rates of timely follow-up within a hospital, health plan or state over time and/or to examine changes in timely follow-up rate that may occur following mental-health related quality improvement interventions in a hospital or health plan. Though our results reflect no association between measure performance and hospital-based reutilization outcomes over a 30-day follow-up period, previous findings have established that timely follow-up after an ED visit for suicidal ideation or self-harm is associated with a longer period of time to the next ED visit, and lower rates of subsequent suicidal ideation and deliberate self-harm.
7
,11
, 12
, 13
, 14
, 15
, 16
, 17
Nonetheless, future work is needed to establish this measure's predictive validity using more prevalent outcomes, eg, recurrence of suicidal ideation.Acknowledgments
Funding: This study was done under funding from a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare and Medicaid Services, grant number U18HS025291. Research reported in this publication was also supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002537. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in the study design; in the collection, analysis, or interpretation of data; in the writing of the report; nor in the decision to submit the article for publication.
The views expressed in this article are those of the authors, and no official endorsement by the Agency for Healthcare Research and Quality (AHRQ), the Centers for Medicare and Medicaid Services (CMS), or the Department of Health and Human Services (DHHS) is intended or should be inferred.
This article is published as part of a supplement sponsored by the US Department of Health and Human Services, the Centers for Medicare and Medicaid Services, and the Agency for Healthcare Research and Quality.
Appendix
ICD-10 codes used to identify ED visits for suicidal ideation or self-harm
T1491, R45851, X71.8XXA, X71.9XXA, X72.XXXA, X73.0XXA, X73.1XXA, X73.2XXA, X73.9XXA, X75.XXXA,X74.01XA, X74.02XA, X74.9XXA, X78.9XXA,X80.XXXA,X81.8XXA, X76.XXXA,X77.2XXA, X83.2XXA, X83.1XXA, X82.8XXA, X83.0XXA, X83.8XXA, X79, X76, X77.9, X77.2, X77.3, X77.8, X77.0, X77.1, X77.0, X83.2, X82.0, X82.8, X82.1, X82.2, X83.0, X78.9, X78.2, X78.0, X78.1, X78.8, X71.9, X71.0, X71.3, X71.8, X71.1, X71.2, X83.1, X75, X76, X74.9, X74.01, X72, X73.1, X73.9, X73.8, X73.2, X73.0, X74.8, X83.8, X81.8, X77.0, X81.0, X81.1, X80
Current Procedural Terminology (CPT) codes used to identify mental health care follow-up
CPT codes:
- 2014: 90785, 90791-90792, 90832-90834, 90836-90838, 90839-90840, 90845-90847, 90849, 90853, 90863, 90865, 90867-90870, 90875-90876, 90880, 96101, 96103, 96153, 96154,97770, 96155, 96120, 96151
- 2016: added 99354, 99355, 99415, 99416, 98968, 99443
- HCPC codes: H2010, H2011, H2015, S0201, S9480
References
CDC. Ten Leading Causes of Death by Age Group, United States –2018. Available at: https://www.cdc.gov/injury/images/lc-charts/leading_causes_of_death_by_age_group_2018_1100w850h.jpg. Accessed June 1, 2020.
CDC. Youth Risk Behavior Survey Data Summary and Trends Report 2009-2019. Available at:https://www.cdc.gov/healthyyouth/data/yrbs/pdf/YRBSDataSummaryTrendsReport2019-508.pdf. Accessed February 2, 2021.
- Suicidal attempts and ideation among children and adolescents in US emergency departments, 2007-2015.JAMA Pediatr. 2019; 173: 598-600
Pallone F. Children's Health Insurance Program Reauthorization Act of 2009. Office of the Legislative Council 111th Congress. Public Law No. 111–113.
Strokoff SL. Patient Protection and Affordable Care Act of 2010. Office of the Legislative Council 111th Congress. Public Law No. 111–148: 288–291.
- Suicidal children and adolescents with first emergency room presentations: predictors of six-month outcome.J Am Acad Child Adolesc Psychiatry. 2001; 40: 580-587
- Mental health follow-up among adolescents with suicidal behaviors after emergency department discharge.Arch Suicide Res. 2013; 17: 323-334
- Emergency room evaluation of the adolescent who attempts suicide: compliance with follow-up.J Adolesc Health Care. 1983; 4: 106-108
- Association of timely outpatient mental health services for youths after psychiatric hospitalization with risk of death by suicide.JAMA Network Open. 2020; 3 (e2012887-e2012887)
- Outpatient care of young people after emergency treatment of deliberate self-harm.J Am Acad Child Adolesc Psychiatry. 2012; 51 (213-222. e211)
- A review of evidence-based follow-up care for suicide prevention: where do we go from here?.Am J Prev Med. 2014; 47: S209-S215
- Cognitive therapy for the prevention of suicide attempts: a randomized controlled trial.JAMA. 2005; 294: 563-570
- Treatment for adolescents following a suicide attempt: results of a pilot trial.J Am Acad Child Adolesc Psychiatry. 2005; 44: 113-120
- Feasibility of dialectical behavior therapy for suicidal adolescent inpatients.J Am Acad Child Adolesc Psychiatry. 2004; 43: 276-282
- Effectiveness of brief intervention and contact for suicide attempters: a randomized controlled trial in five countries.Bull World Health Organ. 2008; 86: 703-709
- A follow-up program for suicide attempters: Evaluation of effectiveness.Suicide Life Threat Behav. 1977; 7: 17-30
- Effect of telephone contact on further suicide attempts in patients discharged from an emergency department: randomised controlled study.BMJ. 2006; 332: 1241-1245
- Practitioner review: self-harm in adolescents.J Child Psychol Psychiatry. 2012; 53: 337-350
- Practice parameter for the assessment and treatment of children and adolescents with suicidal behavior.J Am Acad Child Adolesc Psychiatry. 2001; 40: 24S-51S
- Trial of Therapeutic Assessment in London: randomised controlled trial of Therapeutic Assessment versus standard psychosocial assessment in adolescents presenting with self-harm.Arch Dis Child. 2011; 96: 148-153
- An emergency department intervention for linking pediatric suicidal patients to follow-up mental health treatment.Psychiatr Serv. 2011; 62: 1303-1309
- Follow-up care after emergency department visits for mental and substance use disorders among Medicaid beneficiaries.Psychiatr Serv. 2017; 68: 566-572
- A test of outreach and drop-in linkage versus shelter linkage for connecting homeless youth to services.Prev Sci. 2016; 17: 450-460
- ED visits and readmissions after follow-up for mental health hospitalization.Pediatrics. 2020; 145
- Development of new quality measures for hospital-based care of suicidal youth.Acad Pediatr. 2018; 18: 248-255
- Practice parameter for the assessment and treatment of children and adolescents with suicidal behavior.J Am Acad Child Adolesc Psychiatry. 2001; 40: 24S-51S
- Practice parameter for the assessment and treatment of children and adolescents with depressive disorders.J Am Acad Child Adolesc Psychiatry. 2007; 46: 1503-1526
- Hospital-based quality measures for pediatric mental health care.Pediatrics. 2018; e20173554
- Demographic characteristics of individuals admitted to the Hospital for Suicidal Ideation in the emergency department.Am J Emerg Med. 2016; 34: 1174-1175
- A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data.Pharmacoepidemiol Drug Saf. 2012; 21: 174-182
- Suicide-related behaviors in older patients with new anti-epileptic drug use: data from the VA hospital system.BMC Med. 2010; 8: 1-7
- Introduction to Measurement Theory.Waveland Press, 2001
National Quality Forum. Measure Evaluation Criteria and Guidance for Evaluating Measures for Endorsement. Available at:http://www.qualityforum.org/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=86084. Accessed June 6, 2018.
U.S. Census Bureau, 2010 Census. American FactFinder, Race and Hispanic or Latino Origin: 2010. Available at:https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=CF. Accessed July 14, 2017.
CDC Injury Prevention & Control: Division of Violence Prevention. Available at:http://www.cdc.gov/violenceprevention/suicide/youth_suicide.html. Accessed August 4, 2016.
- Common and costly hospitalizations for pediatric mental health disorders.Pediatrics. 2014; 133: 602-609
- Compliance in health care.Johns Hopkins University Press, Baltimore, MD1979
- Factors related to adherence to medication regimens in pediatric seizure patients.J Pediatr Psychol. 1990; 15: 543-555
- Brief cognitive-behavioral treatment for adolescent suicide attempters and their families.J Am Acad Child Adolesc Psychiatry. 1994; 33: 508-517
Rates of Mental and Behavioral Health Service Providers by County, 2015. Centers for Disease Control and Prevention. Available at:https://www.cdc.gov/childrensmentalhealth/stateprofiles-providers.html. Accessed May 4, 2021.
Mental Health Care Health Professional Shortage Areas (HPSAs). Kaiser Family Foundation. Available at:https://www.kff.org/other/state-indicator/mental-health-care-health-professional-shortage-areas-hpsas/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Percent%20of%20Need%20Met%22,%22sort%22:%22desc%22%7D. Accessed May 4, 2021
- The PHQ-9: validity of a brief depression severity measure.J Gen Intern Med. 2001; 16: 606-613
- Disparities in behavioral health diagnoses: considering racial and ethnic youth groups.N C Med J. 2020; 81: 126-129
- Racial/ethnic differences in mental health service use among adolescents with major depression.J Am Acad Child Adolesc Psychiatry. 2011; 50: 160-170
Smedley BD, Stith AY, Nelson AR. Committee on understanding and eliminating racial and ethnic disparities in health care. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care.2003:180-191.
- Mental Health: Culture, Race, and Ethnicity—A Supplement to Mental Health: A Report of the Surgeon General.U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, Rockville, MD2001
- Racial and ethnic differences in mental illness stigma and discrimination among Californians experiencing mental health challenges.Rand Health Quart. 2017; 6
- Stigma for common mental disorders in racial minorities and majorities a systematic review and meta-analysis.BMC Public Health. 2020; 20: 1-20
Merino Y, Adams L, Hall WJ. Implicit bias and mental health professionals: priorities and directions for research. In: Am Psychiatric Assoc; 2018.
- Rurality and mental health treatment.Admin Policy Mental Health Mental Health Serv Res. 2007; 34: 255-267
- Patterns of care for rural and urban children with mental health problems.in: Maine Rural Health Research Center, Working Paper #49. 2013
- Integrated care in rural areas.Rural Public Health: Best Pract Prevent Models. 2014; : 67-84
- A statewide child telepsychiatry consult system yields desired health system changes and savings.Telemed e-Health. 2015; 21: 533-537
- The partnership access line: evaluating a child psychiatry consult program in Washington State.JAMA Pediatrics. 2013; 167: 162-168
State Telehealth Laws and Medicaid Policies: 50-State Survey Findings (2018). Manatt Health. Available at:https://www.manatt.com/insights/newsletters/manatt-on-health/state-policy-levers-for-telehealth-50-state-surve. Accessed February 3, 2021.
- Widening rural-urban disparities in youth suicides, United States, 1996-2010.JAMA Pediatrics. 2015; 169: 466-473
- Increasing rural–urban gradients in US suicide mortality, 1970–1997.Am J Public Health. 2002; 92: 1161-1167
- Suicidal behavior in children younger than twelve: a diagnostic challenge for emergency department personnel.Acad Emerg Med. 2007; 14: 810-818
- Emergency department management of suicidal adolescents.Ann Emerg Med. 2004; 43: 452-460
- Suicide attempts among formerly hospitalized adolescents: A prospective naturalistic study of risk during the first 5 years after discharge.J Am Acad Child Adolesc Psychiatry. 1999; 38: 660-671
- Parameters of suicidal crises vary as a function of previous suicide attempts in youth inpatients.J Am Acad Child Adolesc Psychiatry. 2000; 39: 876-880
- Mental health treatment for self-injurious behaviors: clinical practice guidelines for children and adolescents in the emergency department.National Acad Pract Forum. 1999; (1999): 35-40
Article info
Publication history
Accepted:
March 5,
2021
Received:
December 10,
2020
Footnotes
The authors have no conflicts of interest to disclose.
Identification
Copyright
Copyright © 2021 by Academic Pediatric Association
