Advertisement

Screening for Adolescent Problematic Internet Use: Validation of the Problematic and Risky Internet Use Screening Scale (PRIUSS)

      Abstract

      Objective

      Problematic Internet use (PIU) is an emerging health concern that lacks screening measures validated for use with adolescents and young adults. This study aimed to validate the Problematic and Risky Internet Use Screening Scale (PRIUSS) for use with older adolescents and to increase its clinical utility by determining scoring guidelines and assessing the relationship between PIU and other mental health conditions.

      Methods

      This cross-sectional survey study took place at a large, public Midwestern university among 330 older adolescents aged 18 to 25 years. Confirmatory factor analysis and Spearman's correlations were used to assess the PRIUSS' structural and construct validity, respectively. A risk-based scoring cutoff was estimated using a Bayesian latent class modeling approach to computing a receiver operating characteristic curve.

      Results

      The confirmatory factor analysis indices for the 3-factor model indicated an acceptable fit (goodness-of-fit index 0.89, root mean square error of approximation 0.07). A cutoff of 25 (sensitivity 0.80, 95% confidence interval [CI] 0.47–0.99; specificity 0.79, 95% CI 0.73–0.84) is proposed for identifying those at risk for PIU. Participants at risk for PIU were at significantly greater odds of also reporting symptoms of attention-deficit/hyperactivity disorder (odds ratio [OR] 2.36 95% CI 1.21–4.62, P = .009), depression (OR 3.25, 95% CI 1.65–6.42, P = .008), and social anxiety (OR 3.77, 95% CI 2.06–6.89, P < .000).

      Conclusions

      The PRIUSS demonstrated validity as a PIU screening instrument for adolescents and young adults. Screening for PIU may also help to identify those at high reciprocal risk for other mental health conditions.

      Keywords

      What's New
      Problematic Internet use (PIU) affects approximately 4% to 6% of US adolescents and young adults. This study validates a clinical screening measure, the Problematic and Risky Internet Use Screening Scale (PRIUSS), and provides scoring guidelines for its use in interpreting older adolescents' risk for PIU.
      Problematic Internet use (PIU) is an emerging health concern among US adolescents and young adults (AYAs). A study of high school students throughout Connecticut estimated its prevalence at 4%.
      • Liu T.C.
      • Desai R.A.
      • Krishnan-Sarin S.
      • et al.
      Problematic Internet use and health in adolescents: data from a high school survey in Connecticut.
      Moreover, 3 recent studies sampling from AYA university students estimated the prevalence between 4% and 6%,
      • Christakis D.A.
      • Moreno M.M.
      • Jelenchick L.
      • et al.
      Problematic Internet usage in US college students: a pilot study.
      • Derbyshire K.L.
      • Lust K.A.
      • Schreiber L.R.
      • et al.
      Problematic Internet use and associated risks in a college sample.
      • Yates T.M.
      • Gregor M.A.
      • Haviland M.G.
      Child maltreatment, alexithymia, and problematic Internet use in young adulthood.
      which is a rate comparable to other conditions often addressed clinically among this population such as depression.
      • Lewinsohn P.M.
      • Rohde P.
      • Seeley J.R.
      Major depressive disorder in older adolescents: prevalence, risk factors, and clinical implications.
      PIU has been associated with poor academic performance, stress, and fewer positive health behaviors.
      • Derbyshire K.L.
      • Lust K.A.
      • Schreiber L.R.
      • et al.
      Problematic Internet use and associated risks in a college sample.
      Longitudinal studies have also suggested bidirectional relationships between PIU and other mental health conditions such as depression.
      • Yen J.Y.
      • Ko C.H.
      • Yen C.F.
      • et al.
      The comorbid psychiatric symptoms of Internet addiction: attention deficit and hyperactivity disorder (ADHD), depression, social phobia, and hostility.
      • Ko C.H.
      • Yen J.Y.
      • Chen C.S.
      • et al.
      Predictive values of psychiatric symptoms for Internet addiction in adolescents: a 2-year prospective study.
      • Lam L.T.
      • Peng Z.W.
      Effect of pathological use of the Internet on adolescent mental health.
      Although definitive diagnostic criteria for PIU are yet to be established, Internet use addiction disorder is currently included in the appendix of the Diagnostic and Statistical Manual of Mental Disorders, version 5, as a disorder requiring further study. Irrespective of its official designation, there are undoubtedly a substantial number of youth for whom Internet use has adverse effects and who may thus benefit from intervention. Given that online activities now often begin early in childhood, primary care pediatric clinicians are uniquely positioned to help prevent the development of PIU.
      • Jelenchick L.A.
      • Christakis D.A.
      Problematic Internet use during adolescence and young adulthood.
      Further, PIU appears to follow a similar developmental trajectory to other risk behaviors such as bullying or tobacco or alcohol use, all of which are commonly screened for and addressed during AYA wellness visits.
      Thus, the development of a screening tool specific to PIU during adolescence and young adulthood and suitable for clinical settings is an important step in designing primary prevention efforts. The majority of existing rating scales for PIU symptoms have been adapted from diagnostic criteria for disparate psychiatric disorders and developed for adults.
      • Moreno M.A.
      • Jelenchick L.
      • Cox E.
      • et al.
      Problematic Internet use among US youth: a systematic review.
      • Young K.S.
      Caught in the Net: How to Recognize the Signs of Internet Addiction—And a Winning Strategy for Recovery.
      A recent systematic review of PIU highlighted the diversity of measurement tools that lack validation and the resultant disparate prevalence estimates, and concluded that a uniform and validated approach to measurement is a necessary next step to understanding prevalence of PIU among AYAs.
      • Moreno M.A.
      • Jelenchick L.
      • Cox E.
      • et al.
      Problematic Internet use among US youth: a systematic review.
      Moreno and colleagues
      • Moreno M.A.
      • Jelenchick L.A.
      • Christakis D.A.
      Problematic Internet use among older adolescents: a conceptual framework.
      described the first data-driven conceptual framework for PIU specific to AYAs, including a definition of PIU as, “Internet use that is risky, excessive or impulsive in nature leading to adverse life consequences, specifically physical, emotional, social or functional impairment.” Further, the framework identifies 7 core constructs of PIU: 3 describing the nature of PIU (“risky Internet use” “impulsive Internet use,” “Internet use dependency”); 3 describing its impact on AYA health and well-being (“physical impairment,” “emotional impairment,” “social/functional impairment”); and a final construct describing factors that may predispose AYAs to PIU (“psychosocial risk factors”). In a follow-up study, Jelenchick and colleagues
      • Jelenchick L.A.
      • Eickhoff J.
      • Christakis D.A.
      • et al.
      The Problematic and Risky Internet Use Screening Scale (PRIUSS) for adolescents and young adults: scale development and refinement.
      adapted the comprehensive list of characteristics, behaviors, and symptoms describing each of the framework's construct into an item pool for use in a scale development study. Specifically, 75 of the descriptive terms were assessed using the psychometric methods for scale development described by DeVellis.
      • DeVellis R.F.
      Scale Development: Theory and Applications.
      The end product, the Problematic and Risky Internet Use Screening Scale (PRIUSS), is an 18-item risk-based screening scale for PIU with questions organized into 3 subscales: Social Impairment, Emotional Impairment, and risky/impulsive Internet use. Figure 1 summarizes the process used to integrate Moreno and colleagues' conceptual framework into the development of the PRIUSS.
      Figure thumbnail gr1
      Figure 1Integration of the theoretical framework with scale development methods utilized during construction of Problematic and 300 Risky Internet Use Screening Scale (PRIUSS).
      Although the PRIUSS underwent an empirical development process and demonstrated strong reliability and content validity, further validation is needed to support its use in AYA health settings. Thus, the aims of the current study were 3-fold: 1) to assess the PRIUSS’ structural validity by confirming its factor structure, 2) to extend the construct validation of the PRIUSS, including establishing convergent and divergent validity with other behavioral and mental health conditions, and 3) to determine preliminary scoring guidelines to facilitate its clinical utility.

      Methods

      Subjects and Setting

      This cross-sectional survey study was conducted between January 2012 and June of 2013. Approval of the study was granted by the institutional review board at the University of Wisconsin. Participants were older adolescents aged 18 to 25 years recruited from undergraduate students enrolled in a nutritional sciences course at a public university located in the Midwest, a course that has both a high enrollment number and fulfills general education requirements for a variety of programs and majors.

      Procedure

      Students were initially invited to participate in the study through an in-class announcements; a posting was also placed on course Web sites. All participants provided written consent before completing the survey on a secure online site. The survey contained standard demographic questions and general Internet use questions. Internet use questions included the respondents' average number of times using the Internet per day, total daily hours spent online, number of hours spent on work or school activities, and number of hours spent on recreational activities. When responding to these questions, participants were instructed to think of their use over the last month and include time spent on any Internet application, whether using a computer or mobile device. Time spent text messaging was not included unless text messages were being used to interact with an online application. Finally, the survey contained the PRIUSS, as well as behavioral and mental health measures intended to assess the PRIUSS’ construct validity. Students who completed the survey received a $10 gift card.

      Measures

      Problematic and Risky Internet Use Screening Scale (PRIUSS)

      The PRIUSS has 18 items and 3 subscales: 1) Social Impairment (items 1–6), which assesses the impact of Internet use on both off-line and online social interactions; 2) Emotional Impairment (items 7–11), which assesses degree of emotional attachment to Internet use; and 3) Risky/Impulsive Internet Use (items 12–18), which assess salient problematic behaviors regarding Internet use.
      • Jelenchick L.A.
      • Eickhoff J.
      • Christakis D.A.
      • et al.
      The Problematic and Risky Internet Use Screening Scale (PRIUSS) for adolescents and young adults: scale development and refinement.
      The full text of the PRIUSS as it was presented to participants is available as an Online Supplementary Figure 1.

      Adult ADHD Self-Report Screener (ASRS)

      The ASRS is a 6-item scale adapted from the Diagnostic and Statistical Manual of Mental Disorders, version 4, symptoms for attention-deficit/hyperactivity disorder (ADHD) that has been validated in both adolescents and adults.
      • Adler L.A.
      • Newcorn J.H.
      Administering and evaluating the results of the adult ADHD Self-Report Scale (ASRS) in adolescents.
      • Kessler R.C.
      • Adler L.A.
      • Gruber M.J.
      • et al.
      Validity of the World Health Organization Adult ADHD Self-Report Scale (ASRS) screener in a representative sample of health plan members.
      On the basis of existing recommendations, at least 4 of the 6 symptoms had to be positively endorsed to classify participants reporting symptoms consistent with ADHD.

      Patient Health Questionnaire (PHQ-9)

      The PHQ-9 is 9-item self-report scale for frequency of depression symptoms experienced in the last 2 weeks that has been validated for use in adolescents and adults.
      • Richardson L.P.
      • McCauley E.
      • Grossman D.C.
      • et al.
      Evaluation of the Patient Health Questionnaire-9 Item for detecting major depression among adolescents.
      On the basis of existing recommendations, scores of ≥11 were used to classify participants reporting symptoms consistent with major depressive disorder.
      • Richardson L.P.
      • McCauley E.
      • Grossman D.C.
      • et al.
      Evaluation of the Patient Health Questionnaire-9 Item for detecting major depression among adolescents.

      Social Phobia Inventory (SPIN)

      The SPIN is a 17-item self-report scale for social phobia symptoms that has been validated for use in adolescents and adults.
      • Johnson H.S.
      • Inderbitzen-Nolan H.M.
      • Anderson E.R.
      The Social Phobia Inventory: validity and reliability in an adolescent community sample.
      • Connor K.M.
      • Davidson J.R.
      • Churchill L.E.
      • et al.
      Psychometric properties of the Social Phobia Inventory (SPIN): new self-rating scale.
      On the basis of existing recommendations, scores of ≥21 were used to classify participants reporting symptoms consistent with social anxiety disorder.
      • Johnson H.S.
      • Inderbitzen-Nolan H.M.
      • Anderson E.R.
      The Social Phobia Inventory: validity and reliability in an adolescent community sample.

      Online Cognition Scale (OCS)

      The OCS is a 10-item scale for problematic Internet-related cognitions; example items include: “I am most comfortable online” and “The Internet is more ‘real’ than real life.”
      • Jia R.
      • Jia H.H.
      Factorial validity of problematic Internet use scales.
      • Davis R.A.
      A cognitive–behavioral model of pathological Internet use.
      • Davis R.A.
      • Flett G.L.
      • Besser A.
      Validation of a new scale for measuring problematic Internet use: implications for pre-employment screening.
      Higher scores indicated increased problematic cognitions.

      Marlowe-Crowne Social Desirability Scale Short Form (SDS)

      The SDS is a 13-item adaptation of the full 33-item scale assessing social desirability, a trait that is often negatively associated with nonnormative behaviors.
      • Reynolds W.M.
      Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale.
      Items are responded to as true or false; higher overall scores indicate increased social desirability tendency.

      Brief Sensation Seeking Scale (BSSS)

      The BSSS is a self-report scale for sensation seeking, a personality-based risk factor for problem behavior, which has been validated for use with AYAs.
      • Hoyle R.H.
      • Stephenson M.T.
      • Palmgreen P.
      • et al.
      Reliability and validity of a brief measure of sensation seeking.
      • Stephenson M.T.
      • Velez L.F.
      • Chalela P.
      • et al.
      The reliability and validity of the Brief Sensation Seeking Scale (BSSS-8) with young adult Latino workers: implications for tobacco and alcohol disparity research.
      Total scores range from 0 to 32, and higher scores indicate increased sensation seeking.

      Statistical Analysis

      Statistical analyses were performed by SAS 9.2 (SAS Institute, Cary, NC), R 2.13 (http://www.r-project.org), and OpenBUGS software.
      • Lunn D.
      • Spiegelhalter D.
      • Thomas A.
      • et al.
      The BUGS project: evolution, critique and future directions.
      All P values were 2-sided, and P < .05 was used to indicate statistical significance. Descriptive data were summarized as frequencies and percentages or means and standard deviations. Because of the limited number of nonwhite participants, during analysis, the categories for race were collapsed into a binary white or nonwhite variable. Participants' Internet use was categorized on the basis of their self-reported average number of daily recreational hours; participants were categorized as either average users (<6 recreational hours per day) or high users (≥6 recreational hours per day). A summary of the statistical approach utilized for the scale validation is provided below; additional details can be found in the Online Supplementary Data.

      Structural Validity

      Cronbach's alphas were calculated to examine the internal consistency reliability of the 3 subscales. Pearson's correlation coefficient (r) was calculated to examine the correlation between each subscale. A confirmatory factor analysis was conducted to test the PRIUSS' 3-factor structure. The goodness-of-fit index and root mean square error of approximation estimate were used as the primary measures for evaluating model fit. Models with goodness-of-fit indices between 0.9 and 1 and root mean square error of approximation values between 0.10 and 0.08 are considered a good fit.
      • Stevens J.P.
      Applied Multivariate Statistics for the Social Sciences.
      • Brown M.W.
      • Cudeck R.
      Alternative ways of assessing model fit.

      Construct Validity

      To evaluate the scale's construct validity, nonparametric Spearman's correlation was used to determine associations between PRIUSS scores and the theoretically related variables.

      Scale Cutoff for Risk-Based Assessment

      To estimate a cutoff for the PRIUSS that could be used to discriminate between those at risk for PIU from those not at risk, a receiver operating characteristic curve was computed using a Bayesian latent class modeling approach. Participants' Internet use categorization (average vs high use) was incorporated as an additional model parameter based on related studies of PIU that identified ≥6 daily recreational hours as the most highly discriminating criterion for excessive use.
      • Tao R.
      • Huang X.
      • Wang J.
      • et al.
      Proposed diagnostic criteria for Internet addiction.
      Of the different model specifications suggested to address the problem of an absent reference standard,
      • Reitsma J.B.
      • Rutjes A.W.
      • Khan K.S.
      • et al.
      A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard.
      we chose a Bayesian latent class model as a useful tool for estimating sensitivity and specificity of a new screening test in the absence of a reference standard.
      • Joseph L.
      • Gyorkos T.W.
      • Coupal L.
      Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard.
      Sensitivity and specificity estimates of the PRIUSS were reported along with the corresponding 95% confidence intervals (CI).
      Finally, logistic regression analysis was used to assess associations between PRIUSS risk category and the probability of experiencing ADHD, depression, or social anxiety symptoms, with race included as a cofactor. Odds ratios (ORs) and the corresponding 95% CIs were computed to compare the proportions of participants reporting significant mental health symptoms between the PRIUSS scoring categories.

      Results

      Participants

      A total of 330 participants completed the survey (50% response rate). Participants reported on average 5.0 (SD 3.0) total daily hours of Internet use, and 5.8% (n = 19) reported >6 hours of daily recreational use and were categorized as a high user. PRIUSS scores ranged from 0 to 49; the median score was 17. Nonwhite participants scored higher on the PRIUSS overall (mean 22.5, SD 11.5 versus mean 17.6, SD 9.9, P = .004). There were no significant differences by sex (P = .700), year in school (P = .440), or major field of study (P = .560). Detailed demographic information, as well as average scores and categorical distributions for the PRIUSS and other health measurements, are presented in Table 1.
      Table 1Demographic Information and Distributions and Scores Across Descriptive Variables for 330 Study Participants
      CharacteristicVariableValue
      Data are presented as n (%) or mean (SD).
      Age19.7 (1.9)
      SexFemale221 (67%)
      Male109 (33%)
      RaceAfrican American/black10 (3%)
      Asian/Asian American30 (9%)
      White273 (83%)
      Hispanic/Latino9 (3%)
      Native American/Alaskan Native3 (1%)
      Multiracial/unspecified5 (2%)
      Major field of studyAgricultural, biological, environmental sciences42 (13%)
      Business63 (19%)
      Engineering and physical sciences37 (11%)
      Health sciences51 (15%)
      Humanities44 (13%)
      Social sciences37 (11%)
      Education14 (4%)
      Undeclared/unspecified42 (13%)
      Year in schoolFreshman168 (51%)
      Sophomore79 (24%)
      Junior51 (15%)
      Senior32 (10%)
      Daily Internet hoursWork or school related2.3 (2.0)
      Recreational2.6 (1.7)
      PRIUSSOverall18.5 (10.3)
      Social Impairment subscale4.5 (3.6)
      Emotional Impairment subscale4.9 (3.9)
      Risky/Impulsive Use subscale9.1 (5.2)
      ASRSNo ADHD272 (82%)
      ADHD58 (18%)
      PHQ-9No depression157 (48%)
      Mild depression112 (34%)
      Moderate depression39 (12%)
      Moderately severe depression18 (5%)
      Severe depression4 (1%)
      SPINNo social phobia243 (74%)
      Mild social phobia48 (155)
      Moderate social phobia25 (8%)
      Severe social phobia14 (4%)
      ADHD indicates attention-deficit/hyperactivity disorder; ASRS, Adult ADHD Self-Report Screener; PHQ-9, Patient Health Questionnaire-9; PRIUSS, Problematic and Risky Internet Use Screening Scale; and SPIN, Social Phobia Inventory.
      Data are presented as n (%) or mean (SD).

      PIU Symptoms

      Figure 2 summarizes the percentage of participants falling within each response category (never to very often) for the symptom described by each PRIUSS question. Overall, participants endorsed greater symptom frequencies for items in the risky and impulsive use subscale; over half of participants described these symptoms as occurring sometimes, often, or very often. The most commonly reported symptoms within the Social and Emotional Impairment subscales were “choosing to socialize online instead of in person” and “feeling vulnerable when away from the Internet,” respectively.
      Figure thumbnail gr2
      Figure 2Percentage of participants falling within each response category for each problematic Internet use symptom assessed by Problematic and Risky Internet Use Screening Scale (PRIUSS).

      PRIUSS Structural Validity

      Cronbach's α coefficients were 0.81, 0.85, and 0.85 for the Social Impairment, Emotional Impairment, and Risky/Impulsive Use subscales, respectively. The factor loadings and subscale correlations for the 3-factor model are depicted in an Online Supplementary Figure 2. The indices estimated for the 3-factor model indicated an acceptable fit for the data (goodness-of-fit index 0.89, root mean square error of approximation 0.07). The majority of items demonstrated strong factor loadings (range 0.61–0.82). A single item from the Social Consequences subscale, item 1, exhibited a weaker factor loading (0.31).

      PRIUSS Construct Validity

      Correlations between PRIUSS scores and the theoretically relevant variables are presented in Table 2. There were significant positive correlations between PRIUSS scores and recreational Internet use, as well as all 3 mental health measures and the measure for problematic online cognitions. The weakest correlations, which were also largely nonsignificant, were seen with the social desirability and sensation-seeking measures.
      Table 2Correlations Between Problematic and Risky Internet Use Screening Scale (PRIUSS) and Theoretically Related Constructs
      ScaleDaily Internet Use HoursMental Health MeasuresOCSSDSBSSS
      TotalSchool/WorkRecreationalASRSPHQ-9SPIN
      PRIUSS0.302
      P < .001.
      0.125
      P < .05.
      0.378
      P < .001.
      0.405
      P < .001.
      0.467
      P < .001.
      0.470
      P < .001.
      0.663
      P < .001.
      −0.186
      P < .05.
      0.002
      Subscales
      Social indicates Problematic and Risky Internet Use Screening Scale—Social Impairment subscale; Emotional, Problematic and Risky Internet Use Screening Scale—Emotional Impairment subscale; and Risky/Impulsive, Problematic and Risky Internet Use Screening Scale—Risky/Impulsive Internet Use subscale.
       Social0.227
      P < .001.
      0.127
      P < .05.
      0.229
      P < .001.
      0.268
      P < .001.
      0.346
      P < .001.
      0.486
      P < .001.
      0.469
      P < .001.
      −0.130
      P < .05.
      −0.048
       Emotional0.225
      P < .001.
      0.1040.301
      P < .001.
      0.241
      P < .001.
      0.329
      P < .001.
      0.281
      P < .001.
      0.558
      P < .001.
      −0.136
      P < .01.
      −0.061
       Risky/impulsive0.274
      P < .001.
      0.0760.376
      P < .001.
      0.433
      P < .001.
      0.454
      P < .001.
      0.405
      P < .001.
      0.578
      P < .001.
      −0.180
      P < .001.
      0.075
      ASRS indicates Adult ADHD Self-Report Screener; BSSS, Brief Sensation Seeking Scale; OCS, Online Cognitions Scale; PHQ-9, Patient Health Questionnaire-9; PRIUSS, Problematic and Risky Internet Use Screening Scale; recreational, hours per day using the Internet for recreational activities; school/work, hours per day using the Internet for school- or work-related activities; SDS, Marlow-Crowne Social Desirability Scale Short Form; SPIN, Social Phobia Inventory; and Total, total hours per day using the Internet for any activities.
      P < .05.
      ∗∗ P < .01.
      ∗∗∗ P < .001.
      Social indicates Problematic and Risky Internet Use Screening Scale—Social Impairment subscale; Emotional, Problematic and Risky Internet Use Screening Scale—Emotional Impairment subscale; and Risky/Impulsive, Problematic and Risky Internet Use Screening Scale—Risky/Impulsive Internet Use subscale.

      Scale Cutoff for Risk-Based Assessment

      A cutoff of 25 was determined for categorizing participants as “not at risk” (≤25) and “at risk” (>25) for PIU based on their overall PRIUSS score. The cutoff has a sensitivity of 0.80 (95% CI 0.47–0.99) and a specificity of 0.79 (95% CI 0.73–0.84), with respect to the latent variable estimated using PRIUSS responses as the inputs. The receiver operating characteristic curve is presented in Figure 3. On the basis of the Bayesian latent class analysis model, the estimated prevalence of participants at risk for PIU in this study population is 6% (95% CI 2.7–12.0). Although conditional dependence might be expected between PRIUSS scores and self-reported Internet use as a result of their dimensional properties, no evidence for this was found in the data, and a more flexible latent class model that allows for conditional dependency between tests was assumed to be adequate.
      Figure thumbnail gr3
      Figure 3Receiver operating curve developed using a Bayesian latent class analysis model with optimum scoring cutoff value of 25 for identifying those at risk for problematic Internet use.
      Participants who scored as being at risk for PIU were at significantly greater odds of also reporting symptoms of ADHD (OR 2.36 95% CI 1.21–4.62, P = .009), depression (OR 3.25, 95% CI 1.65–6.42, P = .008), and social anxiety (OR 3.77, 95% CI 2.06–6.89, P < .000).

      Discussion

      In the current study, evidence supported the structural and construct validity of the PRIUSS in an AYA college student sample. Participants' responses across the PRIUSS items suggest that some degree of PIU may be normative among older AYAs, particularly when it comes to balancing online time with other priorities. Clinicians talking with patients about their Internet use may find it helpful to focus on those who either report significant life consequences in connection to symptoms or to endorse less frequent symptoms, such as decreased social involvement or heavy emotional reliance on online activities. Use of the scoring cutoff for identifying those at risk for PIU may also help identify patients who would benefit from further intervention. The reciprocal risk for other common mental health conditions among those above the PIU risk cutoff identified in this study suggests that screening for PIU with the PRIUSS may be a successful method for identifying AYAs in need of broader mental health evaluation.
      The findings from the PRIUSS construct validation support and provide new insights into the nature of the relationships between PIU and other health factors and risk behaviors. Consistent with previous findings, PIU symptoms were more strongly correlated with recreational Internet use compared to work- or school-related use.
      • Tao R.
      • Huang X.
      • Wang J.
      • et al.
      Proposed diagnostic criteria for Internet addiction.
      • Kubey R.W.
      • Lavin M.J.
      • Barrows J.R.
      Internet use and collegiate academic performance decrements: early findings.
      • Wu J.
      • Lin G.
      • Lin L.
      Analysis of the situation of Internet use and the related health-risky behaviors among the youngsters in Guangzhou City.
      Interestingly, this study found that among specific PIU symptoms, as measured by the PRIUSS subscales, ADHD symptoms correlated most strongly with the Risky/Impulsive Use domain. This may suggest that preventative efforts for PIU among AYAs diagnosed with ADHD should target shared behavioral traits such as impulsivity. Depression symptoms were also most closely related with the Risky/Impulsive Internet use, as opposed to the Emotional Impairment subscale, as may have been expected. This suggests that older adolescents' experiencing symptoms of depression such as anhedonia or impaired concentration may disproportionately spend time online in place of normal daily activities. It is also possible that those who overuse the Internet at the expense of healthy activities of daily living may experience somatic symptoms commonly associated with depression, such as difficulty sleeping or concentrating, low energy levels, or poor appetite. As may be expected, social anxiety symptoms were most strongly associated with the Social Consequences subscale. For older adolescents experiencing symptoms of social anxiety, such as fear during in-person social situations, the Internet may be a preferred setting for interaction. Thus, future studies may benefit from considering the role of Internet use as both a positive and negative factor influencing AYAs dealing with social anxiety.
      The moderate correlation between the PRIUSS overall score and the OCS suggests a similar, but not identical, symptom profile, and supports both convergent and divergent validity. Further, the PRIUSS's weak negative association with the social desirability measure suggests that a response bias of underreporting will not meaningfully hinder PRIUSS's performance. Finally, the lack of association between PRIUSS scores and sensation seeking is consistent with studies of similar adolescent populations.
      • Lavin M.
      • Marvin K.
      • McLarney A.
      • et al.
      Sensation seeking and collegiate vulnerability to Internet dependence.
      • Velezmoro R.
      • Lacefield K.
      • Roberti J.W.
      Perceived stress, sensation seeking, and college students' abuse of the Internet.
      • Kim H.K.
      • Davis K.E.
      Toward a comprehensive theory of problematic Internet use: evaluating the role of self-esteem, anxiety, flow, and the self-rated importance of Internet activities.
      Sensation seeking is commonly characterized through physical experiences such as drug use or sexual behavior, which may explain the discrepancy with risk behaviors carried out online.
      Taken together, these findings support the use of the PRIUSS in health settings with AYAs. Mental health resources, including self-screening for common disorders, are increasingly being provided in online formats for patients, particularly AYAs in academic settings (eg, http://www.mentalhealth.umn.edu/screening/). Given the nature of PIU, the PRIUSS may be uniquely suited for inclusion in such a program. Because research has shown that completing a screening assessment and being identified as at risk for a problem behavior can lead to a reduction in that behavior,
      • Epstein E.E.
      • Drapkin M.L.
      • Yusko D.A.
      • et al.
      Is alcohol assessment therapeutic? Pretreatment change in drinking among alcohol-dependent women.
      application of the PRIUSS in this manner may hold promise in reducing the burden of PIU among AYAs. Further, if utilized in an office setting, the PRIUSS can provide a framework for initiating discussion between the health provider, patients, and families. Specifically, reviewing subscale and item scores may help to identify individual problem areas and specific targets for behavioral change.

      Limitations and Future Directions

      The current study is limited by the cross-sectional nature and the use of self-report measures. Methods such as ecological momentary assessment may allow for a richer characterization of Internet use behaviors and how they intersect with PIU.
      • Moreno M.A.
      • Jelenchick L.
      • Koff R.
      • et al.
      Internet use and multitasking among older adolescents: an experience sampling approach.
      Similarly, clinical interviews or medical records reviews to corroborate participants' symptom self-report may also help explore the categorical nature of PIU and provide additional support to the PRIUSS's diagnostic accuracy. Additionally, future studies assessing the PRIUSS in younger adolescents, non-school-attending populations, and diverse minority groups are needed to broaden the PRIUSS's validation and to evaluate additional properties of the tool.
      As the current study took place in a healthy AYA sample, the current cutoff value represents a minimal threshold for identifying those at risk for PIU. The sensitivity and specificity estimates achieved for this cutoff are reasonable considering the PRIUSS's intended use as a risk-based screening measure. Studies sampling from those who report significant existing PIU symptoms will be useful in extending scoring guidelines to include a cutoff that discriminates between risk for PIU and more severe pathologic behavior. Finally, it is important to note that use of the Bayesian latent class model to estimate a scale cutoff is a preliminary method and is not intended as a permanent replacement for screening guidelines informed by a reference standard. Further, given that the underlying assumptions and methods for parameter estimates can vary across latent models,
      • Reitsma J.B.
      • Rutjes A.W.
      • Khan K.S.
      • et al.
      A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard.
      future studies that utilize different model specifications could produce parameter estimates that fall outside the CIs estimated in this study. PIU is currently cited in the appendix of the Diagnostic and Statistical Manual of Mental Disorders, version 5, as a disorder requiring further study; should it be included in future revisions with definitive diagnostic criteria, additional studies will be needed to further develop the PRIUSS's predictive validity and use as a screening measure.

      Conclusions

      Given the growing influence of the Internet in the lives of AYAs, advancing the development of screening and intervention tools for PIU is of critical importance. The present study validates the PRIUSS as a clinical screening measure for PIU in older adolescents and highlights the reciprocal relationship between PIU and other common AYA mental health conditions. Future research that includes longitudinal assessments and at-risk populations will help to refine methods for implementing the PRIUSS in AYA health settings.

      Acknowledgments

      This work was supported by R01DA031580-01. This grant is supported by the Common Fund , which is managed by the OD/Office of Strategic Coordination (OSC). Funding for this project was also provided by the University of Wisconsin Graduate School . These funding sources had no role design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The authors would like to thank the University of Minnesota Medical School Education Office and the University of Minnesota Graduate and Professional Student Assembly for their support; and Rajitha Kota, Angela Davis, and Jaimee Goldish for their assistance with this project.

      Supplementary Data

      References

        • Liu T.C.
        • Desai R.A.
        • Krishnan-Sarin S.
        • et al.
        Problematic Internet use and health in adolescents: data from a high school survey in Connecticut.
        J Clin Psychiatry. 2011; 72: 836-845
        • Christakis D.A.
        • Moreno M.M.
        • Jelenchick L.
        • et al.
        Problematic Internet usage in US college students: a pilot study.
        BMC Med. 2011; 9: 77
        • Derbyshire K.L.
        • Lust K.A.
        • Schreiber L.R.
        • et al.
        Problematic Internet use and associated risks in a college sample.
        Compr Psychiatry. 2013; 54: 415-422
        • Yates T.M.
        • Gregor M.A.
        • Haviland M.G.
        Child maltreatment, alexithymia, and problematic Internet use in young adulthood.
        Cyberpsychol Behav Soc Netw. 2012; 15: 219-225
        • Lewinsohn P.M.
        • Rohde P.
        • Seeley J.R.
        Major depressive disorder in older adolescents: prevalence, risk factors, and clinical implications.
        Clin Psychol Rev. 1998; 18: 765-794
        • Yen J.Y.
        • Ko C.H.
        • Yen C.F.
        • et al.
        The comorbid psychiatric symptoms of Internet addiction: attention deficit and hyperactivity disorder (ADHD), depression, social phobia, and hostility.
        J Adolesc Health. 2007; 41: 93-98
        • Ko C.H.
        • Yen J.Y.
        • Chen C.S.
        • et al.
        Predictive values of psychiatric symptoms for Internet addiction in adolescents: a 2-year prospective study.
        Arch Pediatr Adolesc Med. 2009; 163: 937-943
        • Lam L.T.
        • Peng Z.W.
        Effect of pathological use of the Internet on adolescent mental health.
        Arch Pediatr Adolesc Med. 2010; 164: 901-906
        • Jelenchick L.A.
        • Christakis D.A.
        Problematic Internet use during adolescence and young adulthood.
        in: Strasburger V.C. Moreno M.A. AM: STARs Social Networking and New Technologies: Adolescent Medicine State of the Art Review. American Academy of Pediatrics, Elk Grove Village, Ill2014: 605-620
        • Moreno M.A.
        • Jelenchick L.
        • Cox E.
        • et al.
        Problematic Internet use among US youth: a systematic review.
        Arch Pediatr Adolesc Med. 2011; 165: 797-805
        • Young K.S.
        Caught in the Net: How to Recognize the Signs of Internet Addiction—And a Winning Strategy for Recovery.
        John Wiley & Sons, New York, NY1998: 256
        • Moreno M.A.
        • Jelenchick L.A.
        • Christakis D.A.
        Problematic Internet use among older adolescents: a conceptual framework.
        Comput Human Behav. 2013; 29: 1879-1887
        • Jelenchick L.A.
        • Eickhoff J.
        • Christakis D.A.
        • et al.
        The Problematic and Risky Internet Use Screening Scale (PRIUSS) for adolescents and young adults: scale development and refinement.
        Comput Human Behav. 2014; 35: 171-178
        • DeVellis R.F.
        Scale Development: Theory and Applications.
        2nd ed. Sage Publications, Thousand Oaks, Calif2003: 171
        • Adler L.A.
        • Newcorn J.H.
        Administering and evaluating the results of the adult ADHD Self-Report Scale (ASRS) in adolescents.
        J Clin Psychiatry. 2011; 72: e20
        • Kessler R.C.
        • Adler L.A.
        • Gruber M.J.
        • et al.
        Validity of the World Health Organization Adult ADHD Self-Report Scale (ASRS) screener in a representative sample of health plan members.
        Int J Methods Psychiatr Res. 2007; 16: 52-65
        • Richardson L.P.
        • McCauley E.
        • Grossman D.C.
        • et al.
        Evaluation of the Patient Health Questionnaire-9 Item for detecting major depression among adolescents.
        Pediatrics. 2010; 126: 1117-1123
        • Johnson H.S.
        • Inderbitzen-Nolan H.M.
        • Anderson E.R.
        The Social Phobia Inventory: validity and reliability in an adolescent community sample.
        Psychol Assess. 2006; 18: 269-277
        • Connor K.M.
        • Davidson J.R.
        • Churchill L.E.
        • et al.
        Psychometric properties of the Social Phobia Inventory (SPIN): new self-rating scale.
        Br J Psychiatry. 2000; 176: 379-386
        • Jia R.
        • Jia H.H.
        Factorial validity of problematic Internet use scales.
        Comput Human Behav. 2009; 25: 1335-1342
        • Davis R.A.
        A cognitive–behavioral model of pathological Internet use.
        Comput Human Behav. 2001; 17: 187-195
        • Davis R.A.
        • Flett G.L.
        • Besser A.
        Validation of a new scale for measuring problematic Internet use: implications for pre-employment screening.
        Cyberpsychol Behav. 2002; 5: 331-345
        • Reynolds W.M.
        Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale.
        J Clin Psychol. 1982; 38: 119-125
        • Hoyle R.H.
        • Stephenson M.T.
        • Palmgreen P.
        • et al.
        Reliability and validity of a brief measure of sensation seeking.
        Personality Individual Differences. 2002; 32: 401-414
        • Stephenson M.T.
        • Velez L.F.
        • Chalela P.
        • et al.
        The reliability and validity of the Brief Sensation Seeking Scale (BSSS-8) with young adult Latino workers: implications for tobacco and alcohol disparity research.
        Addiction. 2007; 102: 79-91
        • Lunn D.
        • Spiegelhalter D.
        • Thomas A.
        • et al.
        The BUGS project: evolution, critique and future directions.
        Stat Med. 2009; 28: 3049-3067
        • Stevens J.P.
        Applied Multivariate Statistics for the Social Sciences.
        4th ed. Lawrence Erlbaum Associates, Mahwah, NJ2002
        • Brown M.W.
        • Cudeck R.
        Alternative ways of assessing model fit.
        in: Bollen K.A. Long J.S. Testing Structural Equation Models. Sage, Newbury Park, Calif1993: 136-162
        • Tao R.
        • Huang X.
        • Wang J.
        • et al.
        Proposed diagnostic criteria for Internet addiction.
        Addiction. 2010; 105: 556-564
        • Reitsma J.B.
        • Rutjes A.W.
        • Khan K.S.
        • et al.
        A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard.
        J Clin Epidemiol. 2009; 62: 797-806
        • Joseph L.
        • Gyorkos T.W.
        • Coupal L.
        Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard.
        Am J Epidemiol. 1995; 141: 263-272
        • Kubey R.W.
        • Lavin M.J.
        • Barrows J.R.
        Internet use and collegiate academic performance decrements: early findings.
        J Commun. 2001; 51: 366-382
        • Wu J.
        • Lin G.
        • Lin L.
        Analysis of the situation of Internet use and the related health-risky behaviors among the youngsters in Guangzhou City.
        J Trop Med (Guangzhou). 2007; 7: 816-818
        • Lavin M.
        • Marvin K.
        • McLarney A.
        • et al.
        Sensation seeking and collegiate vulnerability to Internet dependence.
        Cyberpsychol Behav. 1999; 2: 425-430
        • Velezmoro R.
        • Lacefield K.
        • Roberti J.W.
        Perceived stress, sensation seeking, and college students' abuse of the Internet.
        Comput Human Behav. 2010; 26: 1526-1530
        • Kim H.K.
        • Davis K.E.
        Toward a comprehensive theory of problematic Internet use: evaluating the role of self-esteem, anxiety, flow, and the self-rated importance of Internet activities.
        Comput Human Behav. 2009; 25: 490-500
        • Epstein E.E.
        • Drapkin M.L.
        • Yusko D.A.
        • et al.
        Is alcohol assessment therapeutic? Pretreatment change in drinking among alcohol-dependent women.
        J Stud Alcohol. 2005; 66: 369-378
        • Moreno M.A.
        • Jelenchick L.
        • Koff R.
        • et al.
        Internet use and multitasking among older adolescents: an experience sampling approach.
        Comput Human Behav. 2012; 28: 1097-1102