If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Problematic Internet use (PIU) is an emerging health concern that lacks screening measures validated for use with adolescents and young adults. This study aimed to validate the Problematic and Risky Internet Use Screening Scale (PRIUSS) for use with older adolescents and to increase its clinical utility by determining scoring guidelines and assessing the relationship between PIU and other mental health conditions.
This cross-sectional survey study took place at a large, public Midwestern university among 330 older adolescents aged 18 to 25 years. Confirmatory factor analysis and Spearman's correlations were used to assess the PRIUSS' structural and construct validity, respectively. A risk-based scoring cutoff was estimated using a Bayesian latent class modeling approach to computing a receiver operating characteristic curve.
The confirmatory factor analysis indices for the 3-factor model indicated an acceptable fit (goodness-of-fit index 0.89, root mean square error of approximation 0.07). A cutoff of 25 (sensitivity 0.80, 95% confidence interval [CI] 0.47–0.99; specificity 0.79, 95% CI 0.73–0.84) is proposed for identifying those at risk for PIU. Participants at risk for PIU were at significantly greater odds of also reporting symptoms of attention-deficit/hyperactivity disorder (odds ratio [OR] 2.36 95% CI 1.21–4.62, P = .009), depression (OR 3.25, 95% CI 1.65–6.42, P = .008), and social anxiety (OR 3.77, 95% CI 2.06–6.89, P < .000).
The PRIUSS demonstrated validity as a PIU screening instrument for adolescents and young adults. Screening for PIU may also help to identify those at high reciprocal risk for other mental health conditions.
Problematic Internet use (PIU) affects approximately 4% to 6% of US adolescents and young adults. This study validates a clinical screening measure, the Problematic and Risky Internet Use Screening Scale (PRIUSS), and provides scoring guidelines for its use in interpreting older adolescents' risk for PIU.
Problematic Internet use (PIU) is an emerging health concern among US adolescents and young adults (AYAs). A study of high school students throughout Connecticut estimated its prevalence at 4%.
Although definitive diagnostic criteria for PIU are yet to be established, Internet use addiction disorder is currently included in the appendix of the Diagnostic and Statistical Manual of Mental Disorders, version 5, as a disorder requiring further study. Irrespective of its official designation, there are undoubtedly a substantial number of youth for whom Internet use has adverse effects and who may thus benefit from intervention. Given that online activities now often begin early in childhood, primary care pediatric clinicians are uniquely positioned to help prevent the development of PIU.
Further, PIU appears to follow a similar developmental trajectory to other risk behaviors such as bullying or tobacco or alcohol use, all of which are commonly screened for and addressed during AYA wellness visits.
Thus, the development of a screening tool specific to PIU during adolescence and young adulthood and suitable for clinical settings is an important step in designing primary prevention efforts. The majority of existing rating scales for PIU symptoms have been adapted from diagnostic criteria for disparate psychiatric disorders and developed for adults.
A recent systematic review of PIU highlighted the diversity of measurement tools that lack validation and the resultant disparate prevalence estimates, and concluded that a uniform and validated approach to measurement is a necessary next step to understanding prevalence of PIU among AYAs.
described the first data-driven conceptual framework for PIU specific to AYAs, including a definition of PIU as, “Internet use that is risky, excessive or impulsive in nature leading to adverse life consequences, specifically physical, emotional, social or functional impairment.” Further, the framework identifies 7 core constructs of PIU: 3 describing the nature of PIU (“risky Internet use” “impulsive Internet use,” “Internet use dependency”); 3 describing its impact on AYA health and well-being (“physical impairment,” “emotional impairment,” “social/functional impairment”); and a final construct describing factors that may predispose AYAs to PIU (“psychosocial risk factors”). In a follow-up study, Jelenchick and colleagues
adapted the comprehensive list of characteristics, behaviors, and symptoms describing each of the framework's construct into an item pool for use in a scale development study. Specifically, 75 of the descriptive terms were assessed using the psychometric methods for scale development described by DeVellis.
The end product, the Problematic and Risky Internet Use Screening Scale (PRIUSS), is an 18-item risk-based screening scale for PIU with questions organized into 3 subscales: Social Impairment, Emotional Impairment, and risky/impulsive Internet use. Figure 1 summarizes the process used to integrate Moreno and colleagues' conceptual framework into the development of the PRIUSS.
Although the PRIUSS underwent an empirical development process and demonstrated strong reliability and content validity, further validation is needed to support its use in AYA health settings. Thus, the aims of the current study were 3-fold: 1) to assess the PRIUSS’ structural validity by confirming its factor structure, 2) to extend the construct validation of the PRIUSS, including establishing convergent and divergent validity with other behavioral and mental health conditions, and 3) to determine preliminary scoring guidelines to facilitate its clinical utility.
Subjects and Setting
This cross-sectional survey study was conducted between January 2012 and June of 2013. Approval of the study was granted by the institutional review board at the University of Wisconsin. Participants were older adolescents aged 18 to 25 years recruited from undergraduate students enrolled in a nutritional sciences course at a public university located in the Midwest, a course that has both a high enrollment number and fulfills general education requirements for a variety of programs and majors.
Students were initially invited to participate in the study through an in-class announcements; a posting was also placed on course Web sites. All participants provided written consent before completing the survey on a secure online site. The survey contained standard demographic questions and general Internet use questions. Internet use questions included the respondents' average number of times using the Internet per day, total daily hours spent online, number of hours spent on work or school activities, and number of hours spent on recreational activities. When responding to these questions, participants were instructed to think of their use over the last month and include time spent on any Internet application, whether using a computer or mobile device. Time spent text messaging was not included unless text messages were being used to interact with an online application. Finally, the survey contained the PRIUSS, as well as behavioral and mental health measures intended to assess the PRIUSS’ construct validity. Students who completed the survey received a $10 gift card.
Problematic and Risky Internet Use Screening Scale (PRIUSS)
The PRIUSS has 18 items and 3 subscales: 1) Social Impairment (items 1–6), which assesses the impact of Internet use on both off-line and online social interactions; 2) Emotional Impairment (items 7–11), which assesses degree of emotional attachment to Internet use; and 3) Risky/Impulsive Internet Use (items 12–18), which assess salient problematic behaviors regarding Internet use.
The ASRS is a 6-item scale adapted from the Diagnostic and Statistical Manual of Mental Disorders, version 4, symptoms for attention-deficit/hyperactivity disorder (ADHD) that has been validated in both adolescents and adults.
All P values were 2-sided, and P < .05 was used to indicate statistical significance. Descriptive data were summarized as frequencies and percentages or means and standard deviations. Because of the limited number of nonwhite participants, during analysis, the categories for race were collapsed into a binary white or nonwhite variable. Participants' Internet use was categorized on the basis of their self-reported average number of daily recreational hours; participants were categorized as either average users (<6 recreational hours per day) or high users (≥6 recreational hours per day). A summary of the statistical approach utilized for the scale validation is provided below; additional details can be found in the Online Supplementary Data.
Cronbach's alphas were calculated to examine the internal consistency reliability of the 3 subscales. Pearson's correlation coefficient (r) was calculated to examine the correlation between each subscale. A confirmatory factor analysis was conducted to test the PRIUSS' 3-factor structure. The goodness-of-fit index and root mean square error of approximation estimate were used as the primary measures for evaluating model fit. Models with goodness-of-fit indices between 0.9 and 1 and root mean square error of approximation values between 0.10 and 0.08 are considered a good fit.
To evaluate the scale's construct validity, nonparametric Spearman's correlation was used to determine associations between PRIUSS scores and the theoretically related variables.
Scale Cutoff for Risk-Based Assessment
To estimate a cutoff for the PRIUSS that could be used to discriminate between those at risk for PIU from those not at risk, a receiver operating characteristic curve was computed using a Bayesian latent class modeling approach. Participants' Internet use categorization (average vs high use) was incorporated as an additional model parameter based on related studies of PIU that identified ≥6 daily recreational hours as the most highly discriminating criterion for excessive use.
Sensitivity and specificity estimates of the PRIUSS were reported along with the corresponding 95% confidence intervals (CI).
Finally, logistic regression analysis was used to assess associations between PRIUSS risk category and the probability of experiencing ADHD, depression, or social anxiety symptoms, with race included as a cofactor. Odds ratios (ORs) and the corresponding 95% CIs were computed to compare the proportions of participants reporting significant mental health symptoms between the PRIUSS scoring categories.
A total of 330 participants completed the survey (50% response rate). Participants reported on average 5.0 (SD 3.0) total daily hours of Internet use, and 5.8% (n = 19) reported >6 hours of daily recreational use and were categorized as a high user. PRIUSS scores ranged from 0 to 49; the median score was 17. Nonwhite participants scored higher on the PRIUSS overall (mean 22.5, SD 11.5 versus mean 17.6, SD 9.9, P = .004). There were no significant differences by sex (P = .700), year in school (P = .440), or major field of study (P = .560). Detailed demographic information, as well as average scores and categorical distributions for the PRIUSS and other health measurements, are presented in Table 1.
Table 1Demographic Information and Distributions and Scores Across Descriptive Variables for 330 Study Participants
ADHD indicates attention-deficit/hyperactivity disorder; ASRS, Adult ADHD Self-Report Screener; PHQ-9, Patient Health Questionnaire-9; PRIUSS, Problematic and Risky Internet Use Screening Scale; and SPIN, Social Phobia Inventory.
Figure 2 summarizes the percentage of participants falling within each response category (never to very often) for the symptom described by each PRIUSS question. Overall, participants endorsed greater symptom frequencies for items in the risky and impulsive use subscale; over half of participants described these symptoms as occurring sometimes, often, or very often. The most commonly reported symptoms within the Social and Emotional Impairment subscales were “choosing to socialize online instead of in person” and “feeling vulnerable when away from the Internet,” respectively.
PRIUSS Structural Validity
Cronbach's α coefficients were 0.81, 0.85, and 0.85 for the Social Impairment, Emotional Impairment, and Risky/Impulsive Use subscales, respectively. The factor loadings and subscale correlations for the 3-factor model are depicted in an Online Supplementary Figure 2. The indices estimated for the 3-factor model indicated an acceptable fit for the data (goodness-of-fit index 0.89, root mean square error of approximation 0.07). The majority of items demonstrated strong factor loadings (range 0.61–0.82). A single item from the Social Consequences subscale, item 1, exhibited a weaker factor loading (0.31).
PRIUSS Construct Validity
Correlations between PRIUSS scores and the theoretically relevant variables are presented in Table 2. There were significant positive correlations between PRIUSS scores and recreational Internet use, as well as all 3 mental health measures and the measure for problematic online cognitions. The weakest correlations, which were also largely nonsignificant, were seen with the social desirability and sensation-seeking measures.
Table 2Correlations Between Problematic and Risky Internet Use Screening Scale (PRIUSS) and Theoretically Related Constructs
Social indicates Problematic and Risky Internet Use Screening Scale—Social Impairment subscale; Emotional, Problematic and Risky Internet Use Screening Scale—Emotional Impairment subscale; and Risky/Impulsive, Problematic and Risky Internet Use Screening Scale—Risky/Impulsive Internet Use subscale.
ASRS indicates Adult ADHD Self-Report Screener; BSSS, Brief Sensation Seeking Scale; OCS, Online Cognitions Scale; PHQ-9, Patient Health Questionnaire-9; PRIUSS, Problematic and Risky Internet Use Screening Scale; recreational, hours per day using the Internet for recreational activities; school/work, hours per day using the Internet for school- or work-related activities; SDS, Marlow-Crowne Social Desirability Scale Short Form; SPIN, Social Phobia Inventory; and Total, total hours per day using the Internet for any activities.
∗ P < .05.
∗∗ P < .01.
∗∗∗ P < .001.
† Social indicates Problematic and Risky Internet Use Screening Scale—Social Impairment subscale; Emotional, Problematic and Risky Internet Use Screening Scale—Emotional Impairment subscale; and Risky/Impulsive, Problematic and Risky Internet Use Screening Scale—Risky/Impulsive Internet Use subscale.
A cutoff of 25 was determined for categorizing participants as “not at risk” (≤25) and “at risk” (>25) for PIU based on their overall PRIUSS score. The cutoff has a sensitivity of 0.80 (95% CI 0.47–0.99) and a specificity of 0.79 (95% CI 0.73–0.84), with respect to the latent variable estimated using PRIUSS responses as the inputs. The receiver operating characteristic curve is presented in Figure 3. On the basis of the Bayesian latent class analysis model, the estimated prevalence of participants at risk for PIU in this study population is 6% (95% CI 2.7–12.0). Although conditional dependence might be expected between PRIUSS scores and self-reported Internet use as a result of their dimensional properties, no evidence for this was found in the data, and a more flexible latent class model that allows for conditional dependency between tests was assumed to be adequate.
Participants who scored as being at risk for PIU were at significantly greater odds of also reporting symptoms of ADHD (OR 2.36 95% CI 1.21–4.62, P = .009), depression (OR 3.25, 95% CI 1.65–6.42, P = .008), and social anxiety (OR 3.77, 95% CI 2.06–6.89, P < .000).
In the current study, evidence supported the structural and construct validity of the PRIUSS in an AYA college student sample. Participants' responses across the PRIUSS items suggest that some degree of PIU may be normative among older AYAs, particularly when it comes to balancing online time with other priorities. Clinicians talking with patients about their Internet use may find it helpful to focus on those who either report significant life consequences in connection to symptoms or to endorse less frequent symptoms, such as decreased social involvement or heavy emotional reliance on online activities. Use of the scoring cutoff for identifying those at risk for PIU may also help identify patients who would benefit from further intervention. The reciprocal risk for other common mental health conditions among those above the PIU risk cutoff identified in this study suggests that screening for PIU with the PRIUSS may be a successful method for identifying AYAs in need of broader mental health evaluation.
The findings from the PRIUSS construct validation support and provide new insights into the nature of the relationships between PIU and other health factors and risk behaviors. Consistent with previous findings, PIU symptoms were more strongly correlated with recreational Internet use compared to work- or school-related use.
Interestingly, this study found that among specific PIU symptoms, as measured by the PRIUSS subscales, ADHD symptoms correlated most strongly with the Risky/Impulsive Use domain. This may suggest that preventative efforts for PIU among AYAs diagnosed with ADHD should target shared behavioral traits such as impulsivity. Depression symptoms were also most closely related with the Risky/Impulsive Internet use, as opposed to the Emotional Impairment subscale, as may have been expected. This suggests that older adolescents' experiencing symptoms of depression such as anhedonia or impaired concentration may disproportionately spend time online in place of normal daily activities. It is also possible that those who overuse the Internet at the expense of healthy activities of daily living may experience somatic symptoms commonly associated with depression, such as difficulty sleeping or concentrating, low energy levels, or poor appetite. As may be expected, social anxiety symptoms were most strongly associated with the Social Consequences subscale. For older adolescents experiencing symptoms of social anxiety, such as fear during in-person social situations, the Internet may be a preferred setting for interaction. Thus, future studies may benefit from considering the role of Internet use as both a positive and negative factor influencing AYAs dealing with social anxiety.
The moderate correlation between the PRIUSS overall score and the OCS suggests a similar, but not identical, symptom profile, and supports both convergent and divergent validity. Further, the PRIUSS's weak negative association with the social desirability measure suggests that a response bias of underreporting will not meaningfully hinder PRIUSS's performance. Finally, the lack of association between PRIUSS scores and sensation seeking is consistent with studies of similar adolescent populations.
Sensation seeking is commonly characterized through physical experiences such as drug use or sexual behavior, which may explain the discrepancy with risk behaviors carried out online.
Taken together, these findings support the use of the PRIUSS in health settings with AYAs. Mental health resources, including self-screening for common disorders, are increasingly being provided in online formats for patients, particularly AYAs in academic settings (eg, http://www.mentalhealth.umn.edu/screening/). Given the nature of PIU, the PRIUSS may be uniquely suited for inclusion in such a program. Because research has shown that completing a screening assessment and being identified as at risk for a problem behavior can lead to a reduction in that behavior,
application of the PRIUSS in this manner may hold promise in reducing the burden of PIU among AYAs. Further, if utilized in an office setting, the PRIUSS can provide a framework for initiating discussion between the health provider, patients, and families. Specifically, reviewing subscale and item scores may help to identify individual problem areas and specific targets for behavioral change.
Limitations and Future Directions
The current study is limited by the cross-sectional nature and the use of self-report measures. Methods such as ecological momentary assessment may allow for a richer characterization of Internet use behaviors and how they intersect with PIU.
Similarly, clinical interviews or medical records reviews to corroborate participants' symptom self-report may also help explore the categorical nature of PIU and provide additional support to the PRIUSS's diagnostic accuracy. Additionally, future studies assessing the PRIUSS in younger adolescents, non-school-attending populations, and diverse minority groups are needed to broaden the PRIUSS's validation and to evaluate additional properties of the tool.
As the current study took place in a healthy AYA sample, the current cutoff value represents a minimal threshold for identifying those at risk for PIU. The sensitivity and specificity estimates achieved for this cutoff are reasonable considering the PRIUSS's intended use as a risk-based screening measure. Studies sampling from those who report significant existing PIU symptoms will be useful in extending scoring guidelines to include a cutoff that discriminates between risk for PIU and more severe pathologic behavior. Finally, it is important to note that use of the Bayesian latent class model to estimate a scale cutoff is a preliminary method and is not intended as a permanent replacement for screening guidelines informed by a reference standard. Further, given that the underlying assumptions and methods for parameter estimates can vary across latent models,
future studies that utilize different model specifications could produce parameter estimates that fall outside the CIs estimated in this study. PIU is currently cited in the appendix of the Diagnostic and Statistical Manual of Mental Disorders, version 5, as a disorder requiring further study; should it be included in future revisions with definitive diagnostic criteria, additional studies will be needed to further develop the PRIUSS's predictive validity and use as a screening measure.
Given the growing influence of the Internet in the lives of AYAs, advancing the development of screening and intervention tools for PIU is of critical importance. The present study validates the PRIUSS as a clinical screening measure for PIU in older adolescents and highlights the reciprocal relationship between PIU and other common AYA mental health conditions. Future research that includes longitudinal assessments and at-risk populations will help to refine methods for implementing the PRIUSS in AYA health settings.
This work was supported by R01DA031580-01. This grant is supported by the Common Fund , which is managed by the OD/Office of Strategic Coordination (OSC). Funding for this project was also provided by the University of Wisconsin Graduate School . These funding sources had no role design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The authors would like to thank the University of Minnesota Medical School Education Office and the University of Minnesota Graduate and Professional Student Assembly for their support; and Rajitha Kota, Angela Davis, and Jaimee Goldish for their assistance with this project.