1. Evaluate the Results and Discussion sections of the accompanying article and critique how the following were addressed. Do not copy and paste from the article. Do not simply respond Yes or No. Describe in your own words how the researchers demonstrated that these elements were addressed.
a) First, check the beginning of the article. What is the hypothesis and/or research question(s)?
b) Now, look at the Results section. Are the results of each of the hypothesis(es) and/or research questions presented?
c) Discuss whether the results are presented objectively.
d) If tables or figures are used, do they meet the following standards?
* They supplement and economize the text
*They have precise titles and headings
*They are not repetitious of the text
e) Now move to the Discussion section. Are the results interpreted in light of the hypotheses, research questions, sampling, data collection, instruments, and other steps that preceded the results?
f) How does the researcher identify the study’s weaknesses (threats to validity) and limitations? Are solutions proposed to address these issues?
g) How does the researcher discuss the study’s clinical relevance and/or implications for nursing?
h) Is generalizability discussed? If so, do these generalizations fall within the scope of the findings?
i) Are any recommendations for future research stated or implied? If so, what are they?
2. Discuss whether the strength of the evidence supports a change in current practice. Support your conclusions.
3. What is your cosmic question? (This should be based on chapter of the week. Pose a research question on the Results or Discussion section of a research article.)
Effect of an E-mental Health Approach to Workers’
Health Surveillance versus Control Group on Work
Functioning of Hospital Employees: A Cluster-RCT
Sarah M. Ketelaar1*, Karen Nieuwenhuijsen1, Fania R. Gärtner1, Linda Bolier2, Odile Smeets2,
Judith K. Sluiter1
1 Coronel Institute of Occupational Health, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands, 2 Innovation Center of Mental Health and
Technology (I.COM), Trimbos Institute, Netherlands Institute of Mental Health and Addiction, Utrecht, The Netherlands
Objective: To evaluate an e-mental health (EMH) approach to workers’ health surveillance (WHS) targeting work functioning
(WF) and mental health (MH) of healthcare professionals in a randomised controlled trial.
Methods: Nurses and allied health professionals (N = 1140) were cluster-randomised at ward level to the intervention (IG) or
control group (CG). The intervention consisted of two parts: (a) online screening and personalised feedback on impaired WF
and MH, followed by (b) a tailored offer of self-help EMH interventions. CG received none of these parts. Primary outcome
was impaired WF (Nurses Work Functioning Questionnaire), assessed at baseline and after three and six months. Analyses
were performed in the positively screened subgroup (i) and in all participants (ii).
Results: Participation rate at baseline was 32% (NIG = 178; NCG = 188). Eighty-two percent screened positive for at least mild
impairments in WF and/or MH (NIG = 139; NCG = 161). All IG-participants (N = 178) received part (a) of the intervention, nine
participants (all positively screened, 6%) followed an EMH intervention to at least some extent. Regarding the subgroup of
positively screened participants (i), both IG and CG improved over time regarding WF (non-significant between-group
difference). After six months, 36% of positively screened IG-participants (18/50) had a relevant WF improvement compared
to baseline, versus 28% (32/115) of positively screened CG-participants (non-significant difference). In the complete sample
(ii), IG and CG improved over time but IG further improved between three and six months while CG did not (significant
Conclusions: In our study with a full compliance rate of 6% and substantial drop-out leading to a small and underpowered
sample, we could not demonstrate that an EMH-approach to WHS is more effective to improve WF and MH than a control
group. The effect found in the complete sample of participants is not easily interpreted. Reported results may be useful for
future meta-analytic work.
Trial Registration: Dutch Trial Register NTR2786 http://www.trialregister.nl
Citation: Ketelaar SM, Nieuwenhuijsen K, Gärtner FR, Bolier L, Smeets O, et al. (2013) Effect of an E-mental Health Approach to Workers’ Health Surveillance versus
Control Group on Work Functioning of Hospital Employees: A Cluster-RCT. PLoS ONE 8(9): e72546. doi:10.1371/journal.pone.0072546
Editor: Jim van Os, Maastricht University Medical Centre, The Netherlands
Received January 11, 2013; Accepted July 11, 2013; Published September 12, 2013
Copyright: � 2013 Ketelaar et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The Mental Vitality @ Work trial was co-financed by a grant from the Dutch Foundation Institute Gak (URL: http://www.instituutgak.nl, PrevBGZ/project
D) and a grant from The Netherlands Organisation for Health Research and Development (ZonMW) (URL: http://www.zonmw.nl/en, grant number 208010001).
The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing Interests: The e-mental health interventions used in this study are stand-alone interventions which were developed by the Trimbos Institute at an
earlier stage. However, neither the authors working at the Trimbos Institute (LB and OS) nor the institute itself derive financial income from the interventions. The
other authors have declared that no competing interests exist.
* E-mail: S.M.Ketelaar@amc.uva.nl
Nurses have a high risk of developing common mental health
complaints, such as distress, depression, and anxiety [1–3].
Impaired mental health of employees in healthcare occupations
can have serious adverse effects, endangering the health and safety
of not only themselves but also their patients. A study by Gärtner
and colleagues found that impaired mental health in nurses and
allied health professionals affects several aspects of their work
functioning, including cognitive aspects (e.g. staying alert) and
causing incidents at work . Another study by Letvak and
colleagues showed that depression in nurses was associated with
presenteeism, which is in turn associated with patient falls,
medication errors, and lower self-reported quality of care .
Adding to this, increased levels of psychological distress, even in a
mild form, have been found to be associated with an increased
likelihood of obtaining a disability pension in later life . To
sustain nurses’ and allied health professionals’ mental health and
to enable them to remain healthy and well-functioning in their
profession until retirement age, it is crucial to periodically screen
PLOS ONE | www.plosone.org 1 September 2013 | Volume 8 | Issue 9 | e72546
these employees and provide interventions to improve their mental
health and work functioning.
A potentially promising method for the early detection of
impaired mental health and subsequent treatment in nurses and
allied health professionals, is offering a mental module for workers’
health surveillance (WHS). Although attention has been paid to
the occupational hazards of healthcare employees , WHS
targeting work functioning and mental health of nurses and allied
health professionals has, to our knowledge, not been reported
WHS is an important component of occupational healthcare
. It is a means to implement preventive action by identifying
and treating health complaints relevant to work, and it should be
an essential component of programmes aimed at the protection of
employees . In the Netherlands, it has three aims: 1) to prevent
the onset, recurrence, or worsening of work-related diseases, 2) to
monitor and promote work-related health, and 3) to monitor and
improve work functioning and employability . It can be used
to periodically monitor employees’ health and work functioning to
detect impairments early and to bring timely interventions into
action to prevent further impairment. It is recommended to apply
a job-specific assessment, to allow for tailoring of interventions to
the specific detected work functioning impairments as fitting as
possible . In this study, we detect early signs of impaired
mental health and impaired work functioning in nurses and allied
health professionals, and offer interventions using an e-mental
E-mental health (EMH) is the use of information and
communication technology, and in particular the many tech-
nologies related to the Internet, to support and improve mental
health . Applying EMH might be a useful and feasible
approach to perform a mental module for WHS. Online
screening is a practical and efficient method to screen for self-
reported impaired work functioning and impaired mental
health. Furthermore, EMH offers possibilities regarding the
subsequent interventions. Ritterband and colleagues defined
Internet interventions as typically focused on behavioral issues,
aiming to institute behavior change and subsequent symptom
improvement, usually self-paced, interactive, and tailored to the
user, and making use of the multimedia format offered by the
Internet . EMH interventions exist which target a wide
variety of common mental disorders such as depression, anxiety,
panic, phobias, and various addictions. Unguided self-help
EMH interventions have been found to have positive outcomes
for a variety of mental health aspects (e.g. Warmerdam et al.
; Farrer et al. ; Riper et al. ; Blankers et al. ;
Billings et al. ), although to our knowledge their effects on
work functioning have not been studied in a specific working
population such as nurses and allied health professionals.
Moreover, EMH interventions have thus far only been offered
as stand-alone interventions for a specific mental health
complaint. In our study, we offer a choice of EMH interven-
tions, tailored to the specific complaints as indicated by the
individual’s screening results.
In this paper, we study the effect of an EMH-approach to
WHS targeting work functioning and mental health of hospital-
employed nurses and allied health professionals, on their work
functioning, distress, work-related fatigue, posttraumatic stress,
and work ability in a cluster-randomized controlled trial. We
hypothesized that WHS, consisting of online screening on
impaired work functioning and impaired mental health
followed by personalised feedback and a tailored offer of self-
help EMH interventions, will improve work functioning and
The protocol for this trial and supporting CONSORT checklist
are available as supporting information; see Checklist S1 and
Protocol S1 (http://www.biomedcentral.com/1471-2458/11/
The Medical Ethics Committee of the Academic Medical
Center Amsterdam approved this study (for approved protocol see
Protocol S2). All participants gave their written informed consent
before taking part.
The study was designed as a cluster-randomised trial with block
randomisation carried out at ward level. In order to guarantee
allocation concealment, randomisation was performed by one
researcher (KN) who was not involved in the practical recruitment
of employees, using the computer software program Nquery
Advisor with a block size of three. The complete trial included two
intervention groups and one control group . The present study
compared one of the intervention groups, the e-mental health
approach (EMH-approach) group, to the control group. The other
intervention group consisted of an invitation for a preventive
consultation with an occupational physician. A pre-randomisation
procedure with incomplete-double-consent design was applied
, meaning that individuals were only informed about their own
Outcome measures were obtained from all participants at
baseline (March 2011) and follow-up measures were obtained
three and six months after baseline.
The design, conduct and reporting of this study adhere to the
Consolidated Standards of Reporting Trials guidelines [21,22].
Details of the study design are reported elsewhere . The trial
registration number of the study is NTR2786 (Dutch Trial
The study population of the complete trial was formed by all
nurses, including surgical nurses and anaesthetic nurses, and allied
health professionals (such as physiotherapists and radiotherapists)
employed at one academic hospital in the Netherlands (N = 1731).
Nurses and allied health professionals form two large groups of
hospital employees, and many of their work demands and work
conditions are similar. Since it regarded a preventive study,
participants were included if they were not, or were not expecting
to be on sick leave for more than two weeks at baseline.
All eligible employees were invited for participation in the study.
To detect a clinically significant effect (effect size$.33), while
conducting the tests with alpha = .05 (two-tailed) and power = .80,
and allowing for possible cluster effects and loss to-follow-up, the
minimum required sample size was 718 participants for the
complete trial . After randomization at ward level (N = 86), 29
wards with 579 employees were assigned to the EMH-approach
group and 29 wards with 561 employees to the control group
In March 2011, potential participants received an invitation by
e-mail to fill out the online baseline questionnaire which could be
filled out at any time during six weeks. It was possible to
discontinue the questionnaire and complete it after logging in
again. Three reminders were sent, as well as an information letter
to their home address. Those who had completed the baseline
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 2 September 2013 | Volume 8 | Issue 9 | e72546
questionnaire were invited to fill out the follow-up questionnaires
three and six months after baseline.
E-mental health approach group. At baseline, participants
in the EMH-approach group were screened on the following
aspects (for details see Table 1 and Gärtner and colleagues ):
impaired work functioning, distress, work-related fatigue, risky
drinking behaviour, depression including suicide risk, anxiety,
panic disorder, and posttraumatic stress. Participants received
personalised feedback on their screening results immediately after
filling out the baseline questionnaire, both onscreen and in an e-
The personalised feedback was followed by an invitation for a
tailored offer of self-help EMH interventions, on the basis of an
algorithm based on the specific symptoms and the work-
relatedness of the symptoms (available as supporting information,
see Algorithm S1). Participants were mostly offered a choice of two
or three EMH interventions to leave room for personal
preferences. Participants who screened negative on all mental
health complaints were invited to follow an EMH intervention
aimed at enhancing and retaining their mental fitness.
The EMH interventions used in this study are self-help
interventions on the Internet aimed at reducing specific mental
health complaints or enhancing wellbeing. The interventions are
mainly based on the principles of cognitive behavioural therapy
and combine a variety of aspects, e.g. providing information and
advice, weekly assignments, the option of keeping a diary and a
forum to get in contact with others who have similar complaints.
The EMH interventions were developed as stand-alone interven-
tions by the Trimbos Institute (Netherlands Institute of Mental
Figure 1. Flow of participants through the trial.
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 3 September 2013 | Volume 8 | Issue 9 | e72546
Health and Addiction) at an earlier stage. The following EMH
interventions were used in the study:
– Psyfit : aimed at enhancing mental fitness. Also applied for
– Strong at work : aimed at gaining insight into work stress and
learning skills to cope with it.
– Colour your Life : aimed at tackling depressive symptoms.
– Don’t Panic Online : aimed at reducing panic symptoms for
subclinical and mild cases of panic disorder.
– Drinking Less : aimed at reducing risky drinking behaviour.
Psyfit was found to be effective in decreasing symptoms of
depression and anxiety and improving well-being and vitality .
Warmerdam and colleagues  showed that Colour your Life
resulted in significantly lower depression and anxiety scores
compared to a waiting-list control group and to significantly
higher quality of life scores. The number of participants showing
clinically relevant change regarding depression after 12 weeks was
significantly higher. Spek and colleagues also found a significantly
larger improvement in depressive symptoms compared to a
waiting-list control group [28,29]. Drinking Less resulted in more
participants who reduced their drinking successfully to within
guideline norms, and a significantly larger decrease in mean
weekly alcohol consumption than a control group .
In case of positive screening on impaired work functioning
(regardless of their mental health status), participants received an
onscreen educational leaflet on how to improve their work
functioning (available upon request).
Control group. Participants in the control group filled out
the same baseline questionnaire as the EMH-approach group, but
did not receive an intervention, and thus no screening results
either. However, they were informed that they would receive
personalised feedback and a tailored offer of self-help EMH
interventions after six months, following the six months follow-up
All outcomes were measured at baseline and at three and six
Primary outcome. The primary outcome of this study was
impaired work functioning, measured with the total score of the
Nurses Work Functioning Questionnaire (NWFQ) . This
questionnaire has been developed to assess impaired work
functioning in nurses and allied health professionals. In the
screening phase, all seven of the original subscales were used.
Participants scored either green, orange or red on each subscale.
A red score on one or more subscales and/or three or more
orange scores led to case identification of impaired work
functioning (i.e. scoring above cut-off point on impaired work
Only six of the seven original NWFQ subscales were used for
the outcome measure, in contrast to what was described in the
trial’s design study , because the reproducibility of the impaired
decision-making subscale was found to be poor . The total score
on the NWFQ was calculated with the 47 items of the remaining
six subscales, with a total score range of 0–100, a higher score
indicating more severely impaired work functioning.
The difference between the EMH-approach group and the
control group regarding impaired work functioning was investi-
gated using the continuous outcome and the percentage of
individuals who had improved relevantly at follow-up .
Secondary outcomes. The secondary outcomes included
distress, work-related fatigue, posttraumatic stress, and work
Distress was measured with the distress subscale of the Four-
Dimensional Symptoms Questionnaire (4DSQ) [32,33]. The 16-
item questionnaire uses a 5-point response scale (0 = no, 4 = very
often) and has a total score range of 0–32, a higher score indicating
a higher level of distress (cut-off point $11 ).
Work-related fatigue after working time was measured using the
need for recovery subscale of the Dutch Questionnaire on the
Experience and Evaluation of Work (QEEW) . The 11-item
questionnaire with dichotomous response categories (yes, no) has a
total score range of 0–11 and a standardized score range of 0–100,
a higher score indicating a higher level of work-related fatigue (cut-
off point $54.5 ).
Posttraumatic stress was measured with the Dutch version of the
Impact of Event Scale [37,38]. The 15 items can be answered on a
4-point response scale (0 = not at all, 3 = often). Total scores range
from 0–75, a higher score indicating a higher level of posttrau-
matic stress (cut-off point $26 ).
Work ability was assessed with the first item of the Work Ability
Index (WAI) . This item concerns the evaluation of current
work ability compared to their lifetime best on an 11-point scale
Table 1. Screening instruments and cut-off points.
Aspect Instrument Cut-off point
Impaired work functioning Nurses Work Functioning Questionnaire (7 subscales)  Red score on $1 subscales and/or orange
score on $3 subscales 
Distress Four-Dimensional Symptoms Questionnaire, distress subscale [32,33] Total score$11 
Work-related fatigue Need for recovery subscale of the Dutch Questionnaire on the Experience
and Evaluation of Work 
Standardised total score$54.5 
Risky drinking behaviour AUDIT-C  Total score$5 for men, $4 for women 
Depression Brief Symptom Inventory, depression subscale  Mean score$0.42 
(Suicide risk) (One item from Brief Symptom Inventory, depression subscale ) (Score$3 on 0–4 scale)
Anxiety Brief Symptom Inventory, anxiety subscale  Mean score$0.42 
Panic disorder Patient Health Questionnaire , only assessed for participants identified
as having anxiety complaints
2 answers affirmative on the first 4 items
plus 4 symptoms affirmative on the
following 11 items 
Posttraumatic stress Dutch translation of the Impact of Event Scale [37,38] Total score$26 
E-mental Health Approach to WHS
PLOS ONE | www.plosone.org 4 September 2013 | Volume 8 | Issue 9 | e72546
(0 = completely unable to work, 10 = work ability at its best), a higher score
indicating a higher level of work ability.
All participants who completed the baseline questionnaire and
who screened positive on impaired work functioning and/or
impaired mental health (the targeted sample) were analysed, as the
work functioning and mental health of these participants could be
expected to change due to the intervention. However, since this
was not pre-specified in the trial registration, the analyses were also
performed with the total sample of participants (i.e. all partici-
pants, regardless of their screening results).
To describe participants, we used the following demographics:
sex, age, occupation, specialization (yes/no), years of working
experience, working hours per week, and type of contract.
Additionally, the number of participants scoring above cut-off
point for impaired work functioning and mental health complaints
The analyses were performed at the level of the individual
employee, according to the intention-to-treat principle. The
significance level was set at a = .05. All analyses were carried out
using the statistical package IBM SPSS Statistics 19.
Drop-out analysis. A drop-out analysis was performed to
detect whether dropping out of the trial was related to the
primary outcome impaired work functioning, and to identify
potential predictive variables of drop-out. Dropping out of the
trial was defined as completing the baseline and three months
follow-up questionnaires, but not the six months follow-up
questionnaire; or completing the baseline questionnaire, but
none of the follow-up questionnaires. Differences between drop-
outs and non drop-outs in impaired work functioning over time
in both separate groups were explored in graphs. If different
patterns of the effect after three months were detected, a Mann-
Whitney U test was performed to test the significance of the
differences. In the event of statistically significant differences, a
multiple logistic regression analysis was performed with drop-
out as the dependent variable, to identify potential predictive
variables for drop-out. Screening positive on mental health
complaints at baseline (yes/no) and age were included as the
independent variables, as we expected that these two aspects
might be related to dropping out of the trial. If the multiple
logistic regression analysis showed one or both of these aspects
to have a statistically significant effect on drop-out, they were
included as a covariate in the effect analyses .
Effect analysis. To analyse the differences over time
between the EMH-approach group and the control group on
each outcome, Linear Mixed Models (LMM) were applied. If the
assumption of a normal distribution of residuals was not met, a
log-transformation was used for the LMM and the median and
range were used to describe the outcome. Otherwise, the mean
and standard deviation were used to describe the outcome.
For each outcome, the scores at three and six months follow-up
were included as dependent variables in the LMM, while the
baseline score was included as a covariate. The main effects of
group and time of measurement, and the interaction of group*time of
measurement were included as fixed effects in the model. Ward (the
cluster level) and subject (the individual level) were included as
random effects; however if the cluster level did not have a
statistically significant effect, it was considered negligible and was
therefore excluded from the model. The effects of interest were the
main effect of group (interpreted as the difference between the
groups from baseline to six months follow-up) and the interaction
effect of group*time of measurement (interpreted as the difference
between the groups from three to six months follow-up).
For all outcomes in the positively screened subgroup, we
calculated Cohen’s d  by determining the mean difference
between the baseline score and the score at follow-up, divided by
the pooled standard deviation. For Cohen’s d, a score of 0.2 to 0.5
can be considered a small effect, 0.5 to 0.8 a medium effect, and
greater than 0.8 a large effect .
Additionally, the relative change scores of individuals on
impaired work functioning after three and after six months of
follow-up compared to their baseline score were calculated.
Individuals with a relative improvement on their NWFQ total
score of 40% or more, which is the minimal important change
(MIC) value of the NWFQ total scale , were defined as
relevantly improved. The percentages of individuals who had
improved relevantly in each group were compared using a Fisher’s
exact test, for both three months and six months follow-up.
Figure 1 presents the flow of participants through the trial. From
until April 26
, 423 employees (37%) started on the
baseline questionnaire. Of those, 366 (32% of invited employees)
were eligible for participation, 178 (31%) in the EMH-approach
group and 188 (34%) in the control group. In the EMH-approach
group, 80 participants (45%) were lost to follow-up, compared to
33 participants (18%) in the control group. Reasons for withdrawal
were not assessed. Fifty-six participants (31%) in the EMH-
approach group and 126 participants (67%) in the control group
completed all three questionnaires.
Analyses were performed on the participants who screened
positive (primary outcome: EMH N = 75, 54%; control N = 131,
81%), and additionally on all participants (primary outcome:
EMH N = 98, 55%; control N = 155, 82%) who had participated
in at least one follow-up.
Twenty-two participants (17 positively screened) logged into
Psyfit, seven logged into Strong at work, four logged into Colour your
Life, and no-one logged into Don’t Panic Online or Drinking Less. Nine
participants (all positively screened) followed an intervention to at
least some extent (Psyfit: 6, Colour your Life: 3).
Study population at baseline
As shown in Table 2,