Rohrbach L.A., Sun P., Sussman S.
Preventive Medicine: 2010, 51, p. 313–319.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Rohrbach at firstname.lastname@example.org. You could also try this alternative source.
Disappointing results from this first evaluation of widespread dissemination of the Project TND drug education curriculum reinforce concerns that with usual schools, teachers and pupils and usual training and support, programmes previously found effective may not live up to their promise. The possibly important exception is in respect of curbing 'hard drug' use.
Summary The featured study tested whether a US drug education curriculum for secondary schools remains effective when widely disseminated, and whether its impact is greater when teachers are given more post-workshop training and support.
Despite research support, in the USA few schools use effective school-based substance use prevention programmes. A critical public health challenge is to ensure these programmes are widely implemented and sustained once proven effective.
One such programme is Project Towards No Drug Abuse (TND), nationally recognised as effective on the basis of randomised trials which found it reduced adolescents' use of alcohol, tobacco, cannabis, and 'hard drugs' Stimulants, cocaine, hallucinogens, inhalants, ecstasy, and other drugs including PCP, depressants, steroids, and heroin. in the short term, and that effects on the latter remained up to five years later. The curriculum targets substance use and violence during 12 classroom sessions designed to be implemented over four weeks. Interactive teaching techniques aim to enhance motivation, provide information about the consequences of drug use, correct misperceptions, develop communication, coping, and decision making skills, and promote smoking cessation.
The featured trial tested two ways of training and supporting teachers to implement the curriculum:
• Workshop-only: A one-day workshop conducted at the schools by a trainer certified to train teachers in the curriculum. They summarised its theoretical foundations and evidence base, provided detailed instruction about each lesson, and offered opportunities for teachers to practice key activities.
• Workshop plus continuing support: Additional to the workshop, this offered two on-site coaching sessions from the trainer each based on observations of a lesson, web-based support (discussion forum; teaching tips; downloadable scientific articles), and on-call technical assistance from programme specialists.
The substance use of the pupils where Project TND was implemented in these ways was compared with that of control A group of people, households, organisations, communities or other units who do not participate in the intervention(s) being evaluated. Instead, they receive no intervention or none relevant to the outcomes being assessed, carry on as usual, or receive an alternative intervention (for the latter the term comparison group may be preferable). Outcome measures taken from the controls form the benchmark against which changes in the intervention group(s) are compared to determine whether the intervention had an impact and whether this is statistically significant. Comparability between control and intervention groups is essential. Normally this is best achieved by randomly allocating research participants to the different groups. Alternatives include sequentially selecting participants for one then the other group(s), or deliberately selecting similar set of participants for each group. schools where it was not implemented at all. The expectation was that a year later substance use in TND schools would have increased less than in control schools, and especially so in schools where TND teachers had been offered continuing support. These expectations were bolstered by initial findings that relative to controls the TND pupils had changed in ways thought to underpin the curriculum's impacts, and that the programme had been best implemented by teachers offered post-workshop support.
To test these expectations the study had recruited 65 US secondary schools from 14 school districts (at least three schools in each district) whose staff had contacted Project TND to ask about the programme. Each school had agreed to recruit a teacher to be trained and deliver the curriculum to two of their classes. Within each district, schools were randomly assigned to act as controls or implement the curriculum after one of the two levels of training and support.
Across all 65 schools, of 4351 pupils in the selected classes, 3346 (about three quarters) completed pre-programme questionnaires, which among other things assessed the frequency of their substance use over the past 30 days. Of these, 2538 (58% of all pupils in the classes) satisfactorily completed follow-up questionnaires a year later. They averaged about 15 years of age and were less likely to be substance users and had more promising family and socioeconomic backgrounds than pupils not followed up. About 13% had each smoked tobacco or cannabis in the past 30 days, nearly a third had drunk, and nearly 6% used 'hard drugs'.
Generally the expected effects of the curriculum and of enhanced training/support to implement it were not apparent in the substance use of the pupils a year later. The only finding to reach the study's criterion for statistical significance was that fewer pupils who had initially not used these drugs had gone on to use 'hard drugs' in the past 30 days if their classes had been allocated to Project TND. Though not statistically significant, there were also appreciable relative reductions across all pupils in the proportions who had used cannabis and 'hard drugs' in the Project TND classes; for each the chances of a child using these drugs versus not using them was about 75% of that in control schools. Across all pupils there were no appreciable impacts of Project TND on smoking tobacco or drinking, and in no case did enhanced training and support significantly improve on workshop-only training.
This was the first study of the impact of Project TND in the context of a trial of its widespread implementation by usual classroom teachers offered just a day's training or more extended but still not intensive support. Results suggest that in these circumstances the programme may prevent some teenagers initiating use of 'hard drugs' and perhaps too help prevent cannabis use. However, the lack and/or small size of the impacts suggest that as the programme is more widely implemented in real-world mainstream school settings rather than highly controlled trials, Project TND will become less effective.
These findings are consistent with several replication studies which have found previously efficacious prevention programmes have weaker or no effects when evaluated with different target populations, in different settings, or using different kinds of staff. When programmes are 'scaled up' from small efficacy trials, often the target population is broader, the quality of implementation lower, the programme may be substantially altered, and the local infrastructure may not be ready to support high-quality implementation.
In the case of Project TND, it was developed for pupils being educated in special non-mainstream schools, who also tend to be more likely to use drugs. Possibly effects are less noticeable among pupils in mainstream schools – the great majority in the featured study.
Though lesson observations indicated that teachers who received enhanced training did more adequately implement the programme, still outcomes for their pupils were no better. In practice, possibly the two training approaches were insufficiently distinct to make a difference. Teachers rarely accessed the extra elements they had to use on their own initiatives, leaving the two on-site coaching sessions as the main additions to the workshop.
While the conditions under which TND was implemented in this study were closer to the 'real world' than is typical in efficacy studies, nonetheless schools and teachers were participating in an evaluation study and being monitored, which may have motivated them to implement the programme more adequately than usual. Also, the findings may apply only to schools with teachers willing to implement evidence-based prevention curricula.
commentary This is not the first time that a prevention programme found effective in small studies in selected schools with highly trained and supervised (often by the programme's developers) teachers or external specialists has not maintained its promise when extended to the general run of schools. Project ALERT, a programme considered promising by US government assessors, offers another example. It leaves the general verdict on drug education unchanged – that however valuable in purely educational terms, internationally and specifically in major UK and European trials, it has proved an inconsistent and usually at best minor contributor to the prevention of unhealthy substance use.
As the authors point out, the reasons why programmes fail when widely disseminated may include the common and sometimes substantial alterations teachers prefer to make or have to make due to the classroom situation and time pressures, a process another study observed in respect of TND and a second curriculum. A common variation was to replace individual with group work and vice versa, variations which could mean the programme is ineffective, effective, or for some pupils, counterproductive. Taught by selected, monitored, and highly motivated health education teachers from the schools, the programme can be almost as well delivered as by TND specialists, and have similar impacts on substance use, but other teachers might struggle with its interactive methods (or conversely, introduce unintended interactivity) and to complete the curriculum as intended.
Also, youngsters who are relatively likely to use substances offer greater scope for prevention programmes to prove their worth. TND in particular proved itself among high-risk pupils removed from or refused mainstream education. Apart from the featured study, the other test of TND in general schools involved just three schools, over a third of the pupils could not be followed up, and the research project's health educators did the teaching, not the schools' own teachers, weakening generalisability to the general run of pupils, schools and teachers. No impacts were found on cannabis or tobacco use a year later and nor were there any on the majority At baseline just 38% had drunk at all. of pupils who had not drunk relatively frequently before the lessons. As baseline drinking increased to unusually high levels, the programme did curb further increases. At the start of the study over 90% of pupils had not tried any of the mixed bag of 'hard drugs', and on these the programme had a small impact which grew to be substantial among what must have been the very few pupils who had used these substances relatively frequently before the lessons. It was unclear from the report whether across all the students there were any statistically significant gains from the programme.
Together with the featured study, these results provide only slim support for the use of the curriculum across the general run of schools and by the general run of teachers. In its favour is that in these schools and in non-mainstream schools, impacts seem greatest and most consistent for the 'hard drugs' society is most concerned to stop its children using (1 2 3). However, hopes for a long-term impact on use of these substances rest on a study of special California schools for pupils diverted from mainstream education, in which fewer than half the pupils could be re-assessed four or five years after the programme had been implemented. More were assessed about two years earlier when no such impact was seen, raising the possibility that the later results were an artefact of loss to follow-up.
Thanks for their comments on this entry in draft to research author Luanne Rohrbach of the University of Southern California in Los Angeles, USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.
Last revised 22 August 2013. First uploaded 14 August 2013
STUDY 2000 Education's uncertain saviour