Crits-Christoph P., Ring-Kurtz S., Hamilton J.L. et al.
Journal of Substance Abuse Treatment: 2012, p. 301–309.
Unable to obtain a copy by clicking title? Try asking the author for a reprint by adapting this prepared e-mail or by writing to Dr Crits-Christoph at firstname.lastname@example.org. You could also try this alternative source.
Evidence that an earlier study feeding back client progress to counsellors did not find improved outcomes because data was aggregated across a caseload rather than identifying individuals doing poorly and recommending remedial actions. After remedying these deficits, a new system significantly improved mental health and reduced substance use.
Summary The featured study builds on an earlier study led by the same researchers which also offered counsellors at non-methadone substance use disorder counselling services regular feedback on their patients' progress in an attempt to improve outcomes. It found no evidence of the expected improvements in drug or alcohol use or treatment process. One reason it was thought was that clinicians were deprived of what for them may have been key information – which individuals were not doing well and might benefit from a change in approach. Instead they were given average figures for their caseload. The researchers argued that clinicians' intrinsic motivation for general performance improvements may be marginal.
Based on the Outcome Questionnaire System, a standardised system for feeding back client progress to therapists, the featured study tested changes intended to make feedback more effective. Feedback was provided on each identified individual, and instead of the patient's assessment of the therapeutic alliance and satisfaction with treatment, a questionnaire Developed for mental health patients. assessed their psychiatric wellbeing and functioning and An addition for the study. their actual drug and alcohol use. These assessments were made just before each counselling session at the start of treatment and then for another 12 sessions or six months, whichever came first. Therapists could immediately see the patient's responses as automatically tabulated and charted by the research project. Other differences were that this was applied to patients as they started treatment, not some time in to treatment, and that patients and counsellors were engaged in one-to-one counselling rather than group therapy.
On the basis of their progress compared to that expected for patients with a similar severity of problems, the revised feedback also categorised patients as doing so well that treatment might be ended, progressing as expected, or doing less well than expected – the so-called 'off-track' patients. It was these patients in particular that the feedback system was expected to help retrieve from continued poor progress. For them the therapist was instructed to consider changing the treatment and given guidance based on a second questionnaire these patients only were asked to complete. This rated barriers to progress in the form of a weak therapeutic alliance, poor motivation, inadequate social support, and stressful life events. Counsellors were trained to interpret the web-based reports made available to them and to consult a manual which offered suggestions Some specific examples of treatment interventions for the therapeutic alliance domain are as follows: work with resistance by retreating when necessary and being supportive; allow the client to assert his or her negative feelings about the relationship; and give and ask for feedback on the therapeutic relationship. Examples of interventions for the social support domain are the following: assess client's social network; use progressive desensitisation to aid client in overcoming social anxieties; and encourage pursuit of hobbies, participation in clubs, service projects. Examples of interventions for the motivation domain are the following: assess how important it is for the client to make target changes, having him or her rate the desire to make specific changes on a scale of 1–10; discuss the client's important values, goals, and aspirations; and ask open-ended questions to see if the client is ready to make a commitment to change. For the fourth domain, life events, the manual instructs the therapist to explore the stressful event and to facilitate the development of coping strategies to help the patient establish stability. for how to respond to areas of concern. When these did not seem to account for poor progress, counsellors were trained to consider a different kind of treatment, such medications rather than just counselling.
At first the three clinics in the study simply asked patients to complete the first questionnaire at baseline and for 12 sessions/six months, but counsellors were given no feedback. Then the feedback system was implemented. Scores on the first questionnaire were used to determine whether patients improved further as a result of offering their counsellors feedback. These scores were available only for patients still in treatment; by session 12, nearly two thirds had left.
In the pre-feedback period 165 patients supplied baseline and at least some follow-up data, and 139 in the feedback phase. Typically they were unemployed men with long-standing alcohol and/or drug problems, but over three quarters were no longer using these substances at the start of the study. About 40% were calculated to be doing less well than expected – off-track At the start of the study these off-track patients had significantly higher scores overall indicating greater problems. However, the number of days on which they said they had drunk or used drugs in the past week did not significantly differ from the rest of the patients, though on average they did initially admit to more days of drinking. – at some stage in their treatment and (in the feedback phase) were asked to complete the second questionnaire.
On all three measures (drug use, drinking, total problem scores) off-track patients whose counsellors were offered feedback on their progress and suggestions about how to respond progressed better than similar patients before the feedback system had been implemented. In respect both of drinking and drug use, these at first poorly progressing patients ended the study drinking and using drugs as little as the ‘on-track’ patients However, only in respect of days of drinking in the past week did the trends differ to a statistically significant degree.
Instead of trends from baseline, another analysis followed up the off-track patients only from the time counsellors were alerted to their poor progress – an attempt to assess the added value of the assessment of what was holding them back and its feedback to therapists along with suggestions about how to respond. This did seem to trigger positive change. Up to the point when counsellors were alerted there was little difference between feedback and non-feedback patients, but from then on feedback patients progressed much better on all three measures (drug use, drinking, total problem scores), though now it was the results for drug use and total problems which diverged to a statistically significant degree, while days of drinking did not.
Finally, feedback made no apparent difference to how long patients in general or off-track patients in particular stayed in treatment.
This study suggests that outcome assessment/feedback systems may be of value for enhancing treatment outcomes in substance use treatment clinics. Among patients not progressing as expected ('off-track'), feedback to their counsellors improved outcomes relative to no feedback on all three measures of psychiatric symptoms and functioning, drinking, and drug use, albeit at different points in the feedback process. In contrast to drinking, which responded to feedback from the start, additional improvement in drug use and psychiatric health were not evident for the off-track patients until counsellors were alerted to their being off-track and offered suggestions about how to respond.
These encouraging results should be seen in the light of the study's limitations. Patients were not randomly assigned but sought treatment before or after the feedback system started; this time gap may have seen some changes in the type of patients or other factors. Neither was the study able to identify which diagnostic categories of patients benefited most from the feedback system nor precisely how this improved outcomes, because there was no data on which interventions counsellors implemented in response to feedback reports.
commentary The implications (supported – see below – by the general psychotherapy literature) of this intriguing study are that treatment services can 'rescue' patients who are not doing well first by identifying them, probing why this is happening, and then offering appropriate responses. One would hope all this was being done routinely as part of good clinical practice, but at these three clinics and with these patients, systematising the process, basing it on evidence about how patients normally progress, and offering scientifically based and/or well worked out responses, did improve outcomes.
However, enthusiasm should be tempered with an appreciation of the clinical as well as statistical significance of the findings. On average patients started the study drinking less than a day a week and using other drugs even less often. Greater improvement from this already low starting point may not signify clinically significant added value. Also, the crunch issue is what the feedback system as a whole achieved, and here the sole statistically significant gain was in reduced drinking, and even without feedback, by the end of the study patients were roughly drinking on average just once a fortnight. Arguably too, a system designed to intercept impending treatment failure is best judged against how many patients met pre-set criteria for failure/success, not average days of substance use.
Several factors might have helped make the difference between an ineffective feedback system in the previous study and an effective one in the current study. Of these, it seems entirely plausible that (as the authors surmise) the active ingredients included individualisation of feedback, its more concrete nature including actual substance use (not just how patients felt about the therapy), and the guidance given to counsellors on how to respond.
This interpretation would be in line with psychotherapy studies of the same system reviewed for the American Psychological Association. The reviewers concluded that real-time feedback enabling therapists to monitor patients' responses to psychotherapy and satisfaction with the therapy relationship improves outcomes overall, and even more so for clients at risk of deterioration or drop-out – the 'off-track' patients of the featured study. Yet more benefit is gained from the additional strategy deployed in the featured study of supplementing feedback with information from off-track patients indicating why things were going wrong and guidance on what therapists might do about it.
If these were indeed the factors which made the difference, it remains to be explained why they did so. The predecessor to the featured study was included in a review by US authors of ways to improve performance of substance use disorder treatment systems. They took its findings as an instance of the more general finding that "if there is no risk of reputational damage, information on one's individual or organizational performance is generally disregarded". They argued that only if this information is made public in circles that matter to the clinician (like their employer) or to the organisation (like commissioners and prospective patients) does it exert leverage.
But this new study suggests that when it concerns the individual patient's 'performance', feedback does have an impact, even when As it appears from the report on the featured study. the counsellor is at no direct risk from disregarding it. Possibly the rather stark warnings that "This client may end up with no significant benefit from therapy", or that patients doing even less well "may drop out of treatment prematurely or have a negative treatment outcome", alerted counsellors to the possibility that variables monitored by management like how many of their patients satisfactorily completed treatment would be affected if they did nothing about the feedback. More positively and as the authors imply, it could also be that people who undertake this work because they want to make life better for the individuals they are seeing grasp an opportunity to do so which requires little effort to grasp; feedback and possible responses were 'laid out on a plate' for them; they did not even have to administer the questionnaires which gathered this information.
This likely explanation is however not the only one. If therapists did discuss patients' scores with them, perhaps the revelation that they were doing relatively poorly was itself enough to jerk them in to doing better – or at least submitting questionnaire returns which indicated this. The pattern of the findings and the general psychotherapy literature suggests this is unlikely. It could also be that the experience of completing the second questionnaire exploring why things might be going wrong in treatment had its own impacts, regardless of how the counsellor responded. Other possibly pertinent differences between this and the previous study were that patients had to consent to a system which would identify them to their counsellors, including whether they were still drinking or using drugs. The resulting caseload might have been relatively highly selected and motivated. Outside the context of a research project requiring informed consent, a less promising set of patients might respond less well. Also, the feedback system in the featured study started right at the start of treatment before patterns and relationships were established, not part way through, and the earlier study was concerned with group therapy involving week to week a possibly different set of patients. It seems probable that this limited the degree to which counsellors could respond to poor progress by any individual in the group, and also that even if they did, this might have less effect than in one-to-one therapy. The timeliness of the feedback offered in the featured study may also have been relevant. What might have been a considerable investment in computerisation technology and expertise meant patients could complete feedback questionnaires just before a session started, and that data could be analysed and organised in to an assimilable format and fed back to the counsellor in time for them to use during the session.
It is not clear why drinking fell from the start in the feedback phase but the other outcomes only after counsellors were alerted to poorly progressing patients. Apart from the interpretations offered by the authors, it seems possible that an institutional focus on drinking, or the fact that off-track feedback patients were at treatment entry still drinking on average nearly a day a week (in the context of the study, unusually high), meant that all it took to prompt the counsellors to deal with the drinking was to know that it was happening. As the authors speculate, in this substance use clinic, psychiatric problems might not trigger a response until counsellors were told in no uncertain terms that these were among the complex which for off-track patients jeopardised the entire treatment process.
Apart from the caveats cited by the authors is the fact that very few patients were left by the end of 12 sessions. The study did not attempt to assess those who left treatment, leaving its conclusions based on fewer and fewer patients as it progressed. What happened to those who left could substantially alter the impression given by the minority who stayed. A related point is that it remains to be explained why a system designed to identify and intercept impending treatment failure which might lead to premature drop-out did not improve retention. At its best it could be that the patients who were rescued from premature drop-out were counter-balanced by those identified by the feedback system as ready for discharge, or that the system elevated the progress of off-track patients to the point where they could safely leave rather than drop out and relapse.
The other main US system for providing feedback on client progress is the Partners for Change Outcome Management System (1 2). It was trialled among US soldiers enrolled in outpatient group therapy at an army substance use treatment programme in 2007 and 2008, typically because they had been referred there by their commanding officer after alcohol or drug-related misconduct. All the soldiers met diagnostic criteria for substance abuse or dependence. Apart from the feedback system’s own assessments, each patient’s commander (who did not know which soldiers had been allocated to feedback) and therapist rated their behaviour and conduct both generally and with respect to substance use. There was, however, no direct measure of substance use.
The 263 soldiers were assigned to therapy groups run by 10 therapists. Within each group roughly half the patients had been allocated at random to have their progress fed back to the therapist and half not. The therapists were presumably limited in the extent to which they could (as the system intends) discuss feedback results with patients since others in the group were not subject to this process, and some of the system’s group-related assessments had yet to be developed. Nevertheless, the results closely duplicated those seen in the featured review: small but significantly greater improvement from before to the end of therapy in feedback versus non-feedback patients, gains not focused on those flagged as progressing below expectations but spread across the entire sample.
Specifically, significantly greater improvements were seen in the system’s own assessments of functioning and wellbeing and in commander and therapist ratings. More feedback patients (28% v. 15%) completed treatment having substantially improved and also having reached scores on the system’s assessments no longer associated with clinically significant mental ill-health and poor functioning. Improvers were recruited from patients who would otherwise have registered no improvements rather those who deteriorated; proportions of the latter did not significantly differ among feedback versus no feedback patients. Feedback patients also attended more of the intended five sessions, and were less likely to drop out prematurely before having reached assessment scores no longer associated with clinically significant mental ill-health and poor functioning. However, attendance was not the active ingredient in how feedback improved outcomes; even among those who had attended all five sessions, greater improvement was seen in feedback than non-feedback patients.
Thanks for their comments to Paul Crits-Christoph of the University of Pennsylvania Medical School in the USA. Commentators bear no responsibility for the text including the interpretations and any remaining errors.
Last revised 15 February 2019. First uploaded 03 October 2012
DOCUMENT 2013 Sometimes best to break the rules
REVIEW 2018 Cohesion in group therapy: a meta-analysis