Drug system change pilots evaluation: final report
Effectiveness bank home page. Opens new windowResearch analysis

This entry is our analysis of a study considered particularly relevant to improving outcomes from drug or alcohol interventions in the UK. The original study was not published by Findings; click Title to order a copy. The summary conveys the findings and views expressed in the study. Below is a commentary from Drug and Alcohol Findings.

Links to other documents. Hover over for notes. Click to highlight passage referred to. Unfold extra text Unfold supplementary text
Copy title and link | Comment/query |

Drug system change pilots evaluation: final report.

Callanan M., Ranns H., Turley C. et al.
London: NatCen Social Research, 2012.
Unable to obtain a copy by clicking title? Try this alternative source.

Based on the yardstick of successful treatment completions, government-funded research in England offers no assurance that recovery-oriented redesigns of local treatment systems have generated more or more rapid recovery from addiction than usual arrangements. Evidence was stronger for focused attempts to improve continuity of care for offenders.

Summary Funded by the Department of Health and Home Office, the Drug Systems Change pilot programme ran from 2009–2011 in England. The programme tested the potential to improve the way drug treatment and related social provision were delivered by adopting more user-led, outcome-focused approaches at regional and local levels in prisons and in the community. A primary goal was to promote recovery and reintegration, Pilot sites were given the opportunity to test a variety of approaches during a two-year period by using funding flexibilities and thinking innovatively about the commissioning, delivery and implementation of services.

Key points icon

Key points
From summary and commentary

Drug Systems Change pilots ran in seven areas in England between 2009–2011 to test whether funding flexibilities and innovation in commissioning, delivery and implementation of services would improve recovery outcomes and continuity of care.

Based on treatment completions, there was considerable doubt over whether the treatment system reforms generated more or more rapid recovery from addiction, but more evidence for improved continuity of care for offenders.

The evaluation was confined to the two-year pilot period, but changes such as those evaluated may take several years to produce measurable results.

Areas were invited to bid to run a pilot, and seven were selected. In 2009 a collaboration led by NatCen Social Research, an independent, not for profit social research organisation, was commissioned to evaluate the pilots. The featured document is their final report. In evaluating impact, it drew on administrative data on continuity of care, and on the proportions of service users in ‘effective treatment’ (treatment completed or treatment journey lasted at least 83 days) and who ‘successfully completed’ treatment (treatment journey ended because completed as intended rather than through drop-out or some other reason).

Main findings

Structural changes Pilots took three broad approaches to changing treatment system structures.

• Single-provider treatment systems: Funds were pooled and a single provider commissioned to deliver an integrated criminal justice treatment system. Strengths were thought to be that a single provider could streamline processes, improve continuity of care and create a coherent treatment system through a coherent management and governance structure. Challenges in implementing a model of this kind included streamlining and pooling commissioning budgets, reconfiguring existing targets, and developing performance management structures that supported a single-provider approach.

• Consortium-based approach: Aims were similar to those of single-provider systems. In two areas a consortium approach was seen as a precursor to creating a single-provider system, while in one, it was viewed as a way to promote partnership-working and coherence, while maintaining diversity of provision. Consortium-based approaches were characterised by regular partnership meetings and partnership working agreements that enshrined shared objectives and system-wide performance management structures. Challenges in implementing a model of this kind echoed those found in the single-provider model. Additional challenges included the complexity of creating coherent treatment systems while individual contracts continued to operate, and ensuring that – without the legal framework of a single, commissioned service – consortium partners could be held to account for the coherence of the treatment system as a whole.

• Improved partnership working: The final approach to creating more coherent treatment systems involved measures to improve partnership working without commissioning a single provider service or formalising partnerships through a consortium arrangement. This approach was characterised by regular provider meetings and workforce development programmes that fostered shared training and role-shadowing across the treatment system. Working groups were developed in one area to improve partnership working between drug treatment and mainstream service providers, with the aim of reducing duplication and improving provision in mainstream settings. Challenges in maintaining these arrangements centred on risks to sustaining partnerships without formal agreements and structures, and the danger that informal arrangements were more vulnerable in times of economic austerity.

System changes: continuity of care Greater continuity of care was identified as a primary goal for a number of pilot areas. Four broad approaches were adopted.

• Integrated community criminal justice system and prison drug treatment teams: With a single governance structure and integrated performance management, these re-structures were designed to promote seamless continuity of care. Key features included measures to improve communication via integrated IT systems, reconfigured ‘whole system’ performance management frameworks, and efforts to streamline referral, assessment and care planning paperwork. Pilots also explored different approaches to case management. Integrated systems of this kind were thought to have strong potential to improve service user treatment experiences by reducing duplication in assessment, promoting more seamless treatment, and improving communication across the system. Analysis of administrative data on the impact of integrated teams on continuity of care was largely inconclusive, although a positive impact was found in one pilot area. Indicating that these may be features of successful integrated teams, in this area the restructure included read-only access by prison staff to the community case management system, a single performance monitoring framework, and a streamlined referral and assessment process.

• Independent case managers: This approach provided continuous case management across the criminal justice treatment pathway via a case manager who supported the service user and acted as an independent broker of services to promote seamless treatment. Not tied to existing targets and structures, a strength was the perceived ability of the case manager to take a person-focused approach and to support service users in accessing a wide range of mainstream and drug treatment services. Analysis of administrative data showed that continuity of care improved in the area which adopted this approach. However, caution must be taken in attributing this improvement solely to the case manager role as it operated on a small scale and other elements of pilot activity may have contributed.

• Embedding criminal justice teams within community drug treatment services: The aim was to facilitate transitions from criminal justice pathways into community drug treatment. Simultaneously, Drug Interventions Programme and Drug Rehabilitation Requirement [two treatment-based programmes for drug using offenders] funding were combined to ensure a consistent range of provision was available to service users subject to either programme. A strength was thought to be improved communication across the treatment system and greater awareness among service users of the range of community provision available. By embedding Drug Interventions Programme provision in community treatment services, the pilot area was also able to improve the geographical spread of their provision, improving access to services. The number of referrals made from the community-based criminal justice teams into the broader treatment system rose during the pilot, suggesting that embedding the teams into community provision improved referrals. However, the proportion of these referrals successfully triaged within six weeks fell, suggesting that increased referrals strained service capacity.

• Linking roles: These involved targeting the transition in the treatment journey from custody to the community, where historically services have failed to join up and service users have been at risk of disengaging. One model involved contracting a third-sector organisation to provide ‘through the gate’ support on release from custody and ongoing support within the community. Another pilot area embedded a team of staff within their community criminal justice treatment teams, with the specific role of ensuring continuity across the system. This type of provision was often used in conjunction with the other measures to improve continuity of care already discussed. A key strength was that it provided support at the most vulnerable points in a service user’s treatment journey. Considered critical to its success was strong communication across the treatment system to ensure staff were equipped with up-to-date information on prison movements and transfers.

System changes: personalisation Personalisation is an attempt to improve treatment outcomes by ensuring services are tailored to the needs of individual service users. Two broad approaches were adopted.

• Self-directed support: A new way of working that gives service users greater choice and control over their treatment and support. This strategy was implemented in two of the pilot sites using pilot funding to pay for services not commissioned in block contracts. Within this model, service users were assessed and allocated a budget which they then used to ‘purchase’ services to meet their treatment needs. This highly personalised approach was felt to improve their engagement in treatment and to foster a sense of self-efficacy and responsibility for their own treatment. Challenges included: communicating the value and purpose of the approach to service users and staff; tackling the complexity of brokering a wide range of personalised provision; and reconfiguring provision to accommodate this approach. Delays in accessing services as a result of this complexity were a particular concern, as this was felt to jeopardise service user engagement. The pilots which adopted this strategy were too small to assess quantitative impact. However, staff and service users identified positive impacts on drug use, health, self-efficacy, and relationships with family and the wider community. This personalised approach was also felt to sharpen the treatment system’s focus on outcomes and encourage greater use of mainstream provision.

• Single point of assessment: One area sought to create a more personalised treatment system by developing a single centre to provide service users with an independent, comprehensive assessment. The aim was to ensure that all the needs of the service user were addressed and they were referred to a wide range of appropriate drug treatment and mainstream services. A key strength was thought to be the independence of assessment staff, who were not allied to any single provider so referred service users to a broad spectrum of services. Contact with the single point was also not limited to one assessment; service users could be re-referred or refer themselves back at any point during treatment or on completion. Service users valued the impartiality of the service and the co-location of other services including mentors and advice on housing and on education, training and employment. Staff also identified positive impacts including greater diversity in the range of service users seeking treatment, increased referrals from a range of agencies including GPs, and improved access to education and training opportunities due to referrals from the assessment service and the co-location of representatives from education and training services in the same building. Staff also identified challenges, including the potential for delays in accessing treatment if the assessment centre experienced high demand, and ensuring that the broad range of referrals were followed up and dealt with effectively. At the site piloting this approach, a positive impact was found on the proportion of service users in ‘effective treatment’. However, other elements of pilot activity may also have contributed. There was also a positive but not statistically significant increase in ‘successful’ treatment completions.

Delivery changes Alongside structural and system change, sites took the opportunity to re-orient treatment provision towards a greater focus on recovery and reintegration.

• Service user mentoring: Several areas took the opportunity to develop service user mentoring. The aim was to support current service users accessing treatment while simultaneously providing a stepping stone towards recovery and reintegration for those taking on the mentor role. Impacts identified for mentors were increased self-esteem, confidence and employability. For service users, mentors were felt to encourage engagement and provide positive role models. Key features of effective programmes were identified as good quality staff training and development, ongoing support and supervision, risk-management procedures, ongoing communication to staff and service users, and senior stakeholder engagement to champion the approach.

• Node-link mapping: [Node-link mapping was recommended by England’s National Treatment Agency for Substance Misuse to improve client-worker communication, focus, creativity, and recall. Working together, patient and practitioner place key ideas in boxes (‘nodes’) and connect them to other nodes with lines (‘links’) representing different types of relationships. The boxes can be different shapes to indicate, for example, trigger items, positive items, and decision items.] To improve the quality of assessment and care planning, a number of pilot areas rolled out training on node-link mapping methods and introduced these methods into their assessment and care-planning tools. Practitioners and service users spoke positively about the use of these techniques, describing them as more interactive and service user-led. In addition, evaluation forms designed to improve organisational self-evaluation and monitoring were used and supported providers to critically evaluate their own practice and take responsibility for the quality of their services.

• Family therapy or family group conferencing: Explored by several areas, the aim was to address the needs of the whole family and, in doing so, support the service user in their recovery. A particular focus was providing this form of support in prison. Two distinct models were adopted: single-episode family group conferencing; and systemic family therapy over multiple sessions. Pilots of this kind of support were typically small in scale reflecting their relatively high cost. A key challenge therefore was sustaining this provision post-pilot. In one area the service was discontinued because the cost was unsustainable, while another sought to embed group conferencing into its post-pilot provision through a service level agreement with a partner agency.

• Access to education, training and employment: Widely regarded as an essential element of recovery, it was within this context that pilot areas introduced initiatives to incorporate this provision. Initiatives included: building relationships with Welfare to Work providers; working with employers to dispel common misconceptions about substance misuse; providing education, training and employment mentors; creating a directory of existing provision to make better use of existing services; training staff to increase their awareness of these services; and using self-directed support systems ( above) to enable service users to identify and address their own training needs. Due to the challenging economic climate and unreliable data on employment, staff found it difficult to identify the impacts of this activity on service users. Budget cuts both locally and nationally (during and post-pilot) were identified as barriers to developments in this area. Particularly where partnership arrangements had been established between JobCentre Plus and local treatment services, there was concern that cuts would reduce capacity for this form of working.

• Stable housing: Service users identified stable housing as crucial to their recovery, and a housing crisis as a trigger for relapse. Measures piloted included training case managers to provide housing advice; co-locating local authority housing staff within a single point of assessment ( above) to improve access to advice and support; employing a dedicated housing officer to work with service users; emergency bed-and-breakfast accommodation for those released from custody; and advocacy to help service users access mainstream services. From service user interviews, positive impacts of housing support included help with addressing rent arrears and finding accommodation on release from custody. Despite some positive experiences of accessing housing support, service users continued to identify housing concerns as a risk to recovery. Difficulties in accessing appropriate accommodation continued to be a challenge across the pilot areas.

• Mutual aid groups: To support the creation of active recovery communities, some pilots fostered the development of mutual aid groups. Methods included supporting service users to set up their own groups by providing premises and budgets for room hire and refreshments. Pilot area personnel spoke of the importance of ensuring treatment staff had a good understanding of the range of mutual aid groups available and were equipped with the knowledge to refer service users appropriately. Peer mentors were also felt to play a role in raising awareness of mutual aid support. Some stakeholders raised the issue that mutual aid referrals were not recorded on the National Drug Treatment Monitoring System. It was felt that targets and the way in which activities were recorded needed to be revisited to avoid disincentives to providing support of this kind.

Impacts: effective treatment and successful completions The choice of these indicators was constrained by the data available, and the ability to detect impacts and attribute these to particular activities was limited by the complexity of the pilots. The timescale of the evaluation also precluded assessment of long-term impacts. It is important to take account of these factors when interpreting the findings.

• ‘Whole system’ pilots: Two areas were defined as ‘whole system’ pilots, reflecting their intention to effect change across the whole treatment system. Benchmarked against their comparison areas, in both there was a statistically significant increase in proportions of service users in effective treatment, suggesting that the whole system pilots may have bolstered this indicator. Neither pilot could be shown to have had a measurable effect on successful completions, a measure unlikely to be fully affected until some time after the pilots ended.

• Criminal justice pilots: Four of the seven pilot areas were criminal justice pilots, meaning they focused exclusively on improving treatment for service users in the criminal justice system. Because these sites were attempting to impact on a sub-sample of their wider treatment population, they could be expected to have a lesser impact on effective treatment and successful completion rates across the wider cohort. In three areas there were statistically significant negative trends in proportions in effective treatment, perhaps unsurprising as pilot activity might cause short term disruption; longer term follow-up may be required to determine whether these trends persist. In terms of successful completions, the picture was mixed, with no measurable impact in three areas and a significant negative impact in another. In interpreting these results, it is important to remember that these pilots focused on continuity of care between prison and community treatment. It would be reasonable to expect a time lag before such improvements affect service user outcomes.

The authors’ conclusions

Freedoms and flexibilities offered in the pilots were important in changing treatment systems. However, these were not always sufficient on their own to effect change, nor were they always a necessary foundation to reconfiguring local treatment systems; the active engagement and commitment of treatment providers and partner agencies was thought as important. With increased freedom at a local level to design drug treatment systems, there was potential for differences between localities to be exacerbated rather than reduced. In tackling continuity of care and ensuring coherent treatment systems that maximise outcomes for service users, attention must also be paid to coherence between localities as well as within them.

Given the importance of partnership working in effecting change, a key challenge in learning from the pilots will be how to achieve this level of partner engagement and cooperation without pilot status. A commissioning and funding framework that incentivises cooperation will be crucial if local areas are to develop coherent and innovative treatment systems.

The pilots have provided valuable learning in relation to tackling continuity of care. Pooled budgets, partnership arrangements, and system-wide performance monitoring, were viewed as important foundations to achieving continuity of care, as were strong leadership and workforce development. Fragmented communication across the treatment pathway remained a key barrier; whether a solution can be found will be critical to improving continuity of care.

Lessons can be drawn from pilot initiatives which tailored provision to the individual needs of service users. In particular, the single point of assessment system ( above) is similar in design and ethos to the Local Area Single Assessment and Referral Systems (LASARS) developed as part of the payment-by-results pilots. Facilitators of effective delivery included: independence from existing treatment providers; a central location for easy access; clearly defined referral routes; adequate numbers of staff with high quality training; and careful consideration of onward referral routes.

Because self-directed support ( above) was piloted on a small scale, it is difficult to estimate what its impact would be if rolled out more widely. There was some feedback from sites that this form of support was most effective for service users reaching the end of their treatment journeys as a way of supporting and sustaining recovery. The role self-directed support can play in aftercare and sustaining long-term recovery may be a fruitful area for further investigation.

Changes to service delivery ( above) were small in scale and delivered alongside other elements of pilot activity, making it difficult to assess their value and impacts.

From an economic perspective, the value of the improved outcomes relative to the costs of implementing the programmes was critical. Benefits were estimated from the number of clients who had improved outcomes, taking account of any changes that (on the basis of changes in the comparison sites) might have been expected even without the pilots. The two outcome measures were the numbers of clients in effective treatment and successful treatment completions. The benefit expressed in £s of an additional client in treatment or an extra successful treatment completion were derived from earlier studies, particularly the Drug Treatment Outcomes Research Study and a National Treatment Agency study of drug use careers. The calculations of total benefits paralleled those of the impact findings ( above), since both were based on the same outcomes. In the one pilot area where impacts were positive, the benefits too were found to be strongly positive. This occurred in large part because the pay-off for a single instance of each type of outcome is substantial. There are, however, substantial caveats in the case both of estimates of the value of an extra treatment completion or effective episode of treatment and the estimated impacts of the pilots.

Findings logo commentary Successful completion of treatment (especially if there is no short-term return to treatment) is considered to be the prime routinely collected indicator of the recovery outcomes sought by the pilot areas. Based on this indicator, there is considerable doubt over whether radical treatment system reforms in England (at least of the kinds evaluated to date) have generated more or more rapid recovery from addiction than usual arrangements. There seems stronger evidence for a more focused approach integrating criminal justice services to improve continuity of treatment.

The sole statistically significant change in trends in treatment completion rates was a large 20% reduction in one of the criminal justice pilot areas. The figure was calculated as the difference in the reductions from before to during the pilot periods in the pilot area versus its comparator area. The implication is that without the pilot area’s attempted improvements there would have been a 20% greater rise in treatment completions. In this area, pooled criminal justice treatment budgets and a consortium of providers sought to improve continuity of treatment. If this had worked to prevent breaks in treatment and retain patients in treatment the net effect on crime – the main component of cost-savings from treatment – could actually have been positive despite the relative fall in treatment completions. Whether there had been this kind of countervailing retention dividend is unclear. In the same area, a similar analysis yielded a significant reduction in the proportions of patients in ‘effective treatment’, a reduction which could have been due to poorer retention reducing the numbers staying for 12 weeks, or to extended retention reducing the numbers registered as completing and leaving treatment.

In the two ‘whole system’ pilot areas where treatment completion impacts should have been most apparent, there were no significant changes; in one area the non-significant trend relative to a comparator area was negative, in another, positive. Set against this, in both the proportions of the entire caseload categorised as having been in effective treatment significantly increased, meaning patients were more likely to stay for at least 12 weeks or complete treatment before 12 weeks. In both areas there was, however, a slight reduction in the numbers of patients in effective treatment from before the pilot period to during it. It means that on these two nationally recognised indicators, the whole system changes introduced in the pilot areas could not be shown to have been consistently successful.

In the criminal justice pilots the more salient measures reflected the continuity of treatment targeted by the schemes. Overall it would seem continuity of care for offenders with drug problems – an important overdose-prevention tactic – fairly consistently improved in the areas which focused on this objective.

One measure was the proportion of referrals from prison substance misuse counselling services (Counselling, Assessment, Referral, Advice and Throughcare or CARAT services) to community-based criminal justice teams, which had been triaged by those teams within six weeks of the prisoner’s release. Again the issue was the difference in the pilot area versus its comparator area in trends in successful referral from before to during the pilot period. In all four relevant areas successful referral rates improved more in pilot than in comparator areas, and in three cases the difference was substantial (16% to 22% greater increase) and at or near statistical significance.

In the one area where the issue was instead transfer of patients/offenders from community criminal justice teams to treatment services, the proportion of referrals which were successful fell substantially more in relation to the comparator area. However, in this pilot area both the numbers referred and those referred successfully increased substantially from before to during the pilot.

The featured report supplements that of another attempt to fundamentally re-organise drug treatment provision in England by introducing schemes to pay treatment providers per recovery-oriented outcome they achieve. Like the system reforms tested in the featured evaluation, pilot payment-by-results schemes did not improve treatment completion figures. Compared to trends in other areas, these schemes appear to have led to slightly fewer successful completions of treatment and more patients declining treatment.

Last revised 30 September 2016. First uploaded 23 September 2016

Open Effectiveness Bank home page

Top 10 most closely related documents on this site. For more try a subject or free text search

DOCUMENT 2017 Better care for people with co-occurring mental health and alcohol/drug use conditions: a guide for commissioners and service providers

DOCUMENT 2017 Drug misuse and dependence: UK guidelines on clinical management

STUDY 2006 Lessons of failure of Scottish scheme to link released prisoners to services

MATRIX CELL 2018 Drug Treatment Matrix cell E2: Treatment systems; Generic and cross-cutting issues

DOCUMENT 2016 Coexisting severe mental illness and substance misuse: community health and social care services

DOCUMENT 2010 Commissioning for recovery. Drug treatment, reintegration and recovery in the community and prisons: a guide for drug partnerships

STUDY 2011 The family drug and alcohol court (FDAC) evaluation project: final report

DOCUMENT 2010 The Patel report: Reducing drug-related crime and rehabilitating offenders

STUDY 2016 After FDAC: outcomes 5 years later (final report)

STUDY 2017 The evaluation of the Drug Recovery Wing pilots: Final report