Scottish Government

Interchange 77: Evaluation of the Matrix Project

This document is also available in pdf format (196k)

Background

The Matrix Project is a programme that has been established in central Scotland with the aim of reducing the risk of offending and anti-social behaviour among vulnerable children. The project is led by Barnardo's and was funded from 1999 to 2002 by the HM Treasury Invest to Save Initiative in partnership with the Scottish Executive. It provides a range of services to children between eight and 11 years of age who are at risk in the local authority areas of Clackmannanshire, Falkirk and Stirling. It builds upon well-established working relationships between Barnardo's, the local authorities and other relevant agencies and adopts an 'evidence-based' approach to the services provided. The objective of the work undertaken by Matrix is to reduce the risk factors associated with offending and anti-social behaviour and promote those factors that provide children with resilience in the face of these risks.

The Scottish Executive commissioned an evaluation of the project aimed at describing the operation of Matrix and assessing its effectiveness in reducing offending and anti-social behaviour among children. The study was carried out between February 2000 and October 2001 by the Social Work Research Centre at the University of Stirling.

Aims and methods of the study

The main objective of the research was to describe and evaluate the approaches adopted by Matrix in their work with vulnerable children and their families. The methodology was primarily qualitative on account of the relatively low number of children who had received services from the project: information was available in respect of 17 children, 13 of whom were no longer attending Matrix. The methods used included:

• the collection of information from project case-files
• interviews with project staff
• interviews with a range of professionals who have contact with vulnerable children
• interviews with children and their parents
• observation of referral meetings.

The original intention had been to identify a comparison group of children who were assessed as suitable for Matrix but who received 'normal' services, that is, the services these children would have received if Matrix did not exist. However, the absence of referrals to the project when it was known to be operating at full capacity prevented the establishment of a 'waiting list' from which comparison cases could be drawn. This limits conclusions that can be reached with respect to the effectiveness and cost effectiveness of Matrix in relation to mainstream services.

Matrix objectives and criteria

The objectives of the project and the criteria for suitability for Matrix were developed by a multi-agency steering group prior to the appointment of the staff group. The Matrix Project was initially staffed by a project leader, three project workers and an administrator, and two sessional workers were subsequently employed. Matrix provides services to the families of children who fall into three categories:

• those who have a history of referrals to the reporter on care and protection grounds and two or more referrals on offence grounds ('high level offending category' or 'Level Three')
• those with a history of care and protection referrals who are referred on offence grounds for the first time ('early intervention category' or 'Level Two')
• children may also be referred to Matrix even though they have not been referred to the reporter on offence grounds if they have a cluster of risk factors in their lives which place them at risk of offending behaviour ('Level One').

The objective of the project is to provide intensive support to families whose children fall into one of these categories with a view to reducing risk factors and enhancing protective factors. Other professionals welcomed the project's emphasis upon early intervention, although project staff observed that the behaviour of even the youngest children they worked with was often entrenched. Whilst this might suggest that the age range of children attending Matrix should be extended downwards, staff were also cautious about the potential effects of labelling very young children.

Referral to Matrix

Perspectives on the referral process

Cases could be referred to Matrix from a variety of sources but all referrals had to be channelled through a local referral group in each of the participating local authorities. The procedures for considering referrals differed from area to area: in two areas referrals to Matrix were considered alongside referrals of children to other specialist resources while in one area a group had been established specifically to consider Matrix referrals. Decisions about suitability for Matrix were made by the referral groups on the basis of information provided by the referrer (including an assessment of risk and protective factors undertaken by a social worker and by the child's school), with Matrix having no input into this process. It appeared on the basis of the characteristics of children referred, that each local authority had different perceptions of the type of service they wished Matrix to provide.

Social workers had received more information than other professionals about Matrix and were conversant with the referral procedures and criteria. However, there was some uncertainty among other professional groups with respect to whether they could refer children to the project, which some tended to perceive as primarily a social work resource.

The characteristics of children referred to Matrix

The Matrix Project began accepting referrals in September 1999. By September 2001 a total of 23 children (from 19 families) had attended Matrix and 17 of these cases had been closed. The families of 17 children agreed to the researchers having access to their project files. Sixteen of these children were boys. Background information on the children was often sketchy, but most were known to have had previous referrals to the reporter on offence and/or care and protection grounds. Around one half of the children were known to be subject to home supervision requirements or voluntary supervision through the children's hearings system. Other agencies known to be involved with the children included educational psychologists, psychiatrists and the police.

The Matrix project workers undertook an assessment of risk and protective factors among the children referred to the project. The risk factors most commonly identified were:

• offending behaviour and anti-social behaviour
• poor parental child management techniques
• poor parental supervision of their children
• low educational attainment
• anti-social behaviour at school
• risk of exclusion from school
• disadvantaged neighbourhood
• community disorganisation and neglect
• availability of drugs
• high housing turnover and lack of community attachment.

In other words, the children tended to come from disadvantaged areas and from families who had not acquired effective techniques for managing their children's behaviour. The children were mostly offending and exhibiting anti-social behaviour that was adversely affecting their school performance and placing them at risk of exclusion.

The Matrix assessment also identified the presence of protective factors that might provide children with resilience in the face of these risks. The most common protective factors were:

• good physical health
• strong bonds in infancy
• an absence of prolonged separations.

Those factors least likely to be present among the children referred to Matrix were positive peer influence and a strong social support network for the child and family.

The work carried out by Matrix

The Matrix approach

Once a child is accepted by Matrix, the allocated Matrix project worker undertakes an assessment of the child and family, lasting on average three months. This phase culminates in the development of a Family Agreement that sets out the difficulties identified, the goals to be achieved to address them and the task to be undertaken in pursuit of the goals. The work undertaken with the child and family is subject to eight-weekly reviews at which progress is assessed and goals and tasks are identified for the next phase of the work.

The areas of difficulty most commonly identified in the initial Family Agreements included:

• anti-social behaviour at home, at school and in the community
• aggressive behaviour and difficulties in relationships with peers
• parents unable to control child and establish boundaries.

The goals most often identified to address these difficulties included:

• offending-related goals (to reduce offending, dangerous or anti-social behaviour)
• educational goals (such as considering others' feelings, improving school-home links and getting along better with other children)
• community goals (including finding positive interests for children and parents)
• child-related goals (including helping the child to explore his/her feelings and develop internal behavioural controls)
• parent-related goals (including developing positive family bonds, confident parenting and the ability to set boundaries).

The tasks that were agreed as a means of achieving these goals ranged from the general to the specific. The former included, for example, agreement about the level of contact between the Matrix worker and the parents and/or child or about the sharing of information between different parties. The latter included tasks such as the parent praising their child's positive behaviour or attendance by the child at after-school activities.

Matrix staff described their approach as 'holistic', 'systemic', 'ecological', 'solution-focused' and 'family-focused,' and informed by research that had identified the risk and protective factors for offending and anti-social behaviour. The project stressed the importance of involving parents actively in the intervention since, as one project worker put it, "the parents have the potential to impact on all the risk factors". A particular strength of the Matrix approach was perceived by other agencies as the ability of the staff to engage families who had a history of social work involvement and were resistant to intervention by welfare or other statutory agencies.

Methods and contexts of intervention

The project employed a range of methods of intervention including counselling, cognitive and cognitive behavioural methods, advocacy, solution-focused work, problem-solving, task-centred casework and systemic methods. The work was undertaken in a variety of places, including the project premises, family home, school and other community locations. The intervention involved individual work with children, individual work with parents, work with the family as a unit, and groupwork with children. Groupwork with children focused, in the main, on helping children to get along with others while the parents' group enabled parents to discuss their problems and identify solutions through interaction with others in a similar situation.

Individual work with children was aimed at getting to know children and their perspectives on their problems, helping to build their self-esteem and relationship skills and helping to improve their relationships with their parents. Much of the work took place through informal activities, which provided an opportunity for staff to model and reinforce appropriate responses and behaviours. Offending was usually not addressed directly, but rather through the development of other insights and skills.

Individual work with families focused upon relationship difficulties and poor family bonds and aimed to promote improved parenting skills and improved parental supervision and management of their children. Project workers explained that their work with parents was often aimed at addressing other problems (such as debt or housing problems) as a means of alleviating the stress experienced by families. Matrix also engaged in joint work with families, which often took the form of family activities aimed at encouraging the family to spend time together as a unit. Informal activities of this kind also provided opportunities for Matrix staff to engage in role modelling and to provide parents with help in managing the behaviour of their children.

Inter-agency work

The project stressed the importance of inter-agency work and, in particular, its role in co-ordinating the services provided by a range of agencies. Inter-agency arrangements were most formalised in respect of social workers and teachers, who usually attended Matrix case reviews. The project also encouraged families to make use of other services and resources in the communities, such as health and mental health services, alcohol services and advice agencies.

Other professionals involved in providing services to Matrix families were content with the level and quality of communication with Matrix staff. Duplication of services was acknowledged as a possibility but where this had happened it had tended to reinforce rather than undermine the work being undertaken by other services. Matrix staff and other professionals also recognised the potential for a blurring of professional boundaries that might result in confusion for families, but this was not thought to have been a particular issue.

Case closure

When Matrix was established, the intention had been to provide special support to families for a period of around three or four months. In practice, however, the mean length of project involvement with a family was 12 months, with a range of between five and 20 months. The extended involvement of Matrix appeared to be a result of further problems coming to light as the project worker gained more knowledge of a family and there was no evidence that Matrix were systematically keeping cases open for longer than was necessary. Other professionals did not believe that cases were being kept open too long and commented that it was unrealistic that sufficient change could be achieved in families with multiple problems within the timescale originally envisaged. Project staff, in turn, appeared alert to the consequences of continuing to work with families longer than was necessary.

How effective is Matrix?

A comparison of risk and protective factors at the start and end of intervention suggested that Matrix had impacted positively upon offending and anti-social behaviour, the child's physical health, parental management and support, and school attainment and behaviour at school. Protective factors that had been enhanced included a positive home environment, good home-school links and positive peer influence. On the other hand, some risk factors remained unchanged, usually because these were either 'historical' and, therefore, 'fixed' or beyond the scope of a project of this kind. Some risk factors - such as parental substance misuse - became more prevalent, presumably because the project workers' intensive involvement with families sometimes had the effect of bringing additional problems to light.

The project appeared to have had less impact on enhancing parenting skills and promoting an environment that responds positively to the child, suggesting that more fundamental parental attitudes and behaviours were more difficult to change. Such a conclusion was endorsed by project staff and by other professionals who stressed the need to be realistic about what could be achieved with damaged families within a limited time frame. However, parents and professionals pointed to increased confidence and improved behaviour on the part of children who attended Matrix, and improvements were also observed by professionals in respect of parents' abilities to establish routines and provide boundaries for their children.

Most children who attended Matrix were thought by Matrix staff and by other professionals to have offended less since participating in the project and most of the children believed that their behaviour had improved. It was more difficult to assess whether the project had succeeded in keeping children in mainstream education or preventing them from being accommodated, though some professional respondents believed that it had in some cases been effective in this regard.

The costs of Matrix services

Matrix works intensively with children and families over extended periods of time (on average just over one year) and the costs per case are, not surprisingly, high, though no more so than those of other projects that work intensively with children and young people at risk of offending. The average direct cost per family was 14,395. The cost effectiveness of Matrix is more difficult to determine, since this would need to be established with regard to the costs of differential outcomes for Matrix families and similar families who receive 'normal' services. However, the costs per family attending Matrix need to be set against the possible cost savings that will accrue if some children are kept in mainstream education, if some children are able to remain at home rather than being accommodated by the local authority and if further offences and further victimisation are prevented.

Conclusions

Matrix was regarded as a valued resource by social workers, teachers and other agencies in contact with vulnerable children. The Matrix workers were viewed by other professionals and by families as enthusiastic, committed and professional. The project's emphasis upon early intervention was considered appropriate - both by project staff and by other professionals - in view of the absence of similar resources for this age group of children. Other professionals regarded the holistic approach adopted by Matrix to be a major strength, along with the ability of project staff to engage effectively with families who were reluctant to respond to statutory intervention as a result of extensive prior involvement with social work services. The project adopted a variety of methods in its work with children and families and emphasised the importance, at a number of levels, of inter-agency work.

The perceived benefits to families who attended Matrix included increased confidence on the part of children and their parents, improved behaviour by children and an increase in parenting skills, though more fundamental changes in parental attitudes and behaviour were more difficult to achieve. There was some evidence of a decrease in the incidence of risk factors and an increase in protective factors at the end of families' involvement with the project and some of the children who attended Matrix had either stopped offending or began offending less while they attended the project. However, it was not possible to establish within the timeframe of this study whether these short-term benefits could be sustained without longer-term support.

Implications

Because of the low number of families that Matrix had worked with, it is premature to conclude how effective it has been, though the findings are encouraging. The study also identified a number of questions that the Matrix Steering Group might wish to reflect upon in taking forward the work of the project. These include:

• Whether it would be desirable to harmonise referral procedures across local authorities?
• Whether the referral criteria and procedures should be revisited to prevent them from being unnecessarily complex?
• Whether the project should be working with younger children before a pattern of anti-social behaviour has become established?
• How the blurring of professional boundaries might be guarded against?
• How inter-agency work might be further enhanced?
• Whether a risk-focused approach encourages conservative decision-making with respect to case closure?
• How the work undertaken by Matrix with children and families might be supported in the longer term?

The approach adopted by Matrix is consistent with the growing evidence of 'what works' in reducing the risk of offending and anti-social behaviour among young people. The emphasis upon risk and protective factors that have been identified in the research literature and the ecological perspective and structured approach employed by the project share many features in common with Multi Systemic Therapy, which has been consistently shown to be associated with improved family functioning and reductions in anti-social behaviour among children and young people in North America (e.g. Edwards et al, 2001). The present findings are also encouraging in this respect and suggest that the Matrix approach is promising and worthy of replication in other sites. This would also allow for a more rigorous evaluation of the approach than has thus far proved possible with a small-scale project and relatively short timescales. It would enable a larger number of cases to be followed up over a longer period of time and relevant outcomes to be compared with a sample of similar children who receive the types of services that would be provided if Matrix or a similar project did not exist.

Reference

Edwards, D.L., Schoenwald, S.K., Henggeler, S.W. and Strother, K.B. (2001) "A Multilevel Perspective on the Implementation of Multisystemic Therapy (MST): Attempting Dissemination with Fidelity" in G.A. Bernfeld, D.P. Farrington and A.W. Leschied (eds.) Offender Rehabilitation in Practice: Implementing and Evaluating Effective Programs, Chichester: Wiley.