The Scottish Government

« Previous | Contents | Next »

Listen

5. Operational Performance

5.1. Standard of service

Most authorities were happy with the standard of service, but hard facilities management services rated relatively poorly.

Figure 13: Standard of service

Almost all authorities were happy with the availability of the asset. In most PPP contracts, availability is by far the most significant factor used in calculating the public sector's payment to the contractor, and this financial incentive appears to be working effectively to encourage good performance in relation to availability. Authorities were less happy with soft facilities management ( FM) service standards (cleaning, laundry, catering etc.) and even less happy with hard facilities management service standards (building repairs etc.). In general, interviewees noted that performance was best in those areas where the financial penalties could be made to bite.

Respondents noted that the quality of soft FM service is dictated by the terms of the contract and is not inherently better or worse because the operator is providing it under a PPP contract - although in some cases it is better because the impact of PPP has been to increase the budget. One interviewee pointed out that authorities may be very happy with the standard of service because their main point of comparison is the service at older, non- PPP facilities. It is extremely difficult for authorities to judge whether the service at PPP facilities is as good as it could be given the new assets and the service levels set out in the contract.

Unhappiness with soft FM provision generally related to lack of flexibility ( see section 5.5 below). The other source of unhappiness, raised in several interviews, was the quality of cleaning, and many authorities cited cleaning reports as the most important performance indicator. There were particular issues with cleaning in some parts of Scotland where contractors had under-estimated the buoyancy of the local employment market and had struggled to maintain staff. In some cases authorities had exercised their right to step in and carry out one-off cleaning services themselves, passing the costs on to the contractor. This is a good example of the contractual remedies working effectively to ensure the required service to the end users, but is frustrating for authorities. In some contracts there is the opportunity to bring cleaning in-house as part of a benchmarking exercise, but this leads to issues over the interface between hard FM and soft FM (for example, using appropriate cleaning products on the floor so that it sees out its expected life).

Issues around heating and cooling were also raised by a number of interviewees. Although contracts include requirements about allowable temperature ranges, it is difficult and time consuming for authorities to build up evidence of a breach. The temperature can differ within one availability area, and can change rapidly over time - in most cases the contractor has a period of several hours to remedy any service failure and so repeated monitoring is needed to demonstrate that the temperature has not been corrected.

Unhappiness with hard FM provision typically related to the fact that repairs often required input from the SPV's off-site staff and therefore took longer. In interviews several authorities commented on the fact that the SPV structure caused frustrations when the on-site FM contractor needed to refer issues back to the SPV for decision or for action by another consortium member. Repairs which the authority considered to be minor (and therefore expected to be resolved by the on-site soft FM team) were sometimes bundled together and classed as a major repair, taking longer to be carried out.

At this relatively early stage in projects' operational phases, most of the issues about repairs raised in interviews related to snagging issues left over from the construction phase rather than to life-cycle maintenance. (Project managers in the water sector seemed happier with the transfer of the interface from construction to operations.) Interviewees noted that it was too early in the operational period to form judgements on whether life cycle maintenance was being carried out effectively.

Recommendation: The Scottish Executive could consider carrying out further research in the area of hard FM provision, with the aim of producing best practice guidelines for authorities and contractors and informing the terms of future contracts.

5.2. Relationship between authority and contractor

The 2001 NAO report 'Managing the relationship to secure a successful partnership in PFI projects' concluded that PFI projects need to be approached in a spirit of partnership, that a successful partnership needs to be established at the outset and that having staff with the right skills is critical to good contract management.

The majority of relationships between authorities and contractors are good.

Figure 14: Current view of authority contractor relationship

85 percent of authorities and 92 percent of contractors said that their relationship was good or very good. There was very little change between the original and current views of the relationships between authorities and contractors, indicating that both parties have made strong efforts to adopt a spirit of partnership from the outset.

Our interviews revealed that while generally, as reported in the questionnaires, relationships between authorities and contractors are good, some authorities do find it frustrating that they need to spend a great deal of time debating interpretations of the contract requirements and maintaining the pressure on contractors to comply with the contract. This was particularly evident in the health sector, probably reflecting the complexity of the interfaces between the private sector, public sector and users in this sector, and the potentially severe consequences of, for example, a power failure or slippery floor.

It was notable in interviews that relationships appeared to be working better where the authority was responsible for a portfolio of projects and was able to use experience from other projects to make judgements about how to manage the contractor. Managing a portfolio of projects also enables more efficient monitoring, for example a dedicated monitoring team reporting on the cleanliness of a large group of schools, is more effective than relying on head teachers to monitor their own schools, alongside their other responsibilities.

Interviewees noted that the personalities of the individuals involved had a big effect on the ability to develop a successful partnership. They also commented that staff turnover on both the public and private sector sides of the contract was relatively high, although not out of line with wider experience in the sector. Authorities noted that the culture of the major contractor in the SPV had an impact on how the relationship was handled and that it was easier to have a happy relationship where the contractor was profitable.

The contractual disputes resolution procedure had been used, and all disputes reported had been resolved without escalation to the courts.

The disputes resolution procedure had been initiated one or more times on 8 projects and an external adjudicator used on 4 projects. Most disputes related to practical interpretation of contractual provisions. Interviewees commented that it was difficult to find suitably qualified adjudicators, given that PPP contracts are relatively new.

5.3. Performance monitoring

PPP contracts should set out how performance will be monitored, based on output measures.

Performance monitoring mechanisms are satisfactory but improvements could be made to performance indicators.

61 percent of authorities thought that performance monitoring mechanisms were working well or better, 36 percent thought that they were satisfactory, and only 3 percent thought that they were poor (see Figure 15 below). These results are very similar to the NAO's findings in its 2001 report 'Managing the Relationship'.

Figure 15: How are project and performance monitoring mechanisms working?

Only 45 percent of authorities thought that the performance indicators were wholly appropriate for measuring contractor performance. 52 percent thought that they were partially appropriate and 3 percent found that they were not at all appropriate. Despite this lack of satisfaction with the performance indicators, there were only a few projects where new indicators had been negotiated. This could be due to the lack of contract flexibility or, equally, to the lack of better alternatives.

Based on our interviews, it appeared that performance indicators ( KPIs) worked best in sectors such as prisons, transport and water where there were clear KPIs for the whole sector, not just PPP projects. Arguably, these sectors also have less complex interfaces between the contractor, public sector staff and systems and users. Interviewees in other sectors commented that interpretation of the drafting of performance indicators was often open to discussion. This issue of ambiguity was exacerbated by changes of staff. In some cases interviewees felt that too many indicators were monitored, not all of which were useful.

Some authorities reported that it was difficult to establish the right balance between monitoring and auditing. Several stated that they had not fully understood their monitoring requirements at the outset of the contract. They were concerned that, given the impact of the payment regime, leaving the contractor to self-monitor would discourage him from being proactive in identifying and rectifying faults.

Several authorities commented that they were surprised at the input required on their part to ensure that the contractor delivered the contracted level of service. PPP contracts are typically self-monitoring, with the authority responsible for auditing the contractor's records of service failures. One interviewee commented that, while mistakes in the self-monitoring invariably had to be put down to human error, the mistakes were skewed in the contractor's favour.

Authorities commented that the ultimate threat of default did work to encourage good performance.

Recommendation: There is a strong case for establishing, for each sector, a relatively short list of performance indicators, enabling comparison between PPP and non- PPP projects. A framework of benchmarking actual performance against periodically restated KPIs and enhanced authority rights to re-tender the contract in the event of persistent shortfalls against KPIs could considerably improve outcomes in the longer term.

5.4. Contractual remedies

Payment mechanism deductions are being implemented on the majority of projects, with some minor relaxations to promote good working relationships.

Out of 32 projects where information was provided in response to questions about the use of contractual remedies, 88 percent had made deductions from the payment mechanism. Figure 16 shows the contractual remedies used.

Figure 16: Use of contractual remedies

41 percent of authorities said that there had been occasions when the contract allowed them to enforce penalties but they had chosen not to. In most cases authorities said that they had waived penalties when the amounts were small and the consequences were not significant, and they felt that it would promote a good working relationship with the contractor. This approach was taken particularly at the start of the operational period, and on the basis that the contractor accepted that there had been failures and had put in place action plans to resolve matters.

PPP contracts typically contain a mechanism for agreeing changes during the operating period. If the authority wishes to implement a change, it starts by documenting its request. The contract sets out the circumstances in which the contractor is obliged to meet the request, and the process by which he agrees or disagrees to proceed. Assuming that the contractor has agreed to proceed, he presents cost estimates, and there are then further procedures by which the authority accepts or challenges these estimates until agreement is reached.

PPP contracts are less flexible than non- PPP contracts. Where changes have been implemented across a sector they have been delivered later at PPP facilities.

All contracts in the survey contained provisions to change the service specification, and these provisions had been used in 50 percent of contracts. Informal changes had also been agreed in 28 percent of contracts, for example changes to opening times or the use of rooms. Interviewees noted that, where the relationship is good, informal changes can be made between the operational staff and periodically swept up into formal variations. All changes were jointly initiated, or initiated by the authority.

34 percent of authorities considered service levels under a PPP contract to be less flexible than alternative contractual arrangements, for example short-term contracting out, or where the authority itself employs staff to carry out the services. 49 percent thought that the form of contracting made no difference, and only 6 percent considered PPP contracts to be more flexible. By contrast, 42 percent of private sector respondents thought that PPP contracts were more flexible (see Figure 17).

Figure 17: How does the flexibility of service delivery compare with non- PPP arrangements?

Although contracts set time limits for each step of the change procedure, the amount of 'to-ing and fro-ing' between authority and contractor has meant that changes can be very slow to agree. Several authorities stated that it had taken one year or longer to agree what they saw as relatively straightforward changes. The lack of flexibility in the PPP sector was particularly apparent where changes were being implemented across a sector (for example Patient Line in hospitals, or fresh water drinking fountains in schools) and were introduced later at PPP facilities than those operated by the public sector. Concerns about value for money were a significant factor in the time taken to agree changes; authorities often challenged contractors' initial cost estimates and requested additional information. Contractors' perceptions that PPP contracts are more flexible than non- PPP arrangements might result from the fact that there is less need in short-term service contracts than in long-term PPP contracts to include complex change mechanisms.

In interviews both authorities and contractors mentioned that in order to simplify matters, and avoid making changes to the project financial model, they often waited until they could agree a set of changes whose net financial impact would be zero. Refinancing was also seen as an opportunity to push through changes.

Recommendation: Further research should be carried out to look at ways of enhancing flexibility without losing the benefits of PPP, focusing on those sectors where flexibility is a key area (e.g. health).

5.6 Allocation of operational risk

Operational risks are, for the most part, appropriately allocated.

96 percent of contractors and 85 percent of authorities thought that risks relating to the ongoing operation of the asset currently were allocated to the party best able to manage them. Some of the risk allocations in the earliest projects had been amended in later projects, for example vandalism risk was retained by the authority (at least when the asset was in use), and utilities risk was transferred to the contractor.

Interviewees commented that appropriate allocation of risk was most difficult to achieve where there was significant interaction between contractor, authority and users. Materials management in hospitals was cited as an example of a risk that is sometimes transferred to the contractor but can be difficult for him to manage because of the need to interact with NHSIT systems. Allocation of risks relating to cleanliness can be difficult if more than one contractor is operating on the same site. This is a particular issue in Scotland where soft FM is sometimes carried out in house, or may be taken in house or let to a different contractor at fixed points in the contract. The interface between different contractors can then lead to difficulties in determining responsibilities if things go wrong.

Interviewees also commented that cost risks arising from unanticipated behaviour of different users of the asset, for example greater wear and tear caused by more severely ill patients in mental health facilities, were not always clearly allocated in the contract.

The example of Mearns Primary School demonstrates how the transfer of major risks has worked in practice. When the roof blew off one wing of the school, the contractor forfeited a significant amount of its availability payments and was responsible for the entire cost of repairs. Although the council was fully protected financially, it was the council's own staff that responded initially to the emergency, since the contractor's staff were not based locally.

« Previous | Contents | Next »