The Scottish Government

« Previous | Contents | Next »

Listen

1. Summary

This report is the result of a research project commissioned by the Scottish Executive to further its knowledge and understanding of the performance of Public Private Partnership ( PPP¹) projects in Scotland. The aims of the research were:

Stage 1: to provide an initial assessment of the relative costs and benefits of PPP procurement in comparison to conventional procurement, and of the underlying causes of good and bad performance;

Stage 2: to assess the potential merits of centrally collecting additional performance monitoring information and other data in order to inform future PPP performance evaluations, and to provide recommendations on what data should be collected.

Section 1.1 summarises the evidence from our survey of operational projects in Scotland organised around the key questions set out in the terms of reference. The section concludes with a list of the immediate recommendations that arise from the survey evidence. As the Scottish Executive recognised at the outset, the high-level nature of the study does not enable us to provide definitive answers to all of the questions posed in the brief. Furthermore, since PPPs are a comparatively recent development, and involve long-term contracts, operational experience of FM services is relatively limited and whole life aspects of the contracts cannot be fully assessed at this stage.

Section 1.2 then provides CEPA's overall assessment of these questions drawing on both the survey work and our previous experience of PPPs. It is important to recognise that our assessment is judgemental.

Section 1.3 summarises our findings on Stage 2 of the work. Our approach has been to consider in particular what further work (including but not limited to the collection of data) would be needed to build up a body of evidence about the relative costs and benefits of PPP compared with conventional procurement.

1.1. Stage 1 summary of evidence

Cost: To what extent does PPP procurement deliver overall cost savings in comparison to conventional procurement? How far can this be assessed with reference to factual data, rather than through comparisons with the assumptions used in Public Sector Comparators?

This question goes to the heart of the assessment of the relative costs and benefits of PPP. See CEPA's overall assessment in Section 1.2 below.

The Public Sector Comparator is clearly an important tool for assessing procurement options. Indeed, in all projects in our survey where a Public Sector Comparator ( PSC) was prepared (84 percent), the ex-ante analysis of the proposed PPP showed a saving versus the PSC.

However, to make a full assessment based on factual evidence, it would be necessary to compare in detail a number of PPP and conventional procurements (of infrastructure and the associated operations) in the same sector over a significant length of time. It has not been possible to do this within the scope of this research. But, as identified in our conclusions on Stage 2 of this work, we do believe that is possible to carry out further work that, at least partially, overcomes these constraints.

Other evidence from our study that relates to cost is as follows. First, more than half of the authorities surveyed believed that their contract offered good or excellent value for money, and only one project was rated as poor value for money. Second, our research identified a number of areas including procurement, performance monitoring, insurance and commissioning additional works, where costs are perceived to be higher for PPP projects. Third, our research also showed that contract managers generally perceive construction and operational risks to be appropriately allocated. Each of these points is discussed in more detail below.

Competition: Is there generally a healthy level of competition for PPP contracts, and hence is the lowest market price for the PPP service being obtained?

Projects included in the survey had received, on average, nine pre-qualification bids. The minimum number of bids received was two (and this only happened in one case). In our view this represents a generally healthy level of competition and was sufficient to achieve the most economically advantageous price.

Authorities and contractors commented that the level of competition has reduced in recent years and there is some anecdotal evidence that there is now less competition for PPP contracts than for other types of contract.

Procurement and construction performance: How far does the use of PPP impact on procurement costs and timescales? To what extent does the use of PPP deliver projects more quickly and on budget in comparison to conventional procurement?

The PPP procurement process is expensive and represents a greater burden for smaller projects. The mean time taken to procure the PPP projects surveyed was 28 months, which was generally perceived to be slower than non- PPP procurement. We note, though, that the increased costs and timescales associated with procurement should be weighed against their impact on overall timing and value for money. It is also worth noting that 'bundling' of smaller projects can help to spread procurement costs across several discrete projects.

There was greater price certainty during construction in the PPP projects surveyed than has been the case historically with conventional procurement. The proportion of projects delivered on time also showed a significant improvement over historical experience.

Nevertheless, on more than half of projects, the price changed at the preferred bidder stage because of a change to the design or service specification.

Design: To what extent are PPP designs perceived to be better (or inferior), e.g. in terms of aesthetics, functionality, environmental performance, etc? Is the relevant good practice guidance being followed?

Our survey found that authorities were happy with design quality. PPP projects were perceived to perform the same as, or slightly better than, non- PPP projects with respect to aesthetics, functionality and environment.

These perceptions reflect, in part, the fact that many authorities played an active role in the design process. More objective approaches to assessing design by sector (e.g. hospitals / schools / prisons) would need to be carried out to make a more informed judgement about actual design quality.

Formal design evaluation tools were not generally used in the projects surveyed, because these projects were amongst the first to be developed and such tools were not available.

Innovation: To what extent do PPP contractors deliver genuinely innovative solutions?

There was evidence from the survey and interviews that PPP stimulated innovation, although authorities were less positive about the role played by the PPP contractors than the contractors themselves. As with design, we believe that the authorities' views reflected their level of involvement in design decisions.

Operational performance: To what extent do PPP operators deliver a better (or worse) standard of service than the public sector, e.g. in terms of availability, cleaning, catering, etc? To what extent do service levels fall short of or exceed the original expectations of the contract?

We found no evidence that PPP operators delivered a better or worse standard of service than the public sector. Authorities commented that service standards were determined by the standards specified in the contract and the budget available.

Most authorities thought that the standard of service met or exceeded expectations. Availability ² performance was rated highly, which might reflect the strong financial incentive on contractors to meet availability targets. Hard facilities management services rated relatively poorly. However, we understand that this reflects some frustration on the part of authorities over the resolution of snagging issues and minor repairs, rather than concerns about lifecycle maintenance. It is too early to draw any conclusions about the impact of PPP on longer-term maintenance. But real issues were also raised about the interaction between hard and soft FM providers in PPP structures.

Flexibility: Are PPPs seen as being more or less flexible than standard contracts, e.g. in terms of making alterations to the asset, levels of service, etc? What has been the experience of attempting to make changes to contracts?

Our survey results showed that PPP contracts are seen as less flexible than non- PPP contracts. Authorities reported that their experience of making changes has been time consuming and slow. Where changes have been implemented across a sector they have been delivered later at PPP facilities.

A number of authorities commented in interviews that:

• The costs associated with making contract changes meant that they often sought to wrap up a number of changes over time into a single negotiation.
• Changes were rarely incorporated within the unitary charge mechanism because of the complication of agreeing the financial impact. But also, for an authority with grant available to spend on a specific change, it made sense to make a one-off payment rather than to incur financing costs.
• Works costs for contract changes were perceived to be expensive in the absence of the ability to tender them. Further work would be required to assess whether costs quoted by PPP partners reflected an allowance for maintenance and renewals.

Clearly it is important for authorities to get the scope of their projects correct when they are preparing Business Cases and ITNs. But it is inevitable that, over the life of a 30 year contract, changes will be required to support, amongst other things, new work practices and different user groups.

Contractual Relationships: Are relationships between customers and contractors perceived to be based on a "partnership" approach or an "adversarial" approach? What has been the use and impact of penalty payments/bonuses and/or the threat of termination of the contract?

Survey evidence showed that the majority of relationships between authorities and contractors are good. In interviews it was clear that both parties recognised the benefits of developing a long-term partnership.

Payment mechanism deductions are being implemented on the majority of projects. Authorities rated availability performance highly, and cited cleaning and minor repairs as the areas where performance was least satisfactory.

Budgetary Issues: How valid is the argument that the use of PPP delivers benefits due to budget restrictions in the public sector (particularly capital budgets)? For example, is it valid to argue that PPP procurement brings forward investment and/or ensures that optimal maintenance strategies are followed?

Many survey respondents commented that, at the time their project was procured, PPP was the only option given the authority's capital constraints. ( See Section 1.2 for commentary on this).

It is too early to compare maintenance strategies between PPP projects and contemporary non- PPP projects. However, our survey confirmed the observation that PPP does ensure that funds for maintenance of particular assets are effectively ring-fenced over the life of the contract, whereas non- PPP projects compete with all of an authority's non- PPP assets for a share of the maintenance / renewals budget. Indeed the overall maintenance budget itself can be subject to pressures from other priorities.

Risk transfer: To what extent is risk transferred in practice? Is it always clear where risk lies? Is there any evidence of contractors or customers seeking to shift risk onto the other party after signing the contract?

Survey respondents believed that risks were, for the most part, allocated appropriately.

In the case of operational risks, authorities noted that ambiguities in the contract drafting sometimes made it unclear where risks lie and presented opportunities for risks to shift. We would expect this to become less of an issue as the lessons learnt from early projects are incorporated into standard contracts. More issues around where risk lies were raised in sectors with significant interfaces between the PPP contractor and public sector staff, users and other contractors. This was most notable in the health sector where the interactions between hard and soft facilities management and the risks associated with the needs of different user groups were not always clearly defined in the contract.

Recommendations

The immediate recommendations arising from the survey, which are not covered in our Stage 2 work are as follows:

• Wherever possible Authorities should ensure that formal feedback systems, for example user surveys, are in place as part of the contract process.
• Further research should be carried out to review whether the contractual remedies relating to snagging provide appropriate incentives to contractors to resolve these issues.
• Authorities should continue to focus on developing the up-front specification to minimise any changes during the construction period, particularly in the light of concerns about value for money. However there may be a case for streamlining the change procedure for relatively minor changes.
• The Scottish Executive / Departments / Local Authorities could consider carrying out further research in the area of hard FM provision, with the aim of producing best practice guidelines for authorities and contractors and informing the terms of future contracts.
• Further research should be carried out to look at ways of enhancing flexibility without losing the benefits of PPP, focusing on those sectors where flexibility is a key area (e.g. health).

In this section we set out our assessment of the performance of PPPs in Scotland taking account of both the evidence gathered in this survey and our prior knowledge of PPPs.

Introduction

First, we believe that the Scottish Executive is right not to place undue emphasis on differences between the net present value ( NPV) cost of the public sector comparator ( PSC) and the NPV cost of payments to the PFI service provider. While a comparison of bid prices with the PSC is clearly one important element of the decision process, a full evaluation needs to be considerably wider. We would note, in particular, the importance of ensuring strong competition so that risk transfer is properly priced.

Second, we believe that a nuanced assessment of PPP needs to consider the different components of services separately. We note that much research (and therefore the available factual evidence) into the performance of PPPs has focused heavily on construction costs and construction cost risk transfer rather than the costs and benefits to the parties over the full life of the contract. This is certainly understandable given: (i) the significance of this issue in terms of overall costs, and the poor historical performance of conventional procurement; (ii) that most PFI contracts have only been operational for a small proportion of their 25-30 year contract lives; and (ii) the inherent difficulties associated with measuring service quality, life-cycle asset management performance, and the costs of loss of flexibility ³. But it does leave a clear gap in Government's knowledge.

In what follows we offer observations on: design; construction; operations (including both hard and soft facilities management); and life-cycle of the assets. It is important to recognise that the performance of PPP can, and we believe does, vary across sectors.

We also note that there are limitations to the methodology used in this broad study. For example, we received significantly fewer responses in the health sector, where there appear to be more post-completion tensions ⁴. It is also important to note that interview responses always have the potential to suffer from interview bias, especially when the respondents have a vested interest in the projects.

Third, we believe that it is important for research to seek to distinguish between the impact of PPPs that is (i) attributable to the specific nature of the procurement structure (e.g. the involvement of private finance and the attempt to create incentives for whole-life costing by transferring operational risks ⁵); and (ii) attributable to improvements in public sector procurement practice, regardless of the procurement structure used. Part of the problem with achieving this, as we and the Scottish Executive have observed from the outset of this work, is the ability to identify convincing counterfactuals to PPP. In particular:

• There are relatively few large-scale non- PPP projects in Scotland against which to compare PPP procurement, except perhaps in the schools sector. It is not ideal to compare new PPP assets with the vast majority of other public assets since it is very difficult to establish how far the benefits perceived by those using the buildings derive from the method of procurement or the fact that the assets are new.
• There are few examples of alternative procurement approaches such as 'Design, Build and Operate ( DBO)' contracts.

It should also be noted that due to the broad scope of this study we have not been able to investigate conventional procurement projects in the same depth as PPP projects. There would be some merit in considering additional, in-depth research in this area in the future.

Design and Construction

The evidence from this research and from earlier research by the National Audit Office ( NAO) demonstrates that PPP procurement has resulted in significant improvements over conventional procurement in delivering projects on time and within budget. PPP provides a strong incentive for contractors to complete assets to an agreed timetable in order to start receiving payments.

In particular, PPP has demonstrated to authorities the benefits of specifying requirements up-front and minimising any changes during the construction period. Authorities should be encouraged to import these lessons into conventional procurement. Similarly, PPP contracts have demonstrated the benefits of using fixed price contracts with clear risk allocation. These lessons could also be applied more widely in conventional procurement ⁶, for example through the use of fixed price turn-key Design and Build contract structures.

The need in PPP projects to specify requirements in detail and early on has also played a role in focusing authorities' attention on design issues. Although detailed design risk is transferred to the private sector, our interviews suggest that authorities in Scotland have remained heavily involved in developing PPP projects. They have looked to international best practice, sought input from users and, more recently, used specialist design tools to support them. Again, we think that these approaches could usefully be imported into conventional procurement methods (although see below for discussion of the impact of PPP on whole life costing).

Life-cycle asset management

CEPA's view is that the incentive that PPP provides to the private sector to design assets to minimise whole life costs, and to carry out necessary maintenance is a major area where this form of procurement could have advantages over conventional procurement. But, we are unable to judge this impact because most PPPs are not currently at the stage where it is possible to fully assess the impact of the approach to life-cycle costs.

There was however some evidence from our survey of whole-life costing at the design and construction stage, for example designs that optimised access to areas that would require maintenance and the use of longer lasting flooring materials.

Another interesting point to note arising from our discussions with authorities is that the ring-fencing of funding for PPP contracts means that, provided the contractor budgeted correctly for life-cycle costs in the original bid, such assets may be better maintained than traditionally procured assets. The argument is not related to the characteristics of PPP itself. Rather, that where money for maintenance (and particularly) renewals comes from general budgets it may be subject to other competing claims.

Operational performance

PPP contracts appear to be performing well in relation to availability of the basic assets (e.g. schools, hospitals, roads etc). Poor availability performance would lead to significant deductions in the payments received by the contractor which creates a strong set of incentives for the contractor to ensure availability. The level of incentives here is, in part at least, a function of the level of debt in the vehicle - which determines the importance of availability performance to the banks.

We have not identified any evidence to suggest that the soft facilities management ( FM) services provided under PPP are any better or worse than those provided by the public sector (either through short-term contracting out or by employing staff directly). Fundamentally the answer to the question depends on the level of service specified in the contracts and the amount that the authority pays for these services. As discussed in Section 1.3 of this report, this is an area where further, in-depth, analysis of a selection of comparator projects is likely to provide more robust evidence and conclusions on relative levels of performance.

We have not been able to draw conclusions on the value for money of the operational services provided under PPP. A key issue for further investigation is how facilities management services get re-priced against the market using benchmarks rather than market testing.

An important point in relation to soft FM in PPP contracts arising from this work relates to the interaction with hard FM and the allocation of risks. This is a particular issue if the authority itself is carrying out soft FM (for example, cleaning carpets), while the contractor carries out hard FM (for example, replacing carpets worn out through over-cleaning). These problems clearly occur in conventional procurement too, although they are rarely explicitly identified.

Where the contractor carries out both functions this risk allocation issue is overcome. However, our interviews have suggested that the authority still risks becoming embroiled in internal wrangles over responsibilities. This is particularly the case in situations in which the hard and soft FM sub-contractors are different:

• The majority of SPVs are construction-led, and there is a risk that, once the construction phase of the project is completed, and the profits relating to that phase extracted, the project will become less of a priority for the construction partner (although they will retain an interest in the SPV profits).
• In the operational phase, day to day contact is often with the soft FM sub-contractor to the SPV and there is some evidence that authorities are finding it difficult to hold the SPV and its construction sub-contractor to their ongoing responsibilities where they fall outside the remit of the on-site soft FM provider.

We believe that this finding suggests that further consideration needs to be given to the impact that contractor structure within the SPV will have on project performance - particularly as it moves to the operational phase.

Flexibility

A key concern with long-term PPP contracts is the level of flexibility that they offer to authorities to make changes either to the use of assets or to the level and type of services offered. As noted above in Section 1.1, our survey findings do suggest that this concern is valid - both in terms of the time and administrative burden of making contract changes and the costs associated with a single tender action with the existing contractor.

On balance the reduced flexibility implied by PPP contracts has probably been a benefit during the construction phase of projects, forcing better up-front specification and reducing cost overruns and delays.

During the operational phase of the contract, the contract inflexibility has a negative impact compared with conventional procurement. Although most contracts provide the option for the authority to use third-party contractors, the costs associated with managing this can be prohibitive. In addition, the introduction of third party providers of services or infrastructure can dilute the extent to which the authority transfers performance risk in practice, or can lead to problems associated with who takes responsibility for ongoing maintenance or renewal (as identified above in the situation where there are different soft and hard FM providers). The costs imposed on the authority by the inflexibility of the contract are likely to vary by sector. In sectors where a larger proportion of final service 'value added' is provided outside of the PPP contract (e.g. health and education) and where the contract interfaces are more complex (e.g. health) our expectation is that the costs will be greater.

The role of finance

This study has not sought to consider the role of finance, and evidence from the survey and interview work does not allow us to draw any conclusions on its importance. The standard arguments about the importance of including private financing as part of the procurement ⁷ relate to the role of the banks in adding discipline, both in terms of due diligence on bids and incentives for performance. We believe that this is an issue that merits further work by the Scottish Executive - for example to monitor the outcome of the pilot projects proposed by HM Treasury to assess the value of public funding with private sector credit guarantees ⁸.

Bringing forward investment

The widespread perception of the respondents to our survey is that PPP provided a mechanism to bring forward investment in new infrastructure at a time when large-scale investment in essential infrastructure was needed. How do we explain these observations, and are they consistent with Government's general approach which suggests that procurement choice should be based on value for money considerations?

Again the scope of the study has not included time to explore these issues in detail. However, on the basis of information provided to us by the Scottish Executive, our understanding is that these responses are an accurate representation from the perspective of public sector managers in local authorities, hospital boards and within the water and sewerage sectors at that time.

The clear implication is that the existence of PPP did allow local public managers to 'bring forward' investment compared to what would have been achieved within the constraints of public capital. We therefore believe that, when the contracts in our Survey were let, public sector managers may have been constrained in relation to the procurement choices available to them. This does not necessarily suggest that the PPP projects undertaken were poor value for money. Indeed, authorities were still required to demonstrate value for money compared to the alternatives. There are of course separate issues which we have not considered here around the robustness of public sector comparator analysis. Nevertheless our survey suggests that the majority of authorities considered PPP to represent good or excellent value for money.

There are two important points to note:

First, the perception of 'bringing forward' investment at the local level is entirely consistent with central control of the levels of investment taking place within Scotland (and indeed within the rest of the UK). Just as public capital is constrained centrally so was the extent to which local authorities and other public sector bodies could use PPP (e.g. by the amount of LPFS support available in the case of local authorities).

Second, these conclusions do not necessarily still hold. Indeed, we understand that the new 'prudential borrowing' regime within local government means that public sector managers will be able to take decisions on procurement options based primarily on value for money concerns.

Conclusions

The evidence supports the strong view that PPP transfers construction risk to the private sector more effectively than historical procurement methods and is likely to deliver value for money where there is strong competition and the projects are large. This is clearer in certain sectors, for example water, than in health. Our survey confirmed that the well known problem of transaction costs for smaller projects is also an issue in Scotland.

There is not yet enough evidence to conclude whether PPP transfers post-asset completion risks any more effectively than conventional procurement. This should be a focus for future research. The issue is whether the potential benefits of PPP, such as whole life costing, are sufficient to outweigh its disadvantages, such as lack of flexibility, and whether there might be variations in the contracting approach that would increase net public sector benefits without losing the benefits of transfer of construction cost risks.

1.3 Stage 2 summary

What are the merits of developing the Scottish Executive's central database of information on Scottish PPP projects in order to inform future long-term performance research?

The majority of research on PPPs to date has focused on the procurement and construction stages of the contracts. Whilst we believe that the Scottish Executive can and should put some effort into standardising the collection and management of this information, we do not regard it as the top priority. Our judgement is that the most valuable areas for further work (albeit the most challenging) relate to the operational phase of PPPs.

Procurement and construction

We recommend that the Scottish Executive's central database is extended to collect additional data on the procurement and construction phases of both PPP and non- PPP projects. This will enable real comparisons to be made about the relative benefits of PPP procurement in the areas of competition, timescales, delivery within budget, construction flexibility, risk transfer and perceptions of design quality and innovation. Details of the information that might be collected are set out in Section 7 of the report.

Operations

The Scottish Executive should consider holding some cost benchmarking data centrally to support contract managers involved in benchmarking or market testing exercises (e.g. on the input costs for soft FM).

In general though, we do not recommend that the procurement and construction database is extended to collect operational data at this stage.

Rather, we recommend that the Scottish Executive commissions sector specific studies to assess the relative operational performance of PPP projects. Depending on the level of resources that the Scottish Executive is able and willing to allocate to this, we suggest that this further work might include some / or all of the following three elements:

• A periodic audit of all PFI schemes (say every five years) along the lines of the current work - but also looking in detail at a smaller number of projects within a particular sector.
• Commission detailed studies into some of the specific components of PPP contracts identified in this Study.
• Commission sector specific studies that seek to compare performance across PPP and non- PPP contracts. We recommend that the Scottish Executive might consider carrying out a further, ongoing evaluation of PPP and non- PPP schools - building on the work carried out by Audit Scotland. If possible, this study should involve the identification of small number of standardised performance indicators and cost data that could be collected over time, and be used in a full comparative efficiency analysis of the sample projects.

« Previous | Contents | Next »