The Scottish Government

« Previous | Contents | Next »

Listen

7. Future data collection

Stage 2 of this study requires an assessment of the potential merits of centrally collecting additional performance monitoring information and other data in order to inform future PPP performance evaluations and to provide recommendations on what data should be collected.

In assessing the options available to the Scottish Executive we have considered the following aspects of the issue:

• The extent to which it is possible and valuable to collect information at a central level for the different elements of the PPP structure. Our recommendation is that the Scottish Executive should have different approaches to the construction and operational phases of projects.
• The 'regulatory burden' that the provision of information will have on both authorities and contractors. As far as possible the Scottish Executive should use existing information and definitions consistent with those used in NAO surveys and other data collection exercises, such as that being carried out by PUK.
• Whether the information should be collected as part of regular returns submitted by authorities or contractors, or as one-off returns (e.g. at particular phases in the project).
• The availability of non- PPP comparators. One of the key constraints in drawing conclusions about PPPs in this and other studies is the lack of comparable data for non- PPP procurements.
• The quality of key performance data (e.g. KPIs) and cost information. Our judgement is that, for most operational PPPs (with the possible exceptions of prisons, roads and water) there is significant work required to establish a common set of performance indicators. This work needs to be completed before it makes sense to consider how and when to collect such data.

In this section, we provide our assessment and recommendations on these issues. Section 7.1 sets out our proposals in relation to procurement and construction data. Section 7.2 considers the more difficult area of post completion data.

As noted earlier in this document, the majority of research on PPPs to date has focused on the procurement and construction stages of the contracts. Whilst we believe that the Scottish Executive can and should put some effort into standardising the collection and management of this information, we do not regard it as the top priority. Our judgement is that the most valuable areas for further work are in the operational phase.

7.1. Procurement and construction data

We recommend that data from the procurement and construction phases of projects is collected centrally. Standard questionnaires, applicable to projects from any sector should be completed by procuring authorities at two fixed points in the process: (i) at financial close; and (ii) at the start of operations. At both of these points data should be readily available - and the authority bid team should be in a position to provide it at relatively low cost ¹⁷. The majority of the data collected would be factual, although some more subjective questions, for example about the quality of design or the relationship with the private sector, could also usefully be included.

A potentially important area of added value going forward in seeking to answer the Scottish Executive's key question would be to collect the same data for non- PPP procurements. Many of the data fields would be applicable to design and build contracts. We therefore recommend the same questionnaire should be used to collect data from non- PPP projects (over a fixed capital value), enabling comparisons to be drawn.

The data fields used should be consistent with the data collected in stage 1 of this study (which was designed to be consistent with previous NAO studies), and with Partnership UK's database. Figure 19 and Figure 20 below show the main categories of data that should be collected and the research areas that the data could be used to inform.

Figure 19: Procurement phase data to be collected at financial close

Figure 20: Construction phase data to be collected at operations start

Most of this data is now available for the projects that responded to the survey. However, it is likely to be a reasonable amount of work to collect data for all projects, both those that did not reply to our survey and those that were not operational at the time of the survey. After that, we anticipate that it should be relatively easy to track and collect the data for new projects, although a single person within Scottish Executive would need to take responsibility for ensuring that returns are completed to the required to standard.

Our suggestion would be that the data is collected using Excel-based questionnaires and is held on a database.

It is of course important to recognise that whilst the development of this database would highlight differences between PPP and non- PPP projects, or changes in the performance of PPP / non- PPP projects over time, it would not give the underlying reasons for those differences. It is likely that this data would highlight areas where further, more in depth analysis would be required.

7.2. Operational phase data

For the operational phase of projects, there may be merit in the Scottish Executive seeking to collect information on unit costs of benchmarked services ¹⁸ e.g. soft FM. This might be of some use to public sector managers within and potentially across sectors in managing their contracts. (An alternative or complementary initiative would be to promote and support networks of contract managers. This is particularly important in sectors such as health, where most contract managers are responsible for a single PPP contract.)

More generally, we do not believe that Scottish Executive should not prioritise the central collection of this data at this stage. This reflects two observations:

First, we do not believe that there is currently a standard set of data that could be collected centrally and used to evaluate systematically the performance of Scottish PPP projects. In addition few sectors have yet developed standard performance indicators for all PPP and non- PPP projects.

Second, we believe that further work would be needed to understand the project specific factors (within each sector) that would lead to explainable variations in cost and efficiency ¹⁹.

Instead, our judgement is that some or all of the following approaches might be pursued:

• Institute a periodic audit of all PFI schemes along the lines of the current work - but also looking in detail at a smaller number of projects within a particular sector.
• Commission detailed studies into some of the specific components of PPP contracts identified in this Study (e.g. on the levels of FM service specified in the contracts and the relative costs of delivery in different procurement models).
• Commission sector specific studies that (i) seek to define a small number of standardised performance indicators and cost data that could be collected for PPP and non- PPP projects; and (ii) where appropriate carry out more detailed benchmarking studies of performance

Each of these ideas is discussed in more detail below.

7.2.1. Periodically repeat the analysis carried out in this study

The analysis of the operational phase of projects carried out in this study was a 'thin' survey across all sectors, which aimed to assess views on performance, flexibility, relationships etc. This type of study needs to be more than a straightforward data collection exercise; the most useful insights in our research came from the free text questionnaire responses and interviews. We also believe that it makes sense to include a component of work that focuses in detail on a small number of PPP and non- PPP projects within the same sector.

We recommend that the Scottish Executive consider repeating this type of relatively broad study on a periodic basis - say every 5 years. The results of Stage 1 of this research would provide a baseline for such analysis.

7.2.2. Detailed studies into specific components of PPPs

Overall, the priority for these studies should be to report on performance post-construction and the extent to which operational and life-cycle risks are transferred in practice. Within this, specific areas of interest might include:

• Differences in the level and costs of service delivery of soft and hard FM, including a more detailed assessment of the relative merits of transferring both soft and hard FM within the contract. Related to this, it would be useful to explore further the way that the SPV structure impacts on performance in specific cases.
• A more detailed assessment of the costs associated with the comparative lack of flexibility in PPP contracts compared to alternative procurement structures. This analysis might look in more detail at specific examples of contract changes within the operational phase.

Sector comparisons (across procurement approaches) on the life-cycle and condition of assets of similar age.

7.2.3. Commission sector specific studies

We believe that the most promising approach for the Scottish Executive to improve the available evidence on the operational performance of PPPs and non- PPPs is to carry out sector specific studies. There are various approaches to carrying out such comparative studies.

The simplest approach would involve carrying out an interview or survey-based study along the lines of this research. Such studies would need to be carried out periodically to track performance over time. One option that the Scottish Executive might consider is to build on the Audit Scotland report on schools ²⁰ and identify a number of PPP and non- PPP schools projects to evaluate over time. Other sectors would of course be equally interesting, however the school sectors has the particular advantage of being a relatively mixed economy in terms of privately and publicly financed infrastructure.

The weakness with the survey-based approach relates to the potential for interview bias. A more robust approach is likely therefore to involve carrying out full comparative efficiency or 'benchmarking' studies. This analysis would aim to look at the efficiency of the overall service offered by a contractor by considering, for example the relative costs of delivering the 'final' outcomes of the contract (e.g. the availability of the asset to an agreed standard). This is the approach that utility regulators take when seeking to compare the economic efficiency of regulated companies (e.g in the water and power sectors).

A starting point for such studies would be the definition of a standard set of KPIs for PPP and non- PPP projects within the same sector. Although this is a non-trivial task - since (as noted above) there is currently a relative lack of comparable performance indicators - we believe that it could be a valuable exercise for a number of sectors.

« Previous | Contents | Next »