Construction Economics and Building
Vol. 21, No. 4
December 2021
RESEARCH ARTICLE
Evaluation of Transportation Infrastructure: A Case Study of Gold Coast Light Rail Stage 1&2
Craig Langston1,*, Charles Crowley1
1 Faculty of Society and Design, Bond University, Gold Coast, Australia
Corresponding author: Craig Langston, Faculty of Society and Design, Bond University, Gold Coast, Australia, clangsto@bond.edu.au
DOI: http://dx.doi.org/10.5130/AJCEB.v21i4.7738
Article History: Received: 26/05/2021; Revised: 21/08/2021; Accepted: 14/09/2021; Published: 20/12/2021
Abstract
Since the COVID-19 pandemic began, there has been increased reliance on new infrastructure projects to counter economic fallout and underpin employment security. Urban and inter-urban transportation projects, such as major road, rail and port facilities, are popular choices for national and state governments in Australia as they provide broad fiscal support across all sectors of the economy. The problem with stimulus is making sure that the quality of the new infrastructure provides collective utility to a community or region. Whether the benefits will be worthwhile and represent best use of resource inputs requires financial, social, ethical and environmental consequences to be evaluated in a comparable format. The aim in this paper is to analyse the Gold Coast Light Rail (GCLR) Stage 1&2 project using a method that is capable of merging tangible and intangible criteria using an ordinal ranking algorithm. While the GCLR case study is undertaken with the benefit of hindsight, normally these types of evaluations are performed in real time as a project progresses from initiation (design) to implementation (deliver) and influence (delight). The method adopted in this study represents a modern form of multi-criteria decision-making, which enables successful projects to be distinguished from unsuccessful ones using a time period from commencement until one full year of operation has occurred. The i3d3 model, developed by a team from Bond University, has the unique benefit of ranking projects from best to worst across an organisational portfolio, geographic region or industry sector. It also supports past project performance to inform new design through application of a continuous improvement process of recording lessons learned. The GCLR case study calculated 100% of the critical success factors in the model to be positive and produced an overall success ranking of 23 (on a scale of -100 to +100). This paper presents the approach taken to evaluate GCLR’s level of success and the calculations that took place to reach this finding. This is the first time i3d3 has been used on an Australian project.
Keywords
Transport Infrastructure; Evaluation Techniques; Multi-Criteria Decision-Making; Case Study; Australia
Introduction
People have been making judgements about the merits of new infrastructure projects for centuries. The discounted cash flow technique that underpins cost-benefit analysis became popular after the stock market crash of 1929 but can be traced back to ancient times. Determining ‘accept–reject’ decisions based on surplus or deficit over time is common sense, but in more recent history other criteria have emerged that need careful consideration.
The need for fiscal stimulus post COVID-19 has led to an increase in construction of large public infrastructure megaprojects. Many of them can be considered as failures. According to Flyvbjerg (2017), over the last 20-30 years, 70-90% of megaprojects experienced cost overruns. These problems may be summarised as “over budget, over time, under benefits, over and over again” (Flyvbjerg, 2017, p.12).
The ‘G20 Principles for Quality Infrastructure Investment’ (G20, 2019) is a recent initiative to create a standard for defining successful infrastructure procurement. The document describes infrastructure as “a driver of economic prosperity that provides a solid basis for strong, sustainable, balanced and inclusive growth and sustainable development” (p.1). But while the objectives may be agreed, the process to compute success remains opaque. The difficulty of combining financial consequences (largely tangible) with social, ethical and environmental consequences (largely intangible) is a main barrier to widespread adoption. It requires a multiple-criteria approach rather than attempting to measure benefits solely in monetary terms. Anecdotal evidence suggests that traditional evaluation methods have become so complex that not only is technical expertise essential, but the process is too time consuming, expensive and open to manipulation. A better approach is needed to ensure that project decisions are balanced, and performance expectations are ultimately achieved.
The aim in this paper is to analyse the Gold Coast Light Rail (GCLR) Stage 1&2 project using a method that is capable of merging tangible and intangible criteria using an ordinal ranking algorithm. This process needs to be able to both predict and validate actual performance in terms of project success while ensuring that envisaged benefits during design are realised in operation.
Literature Review
Evaluation methodologies for transport infrastructure
Commonwealth and State governments in Australia are engaged in unprecedented fiscal stimulus to encourage rapid economic recovery from the COVID-19 pandemic (Infrastructure Australia, 2021). Transportation projects are popular forms of infrastructure that can improve living standards, industry productivity, tourism and safety. They can create a legacy through the provision of vital services while putting upward pressure on employment and creating increased consumer confidence and spending. The problem with stimulus is making sure that the quality of the new infrastructure provides collective utility to a community or region. Understanding the benefits requires financial, social, ethical and environmental consequences to be evaluated in a comparable format.
The evaluation of transport infrastructure projects is complex due to their spatial scale, the scope of their economic, environmental, and social impact, and often their duration – in terms of the design and construction phase as well as their operational lifecycle. This is further complicated by the need to account for a variety of assumptions such as residual value, discounted utility, forecasted economic and demographic growth, and rate of technological progress (Helm and Mayer, 2016; Chi and Bunker, 2020). An extensive and rigorous evaluation process is therefore required to ensure that scarce public resources are invested in an optimal project. Efficiency in the project evaluation phase is critical, as the selection of the ‘right’ project will ultimately enable the realisation of greater impact (higher return on investment) at a specific investment amount (Kyriacou, Muinelo-Gallo and Roca-Segales, 2019). The selection of a meticulous and comprehensive project evaluation methodology is key to achieving this, as it will allow society to effectively determine the opportunity cost of prospective proposals. This section will summarise the predominant evaluation methodologies utilised in transport infrastructure project assessments.
Cost-benefit analysis
The traditional (and most common) method to appraise prospective transport infrastructure projects is by conducting a cost-benefit analysis (CBA) process (Mouter, 2018; Chi and Bunker, 2020). This process is intellectually rooted in the Marshallian theories of welfare economics and collective utility, and thus is primarily concerned with enhancing consumer surplus (Vickerman, 2017). Essentially, a CBA assigns a financial value to the forecast result of an infrastructure project proposal, then seeks to ascertain whether the total amount society is ready to outlay for the proposal surpasses its anticipated costs (Vickerman, 2008; Ronnle, 2017). In practice this is done by offsetting the overall societal benefits of the proposed project over its lifecycle against its overall societal costs, and then discounting them to the present-day at a set discount rate (Helm and Mayer, 2016). For the purposes of infrastructure evaluation in Australia, the primary discount rate is set at 7% by Infrastructure Australia - the national statutory body mandated with assessing and prioritising major infrastructure investments to ensure consistency of evaluation across the country. However, this analysis is accompanied by outlier discount rates for sensitivity analysis assessment of 4% and 10% respectively (Infrastructure Australia, 2018).
From an economic perspective CBAs are typically administered within the confines of a partial equilibrium analysis, in which (in the case of a transport infrastructure project) the transport sector itself is evaluated partially, whereas prices in all other secondary sectors are held fixed (Hansen and Johansen, 2017). In keeping with the assumption of perfect markets in equilibrium, all applicable benefits will be encapsulated within the final estimated transport consumer benefit in the CBA (Hansen and Johansen, 2017). Once finalised, the CBA will deliver a benefit-cost ratio (BCR), which is presented as the economic case for project approval (Atkins, Davies and Bishop, 2017). In theory if the BCR is greater than 1 then this should result in the project being greenlit, while if it is less than 1 then the project should be rejected, since its total social costs outweigh its total social benefits (Helm and Mayer, 2016; Ronnle, 2017). Owing to the scarcity of social resources, however, the most common outcome is that if a project’s BCR is favourable, then it is further compared and evaluated against a portfolio of other prospective (favourable) projects, with public funds being allocated to whichever proposal is estimated to deliver the highest return on investment – greatest BCR (Vickerman, 2008; Helm and Mayer, 2016).
Although often criticised for its narrow economic focus and rigidity of process, there are benefits to utilising a CBA as the chosen methodological tool when evaluating transport infrastructure projects. Paramount amongst these is the method’s sound economic analysis, which compiles a systematic itemisation of all direct benefits and costs (Asplund and Eliasson, 2016; Mouter, 2018). At the governmental level, this provides decision-makers with a clear frame of reference of the benefits and opportunity cost of a project when examining submissions from a broader portfolio of project proposals. This in turn greatly simplifies the overall planning process (Mouter, 2018). By excluding the indirect effects of transport infrastructure investment, CBAs also safeguard the evaluation process against the duplication of economic benefits which can occur in other methodologies that embrace the dynamic (indirect) wider economic benefits ethos of evaluation (Chi and Bunker, 2020). In doing so, this will prevent the aggrandised claims of substantial societal and economic benefits, which often accompany these alternate project evaluation methodology assessments (Chi and Bunker, 2020).
The criticisms of CBA as a tool for project evaluation primarily relate to its inability to capture the dynamic economic effects of transport infrastructure investment, due to its utilisation of Marshallian welfare economics concepts as a theoretical framework (Hansen and Johansen, 2017; Vickerman, 2017). Critics argue that partial equilibrium analysis and the resultant assumption of perfect markets is fundamentally wrong, since secondary markets are often misrepresented, thus causing a situation where the attested consumer benefits no longer equate to the aggregate net benefits of the project (Helm and Mayer, 2016; Hansen and Johansen, 2017). As Coyle (2016) notes, CBA fixates on the incremental direct impacts to the consumer arising from the transport infrastructure investment, while neglecting the wider behavioural adjustments that occur to the labour force and firms as a result of increases in accessibility and subsequent agglomeration of industry. This position is supported by Lakshmanan (2011), who argues that CBAs are typically limited in their outlook owing to their focus on direct productivity output increases to individual consumers and firms. In fact, there is no established principle or acknowledged standard for treating non-monetised factors (intangible impacts) within a CBA (Spackman, 2013). Furthermore, an additional criticism of CBAs is that they do not capture the land-use benefits, such as an increase in the capital values of property, which have been empirically proven to increase in the immediate spatial vicinity of public transport infrastructure projects that increase accessibility (Levkovich et al., 2016; Vickerman, 2017; Yen et al., 2018; Yen, Mulley and Zhang, 2020).
Failing to consider the dynamic indirect effects of transport infrastructure investment can lead to a potential underestimation of aggregate project benefits, which can cause a sub-optimal project selection, and as a result, an inefficient investment of public capital (Hansen and Johansen, 2017). Finally, notwithstanding the contingencies provided by sensitivity analysis, given the significant duration of most project lifecycles, the three key component factors of the analysis (social benefits, social costs, and the social discount rate) are all susceptible to considerable uncertainties over the project lifecycle (Helm and Mayer, 2016).
Wider economic benefits and wider economic impacts
An alternate evaluation methodology framework to CBA specifically for transport infrastructure investment is that of the wider economic benefits (WEB) and wider economic impacts (WEI). The methodology itself is not mutually exclusive to CBAs, but rather seeks to capture all the dynamic economic effects (after accounting for all quantifiable environmental, social, and economic costs) that are not encapsulated in the direct consumer surplus benefits of a conventional transport CBA (Vickerman, 2008; Rothengatter, 2017; Infrastructure Australia, 2018). This is also known as social CBA. As opposed to traditional CBAs, the WEB/WEI philosophical economic basis is in Krugman’s theory of the New Economic Geography (Brathen and Givoni, 2017). Accordingly, an emphasis is placed on the economic effects that result from increases in accessibility and agglomeration. Despite the concerted push to incorporate the concept into project evaluations in recent times, there is still no consensus approach amongst policymakers and transport economists regarding how to integrate the practice within policy guidelines for infrastructure project assessments (Wangsness, Rodseth and Hansen, 2017).
Infrastructure Australia, for example, is yet to develop its own methodology for the process, suggesting that proposed submissions instead seek guidance from the ‘WebTAG’ methodology devised by the UK Department for Transport (Infrastructure Australia, 2018). Brathen and Givoni (2017) contend that as the economic basis behind the theory is quite rational and established, a renewed emphasis should be placed on agreeing upon a uniform methodology for its practical application and empirically testing it. Proposals for measuring WEB/WEI that have been developed so far tend to focus on the main impact factors outlined in the New Economic Geography (NEG), comprising agglomeration effects, increases in the available labour market, and growth in aggregate output in imperfect market conditions (Legaspi et al., 2015; Rothengatter, 2017).
One of the most common methods utilised to assess the WEB/WEI of transport infrastructure projects is that of Spatial Computable General Equilibrium Modelling (SCGEM) or Computable General Equilibrium Modelling (CGEM) techniques (Hansen and Johansen, 2017; Rothengatter, 2017; Vickerman, 2017). SCGEM/CGEMs differ slightly from the core idea of the NEG, by attempting to incorporate the key concepts of spatial economics such as accessibility and agglomeration within the traditional welfare economics general equilibrium structure (Rothengatter, 2017). While these models allow for econometric modelling of a project’s WEB/WEI, the primary criticism levelled at them is that the methodology is often inconsistent, and they are vulnerable to manipulation from ill-defined inputs (Ronnle, 2017; Vickerman, 2017). This can result in a lack of transparency in the assessment process, increasing the likelihood for political tinkering and confirmation bias (Ronnle, 2017). This is particularly concerning for those transport infrastructure projects that would fall within the category of ‘megaprojects’, which already struggle with a culture of optimism bias in the project evaluation phase (Flyvbjerg, 2013; Odeck, 2014).
An example of this can be seen in the UK, where SCGEM modelling on the benefits of runway expansion revealed seismically different results due to the utilisation of different data inputs. The assessment complied by the Department for Transport suggested overall net benefits of £61 billion over the forecasted time period, while an independent assessment commissioned by the Airports Commission and performed by Price Waterhouse Coopers determined the economic lifecycle benefits to lie between £131 and £147 billion (Vickerman, 2017). Examples such as this underline the assertation of Rothengatter (2017) that WEB/WEI methodologies for transport infrastructure investment are often at the mercy of prevailing political trends, and therefore should be viewed as neither comparable to, nor a substitute for, CBA analysis. To this end, Germany’s federal government rejects the usage of WEB/WEI methodology for transport infrastructure investment, while the UK Department for Transport, and Infrastructure Australia, both recommend conducting a WEB assessment in addition to a conventional CBA for transport infrastructure investment analysis (BMVI, 2014; Infrastructure Australia, 2018). It should be noted that not all project assessments show such wide variances when incorporating the WEB method. Infrastructure Australia’s evaluation of the Inland Rail Project revealed an identical BCR of 1.1 (utilising both CBA and WEB methodologies) when conducting an assessment at the standard discount rate of 7%, and only a marginal variance of 2.8% to 2.9% (WEB), when assessing at the lower sensitivity analysis rate of 4% (Infrastructure Australia, 2016). In line with the theory of the NEG, in this case the WEBs were expected to stem from an agglomeration of industry around multimodal freight hubs which were planned to be built at strategic junctures along the rail line, thereby driving a reduction in overall transport costs, which in turn will spur growth in aggregate output (Infrastructure Australia, 2016).
Endogenous macroeconomic modelling
An alternate methodological framework to SCGEM/CGEM for assessing the WEB/WEI of a transport infrastructure project is that of endogenous macroeconomic modelling. This approach aligns with the theoretical basis for economic growth, outlined in Romer’s endogenous growth theory (Rothengatter, 2017). The typical data inputs for these models are national economic figures such as GDP, and aggregate consumption (Rothengatter, 2017). Unlike SCGEM/CGEM models, exogenous growth methodologies do not assume the neo-classical general equilibrium theory but embrace the Keynesian concepts whereby employment levels can be fluid, and full utilisation of resources may not be achieved (Rothengatter, 2017; Cambridge Econometrics, 2021). An example of an endogenous growth model which can be applied for infrastructure assessment, is the E3ME model developed by Cambridge Econometrics which has been applied on transport infrastructure projects in the European Union (Köhler, Jin and Barker, 2008; Rothengatter, 2017).
Multi-criteria assessment
The concluding methodological approach to transport infrastructure assessment is that of multi-criteria decision-making (MCDM). It is also known as multi-criteria decision analysis (MDCA). Like WEB/WEI, this project evaluation approach is best characterised as a theoretical framework, with there being a number of different methods that can be applied (Guitouni and Martel, 1998)1. The common thread between all methods is that they seek to provide clarity, structure and guidance to the assessment process in order to help facilitate a judicious final project decision (Kabir, Sadiq and Tesfamariam, 2014; Van der Meer et al., 2020). MCDM assists decision-makers within government by filtering and arranging project information in such a way as to enable the identification of the vital evaluation criteria for project selection, which in turn allows for clear comparison between competing projects (Van der Meer et al., 2020). In practice, the framework requires the scoring of a prospective project against multiple criteria irrespective of the explicit MCDM method chosen (Broniewicz and Ogrodnik, 2020). The scoring itself is weighted according to the preference of key project stakeholders against pre-determined parameters, such as financial cost and environmental impact (Van der Meer et al., 2020). This weighted scoring of criteria permits a final figure to be assigned to the project, with the highest scoring project at the end of the process theoretically being greenlit (Van der Meer et al., 2020). Although seemingly straightforward, much like a CBA, the utilisation of these methods necessitates the application of detailed quantitative datasets to be effective (Broniewicz and Ogrodnik, 2020).
Kabir, Sadiq and Tesfamariam (2014) advocate the usage of MCDM methods on infrastructure project assessments to enable quality project selection. They suggest that MCDM methods achieve this by providing transparency, rigorous analysis, and an audit trail (Kabir, Sadiq and Tesfamariam, 2014). Chi and Bunker (2020) further promote the utilisation of these methods for infrastructure project evaluation due to their capacity to incorporate economic and non-monetary decision criteria within the same assessment framework. From a risk management perspective, MCDM methods are ideal as they account for broader environmental and social impacts, while adhering to the financial rigour of a CBA, which can even be incorporated into the final assessment as an evaluation criterion (Chi and Bunker, 2020). This is particularly advantageous when evaluating transformational transport infrastructure megaprojects, which undoubtedly deliver sizeable and wide ranging environmental, social and economic (both direct and indirect) impacts that need to be measured, whilst requiring stringent financial controls and cost analysis in order to prevent optimism bias, and to keep project costs in check (Chi and Bunker, 2020). Dimitriou, Ward and Dean (2016) suggest that this balance will become increasingly important as overarching policy considerations such as alignment with sustainable development goals and fiscal austerity (or perhaps in the current economic climate, fiscal prudence in stimulus spending) concurrently elevate the importance of environmental impact assessment (EIA) alongside CBAs in the project approval process. Due to this, the MCDM methodological framework is effective, as it allows project stakeholders to frame the project evaluation in a way that is inclusive of both their primary objectives and key concerns (Dimitriou, Ward and Dean, 2016). Thus, MCDM methods can paint a clearer picture of the relevant potential outcomes of the implemented project to the key stakeholders (Dimitriou, Ward and Dean, 2016).
Across Australia at both the state and federal level, assessment processes for public investment in transport infrastructure utilise MCDM as the overarching framework for the evaluation of projects (Chi and Bunker, 2020). Within this process, CBA is integrated as the mandatory means for economic evaluation (Infrastructure Australia, 2018). WebTAG, the UK Department for Transport’s project evaluation mechanism, is structured in a similar fashion, incorporating CBA, WEB, EIA, and additional social considerations within a broad MCDM framework (DFT, 2019). An example of the method’s practical application was on the assessment of the Northern Line Extension in London, where a ‘policy-led’ MCDM method was utilised for project assessment (Ward et al., 2016). Due to the project’s forecasted transformative WEB (primarily in terms of real estate appreciation in the spatial area of the rail line and increased accessibility), significant social and environmental impacts, and estimated total project cost of upwards of £1 billion, it was necessary to have an evaluation mechanism which was robust enough to adequately account for all these issues (Ward et al., 2016). This was particularly the case owing to the large amount of project stakeholders across multiple levels of government and the community (Ward et al., 2016). In this instance, the MCDM method for project evaluation was touted as a success as it was able to address the social costs and benefits of the project (not just from a financial perspective, but in terms of non-monetary winners and losers) and was flexible enough to address these major stakeholder issues within the same assessment framework (Ward et al., 2016). Indeed, the MCDM method has been praised for its capacity to integrate social considerations into the assessment process for infrastructure evaluation in a way that other evaluation methodologies are unable to do (Sierra, Yepes and Pellicer, 2018).
Criticisms of MCDM methods primarily focus on the potential adverse effects of the methodology’s subjectivity. In this respect, the methods greatest strength is its greatest weakness. The method’s capacity to flexibly integrate decision criteria chosen by stakeholders into the assessment process, whilst nominally allowing the factoring in of local community considerations (e.g., social or environmental issues) also leaves it susceptible to politically expedient decision-making (Te Boveldt, Van Raemdonck and Macharis, 2018; Dean, 2020). Moreover, achieving consensus on criteria when multiple stakeholders are involved is often problematic and can obstruct the assessment schedule. Key project stakeholder bodies frequently have disparate objectives, which are often hard to reconcile in a mutually agreeable fashion, thus leading to an impasse in the criteria selection process (Te Boveldt, Van Raemdonck and Macharis, 2018). Additional criticism of the MCDM methodology’s subjectivity is that it can lead to double counting of a project’s impact (Annema, Mouter and Razaei, 2015; Chi and Bunker, 2020). Chi and Bunker (2020), in their review of infrastructure assessment criteria utilised in Australia on major infrastructure projects, noted that the utilisation of similar evaluation criteria such as ‘service need’ and ‘strategic alignment’ left projects exposed to instances of double counting.
The issue of ‘weighting’ can also severely impact the partiality of an MCDM method. As noted by Gühnemann, Laird and Pearman (2012), a specific criterion’s weighting is representative of its relative status in the eye of policy makers. As there is no global standard or indicative benchmark for the weighting of assessment criteria, and generally none at the national nor provincial level, then the allocation of an individual criterion’s weighting is left in the hands of policymakers (Gühnemann, Laird and Pearman, 2012). If left unchecked, this can result in a biased project assessment, as policy makers are left with wriggle room to adjust weightings and potentially greenlight or kill off a project, if it is deemed politically expedient to do so. Meddling aside, in a best practice situation disproportionate weighting can still nullify the purpose of MCDM methods if one criterion is assigned a lopsided weighting (Dean, 2020). Dean (2020) states that MCDM evaluations are too often based off idealistic principles and assumptions that are disconnected from the real-world of transport infrastructure. Consequently, he urges greater transparency and standardisation of the MCDM methodology and suggests caution in its usage and adjustable weightings (Dean, 2020).
Method
This paper applies a novel evaluation model for measuring project success, known as i3d3, to a real transport infrastructure project: Gold Coast Light Rail (GLCR) Stage 1&2. The model represents a modern form of MCDM and includes twelve critical success factors that span financial, social, ethical and environmental consequences across the phases of project initiate (design), project implement (deliver) and project influence (delight). The basic structure of the model is shown in Figure 1. A key feature of the model is that success is judged by a different group of stakeholders as the project advances.
Figure 1. i3d3 conceptual framework
The twelve critical success factors, mostly measured in different units and combined by phase in unique ways, comprise feasible, useable, achievable and sustainable (design), within budget, on schedule, as specified and no surprises (deliver), and desirable, adaptable, practicable and serviceable (delight). Each is scored on a scale of -100 to +100 based on a measurable criterion, where a value of zero is the breakeven point between success and failure. Criterion outcomes are converted to success scores using fixed translation scales. The alignment of project outcomes to the 17 United Nations Sustainable Development Goals is also embedded within the model.
It is not feasible to fully describe how the i3d3 model works within the word count limit of this paper. A detailed explanation can be found online (CCCR, 2019). Nevertheless, it can be applied to determine the level of success for each project phase (initiate, implement, influence) and each consequence type (financial, social, ethical, environmental) to compute the overall ranking that enables projects to be compared with each other.
The project initiate phase is focused on design decisions that yield balanced performance across each consequence type. Feasibility is assessed using a conventional discounted cash flow approach for tangible benefits and costs over a period of up to 30 years. Useability is determined by surveying the level of local community support using a five-point Likert scale. Achievability is a function of the ratio of project rewards (opportunities) divided by risks (threats) based on their probability of occurrence and impact should they eventuate. Finally, sustainability is computed based on six core environmental factors using a five-star rating scheme. Success is viewed from the perspective of the project owner/sponsor and shareholders.
The project implement phase is focused on cost, time, scope and risk during delivery. This is computed as the variance between planned and actual performance. Cost is the amount spent to realise the project. Time is the duration from commencement to handover. Scope is the extent of work measured in terms of the quantity of functional units produced. Risk is a measure of the uncertainty surrounding project procurement. These factors are integrated together to reflect a preference for bigger scope and smaller cost, time and risk. Success is viewed from the perspective of the project team and regulatory authorities.
The project influence phase is focused on whether the project is well-received by those that use it. The assessment of satisfaction arises from at least 40 generic satisfaction attributes divided into two groups: wants and needs. Wants comprise attributes related to desirability and adaptability, whereas needs comprise attributes related to practicability and serviceability. Each are evaluated separately using a five-point Likert scale for both opinion (-2 to +2) and relevance (1 to 5) and then multiplied together and combined into an overall satisfaction index for each attribute group. The number of respondents that demonstrate a positive experience for both wants and needs divided by the survey sample computes their delight with the project during its first year of usage. Success is viewed from the perspective of project client/end-user and local community.
The i3d3 model clearly covers more than project selection decisions. It needs to both predict and validate actual performance in terms of project success while ensuring that envisaged benefits during design are realised in operation. It is able to combine multiple criteria measured using different units into a single ranking of success and to apply these outcomes for projects that display quite different characteristics. When applied progressively, as opposed to an audit on completion, it helps guide decisions to ensure favourable outcomes are not achieved by accident.
Results
This paper used a case study of GCLR to demonstrate the capability of modern MCDM in delivering major transportation infrastructure. The findings provide evidence that GCLR was successful in meeting its objectives. Although GCLR was not created as a response to COVID-19, future stages are being treated as nation-building fiscal stimulus projects.
In this study, all factors were assessed using criteria based on a combination of balanced scorecard technique, planned v actual progress, and end-user satisfaction surveys conducted in 2020 after the project had been completed and began operation. Data was gathered from documents freely available online, while delight scores were assessed from a specially designed Qualtrics survey distributed to a sample population of 413 people within the local community (with a 33% response rate achieved). The findings of the study were discussed with three senior members of the team at GoldlinQ Pty Ltd that is responsible for the project roll-out, to ensure there were no inadvertent errors or misunderstandings. All effort was made to use evidenced-based information as much as possible.
GCLR was shown to be successful, with an i3d3 ranking of 23. All twelve critical success factors were positive. Across the three phases of the project lifecycle, delight scored highest (35), then design (28) and deliver (5). Across the four consequences, environmental scored highest (28), then ethical (26), social (23) and financial (14). The net benefit for the project was 24 (also on a scale of -100 to +100) with seven stakeholder groups identified as winners, and no losers. The only United Nations Sustainable Development Goals (SDGs) that meet the model’s eligible requirements was ‘17 Partnerships of the Goals’, resulting in a humanity index of 20 (on a scale of 0 to +100). All twelve success factors are equally weighted and hence cannot be used to manipulate outcomes.
The decision support system (DSS) score for design comprised four measurement criteria. The benefit-cost ratio (BCR) was 1.0997 using tangible costs and benefits discounted by a real (inflation-free) rate of 4% over 30 years. Local project support (LPS) was based on a government survey of 2,142 people and was 0.8690 (on a scale of -2 to +2). Risk and reward (RAR) was 1.3226 based on the ratio of intangible benefits (opportunities) divided by intangible costs (threats) according to their probability and consequence. The ecological footprint (EFP) was 15 (on a scale of 0 to +30) equivalent to 2.5 stars.
The project delivery success (PDS) score for deliver also comprised four measurement criteria. The percentage change in cost (C) between planned and actual was 0% due to the guaranteed price public-private partnership agreement in place. The change in time (T) was also 0% change even though Stage 1 was behind schedule by six weeks and Stage 2 was ahead of schedule by the same amount. There was no change in scope (S) either, but a 5.13% reducing in risk (R) between planned and actual performance led to a small positive PDS overall. Basically, the project was delivered on schedule, within budget, as specified and with few surprises. PDS is computed as S3 / CTR between planned and actual. The project was assessed as ‘high complexity’ (12) but played no role in the PDS score.
The end-user satisfaction (EUS) score comprised a combination of wants (desirable and adaptable) and needs (practicable and serviceable). Out of 138 responses, 67.39% indicated they were happy with the project’s performance as represented by the number of data points in quadrant Q1 of the summary chart. Attractiveness comprised 20.85% of the total, flexibility was 14.25%, fit for purpose was 36.52% and enduring was 28.38%. The expected delight during design was computed as 43%, but upon completion the actual delight was slightly lower at 35%. This can, of course, change over time as patronage matures.
The completed template, downloaded from i3d3.net (https://www.i3d3.net), prints as six pages – where each page is presented in Figures 2-7 respectively to evidence the overall process. Data highlighted by light grey shading are the inputs to the model (actual end-user responses are not visible but automatically compute the satisfaction chart). The project’s actual cost was AUD 1,620 million, which meets the definition of a ‘megaproject’ when converted into USD. Use of MCDM in this evaluation did not require high levels of technical expertise, was undertaken over several days and did not involve any expenditure other than the time of the authors in collecting and entering the data.
Figure 2. GCLR evaluation summary
Figure 5. GCLR design assessment
Figure 6. GCLR deliver assessment
Figure 7. GCLR delight assessment
Discussion
MCDM offers improvement over other approaches to evaluating infrastructure projects as it is able to integrate the assessment of tangible values, based on BCR modelling and project delivery success, with the assessment of intangible values, based on project opportunities and threats, environment impact and end-user satisfaction. It can broaden the view of success from a sponsor’s expectations embedded in the decision to proceed, or from the project team’s perspective of meeting contractual requirements, or the often-forgotten end-users’ perspective of whether the designed solution was indeed effective. There isn’t anything fundamentally wrong with traditional evaluation approaches, but MCDM is evidenced in the literature as promising potential advantage. This is demonstrated by the GCLR case study in evaluating multiple stakeholder criteria and providing the opportunity to improve initiation decision-making on future projects based on the lessons learned from routine end-user feedback. This wider view of project performance is important to ensure that benefits realisation is made a top priority for infrastructure procurement that deploys public funds.
Successful projects should aim to bring a positive collective utility to our society, even though there will always be winners and losers. For a project to be successful, it must show this consistently over time (i.e., reflect good design, effective delivery and make a lasting contribution).
The i3d3 model is an example of modern MCDM. It expands the traditional implementation boundaries of project management to include front-end planning and post-handover evaluation. The integration of all three phases into a single measure of successful procurement is at best uncommon and at worst unachievable. For example, CBA might be used for an accept/reject decision to invest, earned value might be used to assess delivery progress and post-occupancy evaluation might be used to assess effectiveness in practice, but only MCDM can merge these diverse criteria into a single-point value. Although the GCLR case study was undertaken in hindsight, the i3d3 template should be progressively completed and updated as actual performance unfolds. The evaluation of delight could also be repeated in the future to see if end-user satisfaction has increased, decreased or remained constant over the long term. Cost calculations in i3d3 do not change with time, the size of the project has no significant impact on performance, and the location of the project does not inhibit its comparison in other jurisdictions. The use of purchasing power parity to compare ‘apples with apples’ is unnecessary with i3d3.
It is important to note that i3d3 can rank ‘apples with oranges’. The model is agnostic to the type of project, its size, its location and its date of completion. Previous case studies available on the i3d3.net website show that the GCLR project (rank = 23) performed better than the Hong Kong–Zhuhai–Macau Bridge project (rank = 22) but worse than the Bangladesh Electricity Upgrade and Expansion project (rank = 50). A study of the Tham Luang Cave Rescue in Thailand in 2018 showed it had a rank = 38. Other projects shared with the team at i3d3.net are intended to form a global database of project performance over time. Students enrolled in the Bachelor of Project Management at Bond University, as part of their capstone subject, are each asked to select and audit the success of a different well-documented project to become proficient in performing these types of studies.
There is a propensity to evaluate good projects and to use the results as a form of award for performance. Over time, it will be possible to objectively demonstrate the performance of projects and to calculate the ratio of success to failure. This requires making evaluations a routine part of the project management process.
The i3d3 model has two key attributes: transparency and simplicity. Depending on the political context within which projects are delivered, these could be advantages or disadvantages. But generally, greater transparency is a good thing and simplicity encourages deployment without introducing cost and time barriers. A complete study of a project undertaken in hindsight could be reasonably expected to take about a week for one person assuming there are no access restrictions that block the process. Competitions for ‘best project for the year’ would encourage engagement with industry if the eligibility criteria were based on a completed i3d3 template supported by an independent audit review.
The i3d3 model is endorsed by the Global Alliance for the Project Professions (GAPPS), which is a volunteer-driven, non-profit alliance that provides a reliable source for comparison of project-based standards and qualifications. Endorsement occurred in 2021. More information about GAPPS can be found at https://globalpmstandards.org/.
GCLR will form an ongoing evaluation activity with at least two more stages to be rolled out. This provides a useful opportunity to study how lessons learned from previous stages are used to improve performance in later stages. This will be reported in a future paper.
Conclusion
It is often said that sponsors no longer see projects delivered on time, within budget, as specified and with no surprises as sufficient evidence of success. Rather, what they really want is projects that are aligned with their strategic stakeholder objectives, providing a range of financial, social, ethical and environmental benefits that are fully realised as anticipated. This thinking is an expansion of project delivery success (with a focus on implementation) to project success (comprising initiation, implementation and influence). Projects are merely vehicles of change, and project managers as change-agents have a vested interest in successful outcomes.
The problem is that contribution to the wider definition of success is achievable only if project managers are involved early during the initiation phase (design) and stay connected during the influence phase (delight), which is not common.
It is found that use of the i3d3 model, a modern form of MCDM, enables successful projects to be distinguished from unsuccessful ones. In addition, projects can be ranked from best to worst across an organisational portfolio, geographic region or industry sector. It also supports past project performance to inform new design through application of a continuous improvement process of recording lessons learned. The case study was shown to have 100% of the critical success factors in the model positive and an overall success ranking of 23 (on a scale of -100 to +100). This paper presents the primary and secondary data and the outcome of calculations that support this finding.
The GCLR case study clearly demonstrates that the evaluation of project success is not a prohibitive activity. There are few, if any, methodological barriers to undertaking evaluations. While successful projects might be celebrated, evidence of unsuccessful projects may be quietly filed. But in either case, the opportunity for continuous improvement and the formal feedback from end-users to designers is ever-present.
The i3d3 model is a forerunner of other MCDM tools that will undoubtedly emerge over time. However, innovation is not a finite pursuit. The measurement of multi-phase project success is a fertile area for future research, especially in the built environment disciplines, as projects tend to be complex, expensive and have lasting effects on the communities they are designed to serve. Poor performance is therefore a waste of valuable resources. Especially in times of fiscal stimulus to boost the COVID-19 recovery, the importance of quality infrastructure investment is paramount.
Acknowledgements
The lead author is one of the team that developed i3d3.
The assistance of the Centre for Comparative Construction Research at Bond University and the team at i3d3.net is much appreciated. Both are dedicated to providing free resources and helpful industry support related to measuring project success in practice.
References
Annema, J.A., Mouter, N. and Razaei, J., 2015. Cost-benefit Analysis (CBA), or Multi-criteria Decision-making (MCDM) or Both: Politicians’ Perspective in Transport Policy Appraisal. Transportation Research Procedia, 10, pp.788-797. https://doi.org/10.1016/j.trpro.2015.09.032
Asplund, D. and Eliasson, J., 2016. Does uncertainty make cost-benefit analyses pointless? Transportation Research Part A: Policy and Practice, 92, pp.195-205. https://doi.org/10.1016/j.tra.2016.08.002
Atkins, G., Davies, N. and Bishop, T.K., 2017. How to value infrastructure? Improving cost-benefit analysis. Joint Report Institute for Government and Project Management Institute (PMI). Available at: https://www.instituteforgovernment.org.uk/sites/default/files/publications/IfG%20Report%20CBA%20infrastructure%20web%20final1.pdf [Accessed 21 August 2021].
Brathen, S. and Givoni, M., 2017. The wider impacts from transport: What do we know, what do we still need to know, and what does it mean? Research in Transportation Economics, 63, pp.1-4. https://doi.org/10.1016/j.retrec.2017.09.005
Broniewicz, B. and Ogrodnik, K., 2020. Multi-criteria analysis of transport infrastructure project. Transportation Research Part D: Transport and Environment, 83, p.102351. https://doi.org/10.1016/j.trd.2020.102351
Bundesministerium für Verkehr und Digital Infrastruktur (BMVI), 2014. Grundsätzliche Überprüfung und Weiterentwicklung der Nutzen-Kosten-Analyse im Bewertungsverfahren der Bundesverkehrswegeplanung. Available at: file://datatwo/12992102$/Profile/Downloads/SchlussberichtFE96.09742011-FE2BVWPNKA.pdf [Accessed 21 August 2021].
Cambridge Econometrics, 2021. E3ME. Available at: https://www.e3me.com [Accessed 21 August 2021].
CCCR, 2019. The application of the i3d3 model for measuring project success (working paper). Centre for Comparative Construction Research, Bond University. Available at: https://bond.edu.au/files/4744/Introducing%20i3d3.pdf [Accessed 21 August 2021].
Chi, S. and Bunker, J., 2020. An Australian perspective on real-lie cost-benefit analysis and assessment frameworks for transport infrastructure investments. Research in Transportation Economics, p.100946. https://doi.org/10.1016/j.retrec.2020.100946
Coyle, D., 2016. Infrastructure cannot be built on empty purses and big promises. March 16. Financial Times. Available at: www.ft.com/content/4bb57c70-e9cd-11e5-888e-2eadd5fbc4a4 [Accessed 21 August 2021].
Dean, M., 2020. Multi-criteria analysis. In: N. Mouter, ed. Standard Transport Appraisal Methods. Cambridge, MA: Academic Press. Ch. 6, pp.165-224. https://doi.org/10.1016/bs.atpp.2020.07.001
Department for Transport (DFT), 2019. Transport Analysis Guidance (TAG). Available at: https://www.gov.uk/guidance/transport-analysis-guidance-tag [Accessed 21 August 2021].
Dimitriou, H.T., Ward, J. and Dean, M., 2016. Presenting the case for the application of multi-criteria analysis to mega transport infrastructure project appraisal. Research in Transportation Economics, 58, pp.7-20. https://doi.org/10.1016/j.retrec.2016.08.002
Flyvbjerg, B. 2013. Quality control and due diligence in project management: Getting decisions right by taking the outside view. International Journal of Project Management, 31 (5), 760-774. https://doi.org/10.1016/j.ijproman.2012.10.007
Flyvbjerg, B. 2017. The Oxford handbook of megaproject management. Oxford Handbooks. https://doi.org/10.1093/oxfordhb/9780198732242.001.0001
G20. 2019. G20 Principles for Quality Infrastructure Investment. Osaka, Japan. https://www.mofa.go.jp/policy/economy/g20_summit/osaka19/pdf/documents/en/annex_01.pdf (accessed 21 August 2021)
Gühnemann, A., Laird, J.J., and Pearman, A.D. 2012. Combining cost-benefit and multi-criteria analysis to prioritise a national road infrastructure programme. Transport Policy, 23, 15-24. https://doi.org/10.1016/j.tranpol.2012.05.005
Guitouni, A., and Martel, J.M. 1998. Tentative guidelines to help choosing an appropriate MCDA method. European Journal of Operational Research, 109 (2), 501-521. https://doi.org/10.1016/S0377-2217(98)00073-3
Hansen, W., and Johansen, B.G. 2017. Regional repercussions of new transport infrastructure investments: An SCGE model analysis of wider economic impacts. Research in Transportation Economics, 63, 38-49. https://doi.org/10.1016/j.retrec.2017.07.004
Helm, D., and Mayer, C. 2016. Infrastructure: why it is under provided and badly managed. Oxford Review of Economic Policy, 32 (3), 343-359. https://doi.org/10.1093/oxrep/grw020
Infrastructure Australia. 2016. Inland Rail Project Evaluation Summary. https://www.infrastructureaustralia.gov.au/sites/default/files/2019-06/Final_Inland_Rail_Project_Evaluation_Summary_0.pdf (accessed 21 August 2021)
Infrastructure Australia. 2018. Assessment Framework: For initiatives and projects to be included in the Infrastructure Priority List. https://www.infrastructureaustralia.gov.au/sites/default/files/2019-06/infrastructure_australia_assessment_framework_2018.pdf (accessed 21 August 2021)
Infrastructure Australia 2021. CEO update: Budget 2020. https://www.infrastructureaustralia.gov.au/listing/newsletter/ceo-update-budget-2020 (accessed 21 August 2021)
Kabir, G., Sadiq, R., and Tesfamariam, S. 2014. A review of multi-criteria decision-making methods for infrastructure management. Structure and Infrastructure Engineering, 10 (9), 1176-1120. https://doi.org/10.1080/15732479.2013.795978
Köhler, J., Jin, Y., and Barker, T. 2008. Integrated Modelling of EU Transport Policy: Assessing Economic Growth Impacts from Social Marginal Cost Pricing and Infrastructure Investment. Journal of Transport Economics and Policy, 42 (1), 1-21.
Kyriacou, A.P., Muinelo-Gallo, L., and Roca-Sagales, O. 2019. The efficiency of transport infrastructure investment and the role of government quality: An empirical analysis. Transport Policy, 74, 93-102. https://doi.org/10.1016/j.tranpol.2018.11.017
Lakshmanan, T. 2011. The broader economic consequences of transport infrastructure investments. Journal of Transport Geography, 19 (1), 1-12. https://doi.org/10.1016/j.jtrangeo.2010.01.001
Legaspi, J., Hensher, D., and Wang, B. 2015. Estimating the wider economic benefits of transport investments: The case of the Sydney North West Rail Link project. Case Studies on Transport Policy, 3 (2), 182-195. https://doi.org/10.1016/j.cstp.2015.02.002
Levkovich, O. Rouwendal, J., and van Marwijk, R. 2016. The effects of highway development on housing prices. Transportation, 43 (2), 378-405. https://doi.org/10.1007/s11116-015-9580-7
Mouter, N. 2018. A critical assessment of discounting policies for transport cost-benefit analysis in five European practices. European Journal of Transport and Infrastructure Research, 18 (4), 389-412. https://doi.org/10.18757/ejtir.2018.18.4.3256
Odeck, J. 2014. Do reforms reduce the magnitudes of cost overruns in road projects? Statistical evidence from Norway. Transportation Research Part A, 65, 68-79. https://doi.org/10.1016/j.tra.2014.04.005
Ronnle, E. 2017. A novel approach to economic evaluation of infrastructure? Examining the benefits analyses in the Swedish high-speed rail project. Case Studies on Transport Policy, 5 (3), 492-498. https://doi.org/10.1016/j.cstp.2017.05.005
Rothengatter, W. 2017. Wider economic impacts of transport infrastructure investments: Relevant or negligible. Transport Policy, 59, 124-133. https://doi.org/10.1016/j.tranpol.2017.07.011
Sierra, L.A., Yepes, V., and Pellicer, E. 2018. A review of multi-criteria assessment of the social sustainability of infrastructures. Journal of Cleaner Production, 187, 496-513. https://doi.org/10.1016/j.rser.2016.04.073
Spackman, M. 2013. Handling non-monetised factors in project, programme and policy appraisal. Centre for Climate Change Economics and Policy, Grantham Research Institute on Climate Change and the Environment. London School of Economics. https://www.lse.ac.uk/granthaminstitute/wp-content/uploads/2014/03/PP-non-monetised-factors-project-programme-policy.pdf (accessed 21 August 2021)
Te Boveldt, G., Van Raemdonck, K., and Macharis, C. 2018. A new railway tunnel under Brussels? Assessing political feasibility and desirability with competence-based multi criteria analysis. Transport Policy, 66, 30-39. https://doi.org/10.1016/j.tranpol.2018.03.002
Van der Meer, J., Hartmann, A., Van der Horst, A., and Dewulf, G. 2020. Multi-criteria decision analysis and quality of design decisions in infrastructure tenders: a contractor’s perspective. Construction Management and Economics, 38 (2), 172-188. https://doi.org/10.1080/01446193.2019.1577559
Vickerman, R. 2008. Transit investment and economic development. Research in Transportation Economics, 23 (1), 107-115. https://doi.org/10.1016/j.retrec.2008.10.007
Vickerman, R. 2017. Beyond cost-benefit analysis: the search for a comprehensive evaluation of transport infrastructure. Research in Transportation Economics, 63 (1), 5-12. https://doi.org/10.1016/j.retrec.2017.04.003
Wangsness, P.B., Rodseth, K.L., and Hansen, W. 2017. A review of guidelines for including wider economic impacts in transport appraisal. Transport Reviews, 37 (1), 94-115. https://doi.org/10.1080/01441647.2016.1217283
Ward, E.J., Dimitriou, H.T., Wright, P., and Dean, M. 2016. Application of policy-led multi-criteria analysis to the project appraisal of the Northern Line Extension, London. Research in Transportation Economics, 58, 46-80. https://doi.org/10.1016/j.retrec.2016.08.004
Yen, B.T.H., Mulley, C., and Zhang, M. 2020. Equity in financing public transport infrastructure: Evaluating funding options. Transport Policy, 95, 68-77. https://doi.org/10.1016/j.tranpol.2020.06.001
Yen, B.T.H., Mulley, C., Shearer, H., and Burke, M. 2018. Announcement, construction or delivery: When does value uplift occur for residential properties? Evidence from the Gold Coast Light Rail system in Australia. Land Use Policy, 73, 412-422. https://doi.org/10.1016/j.landusepol.2018.02.007
1 Broniewicz and Ogrodnik (2020) alone were able to identify 46 different methods of multi-criteria assessment which had been applied in the field of transportation (although not all on the evaluation of transport infrastructure). Indeed, there is a body of literature dedicated to assisting with the selection of an appropriate method (Guitouni and Martel, 1998). Hence, a considered decision was made not to identify nor evaluate individual MCDM methods in this paper.