Understanding Megaproject Success beyond the Project CloseOut Stage

Project success has always been an important topic in the project management literature. One of the main discussions is concerned with how a project’s success is evaluated and what factors lead to achieving this success. Traditionally project success has been measured at the point where the project outputs are handed over, after the close out phase. Recently, questions have been raised in the literature as to whether we should be evaluating project success beyond the close out phase, to better account for organizational and societal outcomes. However, not much has been published about how the long term impacts and outcomes are measured. This is of particular concern in megaprojects as they often attract a high level of public attention and political interest, and have both direct and indirect impacts on the community, environment, and national budgets. In this paper the authors review success factors and criteria that are applicable to projects in general and megaprojects in particular. They identify the significance of evaluating outcomes and impact and propose an ex-post project evaluation (EPPE) framework for megaprojects.


Introduction
The success of a megaproject is typically assessed when the project has reached its goal or objective, and it is usually measured in terms on the conventional criteria of cost, time, and quality/performance, defined during the scoping stage of the project.However, in order to be successful, critical success factors (CSFs) need to be considered, including the specific conditions, events, and circumstances that lead to project results (Ika, 2009).The CSFs for a project become more challenging as a project increases in complexity.Issues such as large investments of time and money, uncertain scope, and increased stakeholder attention all raise the complexity of a project, and are all common factors in large infrastructure projects.Such projects are often categorised as megaprojects due to their tendency to consume large investments and timeframes, as well as involvement of various types of stakeholders (Fiori and Kovaka, 2005;Hertogh and Westerveld, 2014;Jia et al., 2011;Kardes et al., 2013;Turner and Zolin, 2012;Zhai, Xin and Cheng, 2009).
Research shows that most megaprojects fail to meet their stated objectives (Flyvbjerg, 2007).In other words, "megaprojects' characteristics cause significant project management difficulties that lead to underperformance" (Haidar and Ellis, 2010).Megaprojects tend to experience time and cost overruns (Flyvbjerg, 2009(Flyvbjerg, , 2014;;Han et al., 2009;Lehtonen, 2014;Merrow, 1988), and the project management literature has tended to emphasise time, cost and quality as the three primary criteria for understanding project success.However, Ika (2009) argued for these criteria to be extended, to include client satisfaction, realization of the strategic objectives of the client organizations, the satisfaction of end users, and the satisfaction of other stakeholders.However, many of these criteria can only be understood long after a project has been delivered.
Many studies have focused on identifying the CSFs required to deliver the project successfully (Ballantine and Stray, 1998;Chan, Ho and Tam, 2001;Chua, Kog and Loh, 1999;Fortune and White, 2006;Nguyen, Ogunlana and Thi Xuan Lan, 2004;Ogunlana, 2008;Yu et al., 2006).These studies typically only considered success up until the point at which the project was closed and outputs delivered.However, beyond the close-out phase the outputs of project had to produce results that not only demonstrated the performance of the product or service delivered, but which also met the criteria used to sanction the project at an organizational strategic level.This has been classified as the outcome and impact levels of a project (Turner and Zolin, 2012;Vedung, 2010).
Although the success of a megaproject may be classified during the project close-out phase when deliverables are accepted, stakeholders' lingering perception of success and the associated benefits are arguably more influenced by the longer term impacts and outcomes resulting from a project.Managing the CSFs that affect project impacts and outcomes might provide better results than exclusively focusing on CSFs targeted towards the successful acceptance of deliverables.This is particularly significant for megaprojects, where the impact and outcome stages of a project may last for many decades.
This paper reviewed the literature on CSFs and post-project evaluation to understand the factors contributing to the success of megaprojects.First, this paper elaborates on the CSFs in a typical project lifecycle.We then argue for the need to consider success beyond the traditional project lifecycle, the importance of post-project evaluation, and the need to consider CSFs beyond the project close out phase.The paper concludes by highlighting CSFs that are important to megaprojects.

Critical Success Factors in Project Lifecycle
Traditionally, a project's success is measured against its original scope, time of delivery, budget, and the quality or performance of deliverables.The term 'Critical Success Factors' refers to the specific "conditions, events, and circumstances that contribute to project results" (Ika, 2009, p. 8).Table 1 summarises CSFs that have been found to contribute to project success in research over the last fifteen years.
As projects become more complex they may be classified as megaprojects (Flyvbjerg, 2014;Kardes et al., 2013).Megaprojects have to perform in a challenging and unique environment, have to satisfy multiple stakeholders who often have conflicting expectations, and are often expected to deliver outcomes and benefits that go beyond specified budget and schedule (Fiori and Kovaka, 2005;Flyvbjerg, 2014;Jia et al., 2011;Kardes et al., 2013).These demands lead to a selection of issues more commonly faced on megaprojects than on smaller or more contained projects.Priemus (2010Priemus ( , p. 1037) ) identified the following key issues on megaprojects: inadequate problem analysis; lack of project alternatives; no functional programme; uncertainty about the scope of the project; inadequate process architecture; questions on analyses; contested information; issues with land acquisition; type of chosen technology; changing markets; political discontinuity and inconsistencies; and changing standards and changing legislation.Besides consuming a large investment in money and time, megaprojects also attract the interest of a wide range of stakeholders due to their political and social impacts.Attempting to meet the needs of a variety of stakeholders involved in megaprojects tends to cause uncertainty and change throughout the lifecycle.This is particularly significant with respect to the level of stakeholder engagement in decision-making, as pointed out by Priemus (2010).
Despite the length of many megaprojects, their delivery lifecycle still ends at a definite point at which deliverables reach final completion and acceptance from the client.The traditional project lifecycle in the PMBOK Guide (2013) defines four stages of a project: starting the project, organizing and preparing, carrying out the work, and closing out the project.In contrast, Priemus (2010) proposed a five-stage conceptualisation of the process of managing a megaproject.The steps he proposes are: (1) problem analysis; (2) compilation of a functional programme of requirements; (3) elaboration of the technical, practical and economic aspects and preparation of the project until it is ready for execution; (4) realization of the project from the moment the first spade hits the soil to the handover; and (5) the operation of the infrastructure after completion.It is interesting to note that only one of these five stages occurs during project implementation, the period on which traditional project management typically focuses.This is comparable to Lai's (2000) research on project communication, which identified that only one of six broad areas of project communication breakdown occurs during project implementation.
Project close-out marks the stage when project outputs are delivered, regardless of whether a project is local and contained, or a sprawling megaproject, delivered across years and different continents.However, the impact of a project depends on how deliverables are used after the project is completed.It may take some time before benefits are realised, or the impact of outcomes on the wider context within which the project is delivered, are understood.Therefore it is suggested that an additional phase should be included in the project lifecycle when examining CSFs.The following section reviews how project success is traditionally assessed, before exploring issues of temporality and alternative options for the evaluation of megaproject success.

Post-Project Evaluation
Post-project evaluation has been the subject of project management research for many decades (Kasi et al., 2008;Maheswar and Javalagi, 2014;Song and Letch, 2012;Toor and Ogunlana, 2010).However, post-project evaluation faces its challenges in practice.First, post-project evaluation is not an assessment of the sustainability of the benefits the project delivers (Cleland, 1985).Rather, the evaluation targets the time immediately after phase-out (Sandru, 2013).It tends to assess how a project satisfies its agreed plan, which may merely include time, cost, and quality/performance criteria.Archibald, Di Filippo and Di Filippo (2012, p. 29) emphasise the importance of post-project evaluation and note that the "post-project evaluation phase obviously requires a flexible amount of time depending on the type of product that the project has produced" .However, it can be difficult to conduct a thorough evaluation, as project personnel are typically released once the project is finished (Ahsan and Gunawan, 2010).This can create issues associated with access to the data needed to conduct the evaluation.
The second barrier to conducting post-project evaluation relates to management support for the evaluation, and a more general lack of support to continuous improvement, in favour for sanitised reporting buried issues (Bowen, Cheung and Rohde, 2007).In many cases a project has no obligation to conduct post-project appraisal (Ahsan and Gunawan, 2010).It has been found that project managers often perceive evaluations as a mere formality (Al-Yaseen et al., 2010), and that existing policies and procedures can limit the ways that information from post-project reviews is to be relayed back to decision makers (Kumar, 1990).
However post-project evaluation remains an essential part of how organizations learn and improve.Although many of the studies of post-project evaluation focus at the level of those stakeholders who were intimately involved in the process, a management that supports the evaluation process, and values transparency in the process, remains vital to capability development.Despite the significance of post-project evaluation, this project stage seems to have received very little attention in published project management bodies of knowledge.Archibald, Di Filippo and Di Filippo (2012) argue that the traditional project lifecycle promoted by the PMBOK Guide and other similar normative texts, is not complete.They claim that a post-project evaluation stage should be added to the project lifecycle, involving four dimensions: "project management dimension, product dimension, stakeholder satisfaction dimension, and the cognitive constraint dimension" (Archibald, Di Filippo and Di Filippo, 2012, pp. 26-27).

Project outputs, outcomes and impacts
Although project success is usually assessed at the time that deliverables are accepted by the client, the real success of a project is often only understood after the benefits have been achieved (Song and Letch, 2012).Long-term perception of success tends to result from an assessment of deliverables' beneficial impacts on the wider purpose and context.These results are typically experienced at the strategic level, not necessarily at the tactical level at which they were delivered.
At the tactical level finishing the project means that the project has entered the operational phase.At the strategic level, a project's success needs to be assessed much more comprehensively.
One of the difficulties in comprehensively assessing the success of a project lies in what Li (2008) refers to as a "conditional causality" which makes the relationship between project outputs and outcomes somewhat opaque.A simplistic perspective on the link between output and outcome may neglect the importance of the timeframe of evaluation, which becomes increasingly apparent to stakeholders as promised benefits either do, or do not, appear.Within these conditional relationships and timeframes, stakeholders possess different expectations.
Turner and Zolin ( 2012) have referred to this as "different timelines, various stakeholders".They assert that: project success is measured not just by completion of the scope of work to time, cost, and quality, but also by performance of the project's outputs, outcomes, and impacts, and thereby the achievement of the desired business objectives, as assessed by different stakeholders over different timescales (Turner and Zolin, 2012, pp. 87-88).
According to a system developed by the Asian Development Bank (Turner and Zolin, 2012, p. 4) project results can be divided into three stages: project output, project outcome, and project impact.Table 2 summarises factors related to success according to this categorisation, drawing on Turner and Zolin (2012) and sources.
Benefits may be recognized in financial and non-financial forms.On a daily basis the financial benefits include improved system costs, such as operational cost, training cost, maintenance cost, upgrade cost, reduction in other staff cost, reduction in salaries, and other expenses saved (Al-Yaseen et al., 2010;Kumar, 1990;Love and Irani, 2001).Non-financial benefits mostly cover the operational and the intangible benefits (Liu et al., 2003) such as value creation (Archibald, Di Filippo and Di Filippo, 2012;Horvath, Hatfield and Hill, 2011)) through project outputs.Along with the expected outcomes shown in Table 2, the non-financial benefits may include changes in the system's effectiveness and efficiency (Poon and Wagner, 2001) or the quality of programmes (Eldabi, Paul and Sbeih, 2003).It is necessary to consider the intended and unintended benefits of project's outputs if the project success is to be assessed comprehensively (Archibald, Di Filippo and Di Filippo, 2012).Evaluation at the outcome level may also provide benefits that evaluation of outputs at project completion cannot provide.Vedung (2010, p. 273) has identified that evaluation at the outcome stage produces safer knowledge, based on the presentation of empirical evidence of intervention effects.
The general literature on evaluation provides further insight into approaches that may be appropriate for the evaluation of megaproject outcomes and impacts.Conrad and Miller (1987, p. 28) suggest that evaluation has five primary stages; 1) measuring philosophy, 2) means testing, 3) implementing testing, 4) philosophy testing, and 5) reflection.
Due to their extended duration and the wide variety of stakeholders potentially influencing megaprojects' success criteria, goals on a megaproject can become somewhat dynamic.Lehtonen (2014, p. 289) comments, that in dynamic situations it may be necessary to use more than ex-ante evaluation.To account for the dynamic, emerging goals as an evaluation criterion, it would be necessary to broaden the criteria and perspectives considered in an evaluation.The present bias in favour of ex-ante assessment would likewise need to be complemented by ex-post evaluation and ex nunc monitoring.
Combining Lehtonen's (2014) classification of evaluation and Turner and Zolin's (2012) model for assessing project success, it is possible to demonstrate linkage between the stages for conducting the evaluation and the purpose of conducting it (Table 3).In this paper the evaluation of a megaproject's outcomes and impacts is referred to as ex-post project evaluation (EPPE).EPPE refers to both the period and the purpose of evaluation.Ex-ante evaluation is simpler to conduct.It can be performed when a majority of project staff are still available to participate.It is also a simpler process to reach consensus regarding the criteria that should be used in ex-ante evaluation.However, the original goals that initiated a megaproject are more likely to be at the output or impact levels.Assessment at these levels will not be possible during the close out phase, as benefits will likely not yet have been realised.In order to achieve the ultimate goals of a megaproject, it is believed that CSFs do still exist, and determine sustainability.It is expected that EPPE can be of assistance in revealing these factors.

Revealing CSFs beyond project closing out phase
To improve project performance it is important to consider factors leading to the achievement of desired benefits, unexpected outcomes or effects, and the impact of the project.These can only be assessed by conducting a post-project evaluation.Table 4 summarizes previous research into critical success factors at the project outcome level.Vedung (2010, p. 273) identifies the outcome stage as the appropriate level at which to produce safe knowledge and empirical evidence of intervention effects.Outcome evaluations may focus on the individual level, organizational level, community level, and the policy or government level (Mathison, 2005, p. 287).
It is also important to consider whether the project stakeholders planned the outcomes of a project.Turner and Cochrane's (1993) classification of project types may provide one approach to addressing this question.These purposes would be achieved by employing project management tools and techniques, such as suggested, for instance, in PMBOK Guide (Project Management Institute, 2013) and PRINCE2 (AXELOS, 2015).Arguably, the timeframe will determine the consistency of goals and methods used.The longer time period a project requires (e.g.megaproject), the more challenging it becomes to maintain consistent goals and methods.This consistency will also be revealed when conducting a post project evaluation.The comparison between initiated and expected outcome and the actual, expected or unexpected result implies that there are essential elements (CSFs) contributing the project results at later stages.These factors may provide causal-relationship explanation to the achievements.
The issue of temporality in megaproject evaluation is particularly significant, not only due to the extended duration of most megaprojects, but also the period over which benefits are typically realized.Issues of the sustainability of outcomes need to be considered during evaluation.For example, the I-595 Port Everglades Expressway, the Denver Airport Megaproject, Boston's Central Artery/Tunnel Project, and power plants are delivered on the expectation of expected to have sustained economic impacts.A variety of studies have identified outcome specific critical success factors, and these are summarised in Table 4.
However, the majority of factors identified in Table 3 are taken from relatively small to medium sized projects, and caution should be extended when transferring them to megaprojects.Studies of CSFs have generally not depicted essential factors leading to the achievement of megaproject goals at the outcome and impact stages.These CSFs, for example, include managing risks (Kwak and Smith, 2009), top management involvement, business plans, vision, vendor support, change readiness, teamwork, team composition and communication (Ramayah et al., 2007).
Other CSFs related to megaproject delivery may include lack of owner's ability to manage a hitech oriented megaproject; frequent changes triggered by conflicts between public agencies and growing public resistance from environmental concerns; inappropriate project delivery system; a lack of appropriate scheduling for the size of the project (Han et al., 2009).Other factors may include: lack of clear constraint, marketing and estimation issues, and financial capability (Le-Hoai, Lee and Lee, 2008).In the context of megaprojects in the construction industry, instead of referring to success factors, Toor and Ogunlana (2010) refer to key performance indicators (KPIs), and focus on safety, efficient use of resources, effectiveness, satisfaction of stakeholders, and conflicts and dispute reduction.good support of and cooperation with the central agencies concerning the financial reforms results-oriented culture and the acknowledgment of the necessity of cash reporting Paul (1995, p. 63) -Senior management has greater responsibility at the key performance outcome level while middle management and general workforce have greater responsibility at the key performance driver level.-Top management looks to the drivers and the people who manage them to get the outcomes moving in the right direction.Lee (1990) -User satisfaction measures the effectiveness of an information system.(Pinjani and Palvia, 2007) -Group diversity Kassahun ( 2012 Greater organizational support Funnell (2000) -Agreement by business to meetings with program advisers with a view to identifying possible solutions; few refusals -Preparation of action plans that include defined key elements -Business-specific examples of increased willingness Evaluation is conducted in order to measure the congruence planned and actual project results.A common assumption in evaluation is the link between actions taken during the project, and project results.However, Cook (2000) argues that the evaluation is likely to be insufficient to conclude a causal inference; rather than "falsely choosing" between randomization and program theory, the evaluator can make the optimal choice and combine both.In other words, for instance, the evaluation of IT/IS in organizations can be seen as the movement from automating to informing, more recently to transformation (Ballantine and Stray, 1998).In addition, the role of IT/IS has changed from one of support to one of strategic importance, the focus of evaluation has progressed from efficiency to effectiveness, and advanced understanding (Love and Irani, 2001).The tendency of project results is the reason why project outputs need to be evaluated for a period after completion.This stage is one of the additional steps proposed by Archibald, Di Filippo and Di Filippo (2012).Song and Letch (2012, p. 276) present "a descriptive analysis of research on IT/IS evaluation over the last 25 years ".Even though their study only focuses on IT/IS sector, it is a seminal reflection on the importance of conducting post-project evaluation.Their study examines why evaluation is carried out.Their findings confirm the importance of identifying and appraising the IT/IS value (conceptual purpose), as well as the IT/IS planning and implementation (instrumental use).They also examine the timing of evaluations, finding that almost 60% of the assessment is conducted at ex-post or after the implementation stage.Their findings highlight the importance of evaluating a project's outputs after the project has finished.
However, not every post-project evaluation succeeds in revealing factors that contribute to the achievement of the project result at outcome stage.Evaluations typically focus on assessing the contribution to the organization.Critical success factors of a project should also cover later stages of a project after outputs are delivered.However, the literature in this area is lacking.

Conclusion
Project success criteria include time, cost, quality, client satisfaction, the realisation of the strategic objectives of the client organizations, and the satisfaction of end users and other stakeholders.Success is contributed to by critical success factors, defining the specific conditions, events, and circumstances that contribute to project results.As aspects of success extend past the handover of deliverables, critical success factors need to be considered after the project has delivered the outputs.The later stages are termed project outcome and impact level.
To be able to determine the CSFs at project outcome and impact level, post project evaluation needs to be conducted.Such evaluation plays a significant role in establishing a means to assess project performance, comprehensively, at the strategic level.The evaluation determines the causal relationship between project process, outcome and impact, as well the evaluation congruence between planned and actual outcomes.It is anticipated that the CSFs that affect megaproject outcomes and impacts will vary between sectors and industries.Future research may wish to examine these contingent differences, with a view towards understanding how outcomes and impacts can most effectively be achieved.
) -A public sector organization in a developing economy can use BPR to improve process and overall organizational performance if it (a) has accumulated stock of BPR-relevant resources and capabilities; (b) has undertaken BPR with sufficient depth; (c) is developing a post-BPR complementary competencies to sustain and further enhance the BPR changes; and (d) has mitigated the adverse effect of BPR implementation problems Dong, Neufeld and Higgins (2009) Top management support -Top management support to resource provision -Top management support to change management -Top management support to vision sharing Veiga et al. (2014)

Table 1 -
Studies on critical success factors

Table 3 -
Stages in Conducting Project Evaluation

Table 4 -
CSFs at Outcome Level