The four-level project success framework: application and assessment

Success is one of the ultimate goals of any project endeavour. Thus, clarifying the meaning of success is a vital step in achieving the desired success. In this study, the authors reviewed the project success literature and provided a framework for defining and evaluating project success. The framework consists of four levels that contain the possible criteria for assessing and evaluating success. The authors demonstrate the framework by case application. Further, experts in the field of project management conducted an external evaluation of the framework to assess its merits.


Introduction
Perhaps the most beloved word for any project practitioner is 'success'.However, the word's meaning is not universal; it can mean different things to different assessors (Andersen et al 2006;Davis 2013).Unspecified goals are not achievable; therefore, to achieve success, the exact meaning of the term should be specified upfront (Hamidovic & Krajnovic 2005;Thomas & Fernández 2008;Davis, 2013).This can be achieved in any particular project using the various tools available for defining success, such as the iron triangle.However, there is no agreement upon a single universal definition of success that fits all projects (McLeod et al 2012).Thus, defining success in the field of project management (PM) is a project-specific process.Because the world is changing continuously and each project is unique by definition, the need for methods and tools to define success may continue into the foreseeable future.
The aim of this study was to develop a framework for the process of success definition and evaluation.The first step was to review the literature and sum up the current situation in project success studies.Next, the framework was constructed and described.The authors then applied the framework to several cases.For evaluation purpose, the authors sent the framework to several experts and practitioners in the project management field.The experts' opinions are reported in this study along with recommendations for future development.

Literature review
The discipline of PM is relatively new so a clearer definition and understanding of project success took some time to develop (McLeod et al 2012).From the late 1950s up to the 1970s, project success was conceived as completing projects according to time, cost and quality; this is known as the iron or golden triangle (Jugdev & Müller 2005;Ika 2009).This triangle was and still is widely accepted but is criticised for being insufficient to fully define project success (Dvir et al 1998;Milis et al 2003;Ika 2009).Completing a project according to such criteria does not necessarily mean success (Turner & Zolin 2012).For example, the F-20 aircraft project was completed according to the iron triangle, yet it was a failure, and not a single aircraft was sold (Martin & Schmidt 1987).This triangle is very operational and could not assess the strategic dimensions of projects.Also, it lacks the ability to assess the soft dimensions of projects such as customer communication (Davis 2013).The project success concept has more criteria than that of the iron triangle.
De Wit (1988) enhanced the concept of project success by splitting it into two parts: project success (meaning the product), and PM success (meaning the managerial processes).He argued that product success should be assessed separately from the success of PM activities.Successful PM activities are not a guarantee for a successful product and a successful product is not necessarily the result of good PM activities.For example, the Sydney Opera House suffered huge cost and time overruns but it is a world renowned product (Ika 2009).
Further discussions of success include projects with a successful product and efficient PM activities, but no desired result.In the example of the F-20 there were good PM processes and a fantastic aircraft.Thus success is achieved according to de Wit's (1988) proposal but the actual project assessment shows a failure.From the 1990s, there was further development of the concept of project success.Researchers argued that project success should be understood according to frameworks consisting of different levels or dimensions (Jugdev &Müller 2005).Each level or dimension has its own criteria.For example, Shenhar et al (1997) introduced business as a distinct dimension of project success in additionto the previous work of de Wit.They suggest four success dimensions, namely: project efficiency (equivalent to PM activities); impact on the customer (equivalent to the product); business and direct success; and preparing for the future (Shenhar et al 1997).Even though they consider these as four distinct dimensions, direct success and preparing for the future are business dimensions with a timeframe difference; short term and long term, respectively.Later, these four dimensions (containing 13 measures) formed a project success multidimensional strategic framework (Shenhar et al 2001).Assessing the F-20 project according to this framework shows that the project failed at the business dimension when it could not attract sales.Baccarini (1999) introduced the logical framework method to define project success.He argued that four levels of project objectives should be used to define success.These levels are: goals, purpose, output and input.Baccarini's four levels are similar to Shenhar's four dimensions as shown in Table 1.(Jugdev & Müller 2005).These attitudes continue today and they comply more effectively with the multidimensional nature of projects.Projects usually have multiple stakeholders with different points of view who perceive project success differently (Andersen et al 2006;Davis 2013).
Researchers also highlighted the influence of isolated factors on the perception of success.For example, cultural differences affect the perception of project success between teams in cross-national projects (Pereira et al 2008).In addition, the definition of success is sensitive to the time of assessment.Different perceptions, criteria and factors of success are required at different stages of the project lifecycle (Shenhar et al 2001;McLeod et al 2012).For example, during the planning phase a success criterion might be addressing the needs of the right group of beneficiaries while after the completion of the project this changes to having the planned impact on the beneficiaries (Do Ba & Tun Lin 2008).During the planning no impact can be measured and it is useless to address the needs after the completion.Despite all attempts, there is no consensus on the definition of project success (Ika 2009;Han et al 2012).Many factors influence the definition process and every evaluator perceives success from a different point of view (Baccarini 1999;Turner & Zolin 2012;Kerzner 2013).Thus, there is a need for more tools to capture the various aspects of project success.These tools could possibly be the systematic success frameworks.More frameworks are needed as one framework does not fit all projects (Shenhar et al 2002).
The review of the literature showed that the definition of success evolved from a simple definition to systematic success frameworks.These frameworks contain the success criteria and factors at different levels or dimensions.However, these frameworks did not take into account the environment or the context surrounding the project.In fact, the impact of the context on projects is presented in literature (Balachandra & Friar 1997;Engwall 2003;Maaninen-Olsson & Müllern 2009).Many factors that affect project success (such as culture) lie outside the levels mentioned in the previous frameworks, namely, project, product and business.This highlights the need for introducing the context to the levels of success frameworks.By doing so, a wider spectrum of assessment and evaluation needs can be fulfilled.

The four-level project success framework
The literature review supports the proposition that there is an important dimension not explicitly mentioned in the process of formulating the concept of project success -that of context, meaning the circumstances surrounding the project.Each project or a set of closely related projects has its own contextual circumstances that have a significant impact upon the project's success.In addition, the literature contains many success criteria groups such as the project efficiency criteria group (Shenhar et al 2001).However, these groups of criteria do not impact equally on the overall project success and upon each other.Some groups may have a greater impact than others.The literature does not reveal which groups have the greater influence.Based on these observations, a new level, 'context', is added to the three existing levels described in the literature to formulate the four-level project success framework.
Projects differ widely (Shenhar et al 2002) so that claiming a universal set of criteria to measure success or to propose a single universal success definition for all projects might be problematic (McLeod 2012).Rather, the framework in this study is a proposal to guide the process of project success definition, to facilitate planning for success, and to aid in judging and evaluating projects after completion.

The four levels
In the project success literature, there are many success criteria.Regardless of the different names that authors use for these criteria, they can be allocated to one of the following levels: • Project process: this level contains the criteria used to judge the actions taken to provide the required deliverables.Examples of such criteria are: meeting budget and schedule, and efficiency of execution.
• Products and deliverables: this level contains the criteria used to judge the technical requirements and qualities of the products or deliverables resulting from the project.Examples of such criteria are: technical validity, manufacturability and technical performance.
• Business: this level contains the criteria used to judge the benefits and returns (or losses) of the project to the stakeholders.Examples of such criteria are: the contribution of the project to the strategic mission of the firm, preparing for the future, and satisfying the needs of the users.• Context and externalities: this level contains the criteria used to judge the project based on compliance with the contextual circumstances and externalities that affect it, such as the political situation, regime and climate.The project team or organization has little or no control over these externalities.These four levels form the framework.They are called 'levels' because, when they are vertically linked, the relationship between them becomes clearer.

Framework construct and characteristics
The four levels described above are linked in a systematic framework to reveal their practical advantage.Figure 1 depicts this framework.
The following characteristics describe this framework: • The project can be assessed at each level separately as well as overall.The nature of assessment at one level might be different from that at another level.Consequently, different decisions might be made about the overall project or about certain levels of the project.For example, the Sydney Opera House was significantly over budget and behind schedule so it was a failure at the project processes level.However, it is considered one of Australia's greatest icons so it was a success at the products and deliverables level.
• The higher the level, the higher is its influence in the perceived success or failure of the project.For example, the success of the Sydney Opera House at the product level affected the overall perception of the project.Today, the fact that the project suffered overruns is hardly remembered; the overwhelming majority of people talk about its beauty.• The impact of a lower level on a higher level is probable (dashed lines) but that of a higher level on a lower level is certain (solid lines).The overruns of the Sydney Opera House could have destroyed the project but its outstanding success at the product level eliminated that probable impact.• All levels can contain a particular criterion simultaneously; however, the measures of this criterion can differ from one level to another.For example, if time is important, the measure at context level might be (how short is the time taken to get the product to the market), at the business level might be (how quick is the decision-making strategy), at the product level might be (how quickly the product can be manufactured) and at the project level might be (is the project on schedule).• The assessment criteria at a lower level are linked to those at a higher level.Any change at the higher level alters those at the lower level.The significance of any alteration at the lower level depends upon the significance of the change at a higher level.For example, 'compliance with government human resource policies' might be a criterion at the context and externalities level.Any change in these policies will require changes at lower levels.For instance, if policy changes at the context level in regards to women's empowerment or annual leave, changes must be made to gender balance in the recruitment process, or to work schedules, at the project level.• The assessment of a certain level is time dependant and will be subject to change during the life cycle of that level; however, if the life cycle ends, the assessment will remain regardless of the change at higher levels.The case of the P-51 Mustang aircraft project is a good illustration.This project was an outstanding success at all levels until the end of World War II.From that war until today, judgement at the project level has not changed because there was no change at the context level during the project's execution.At the product level, the aircraft was a successful fighter in 1945 but was rendered obsolete in 1950 because there was a change at the context level-the introduction of jet fighters.• Criteria at a lower level may not explicitly appear at higher levels.However, all the criteria at higher levels must be fulfilled through the criteria of at least one of the lower levels.For example, a project to produce a car might have 'emission level' as a criterion at product level; this criterion may not appear with the same title at the project processes level or the business level but it may appear at the context level as 'compliance with government emission regulations'.• If contradicting criteria exist, the criterion with links to a criterion at a higher level must be prioritised.For example, if 'leading the competition' is a criterion at the business level, several criteria can exist at product level, such as 'cheap product' and 'high quality product'.These two criteria are contradictory to some extent.Toprioritise them, the business level criteria must be clarified first.If the criterion at business level was rewritten as 'leading the competition by being most affordable', it is clear that the 'cheaper product' criterion has higher priority than the 'high quality product' criterion at product level.

Setting the success criteria
The process of setting the success-measuring criteria needs to be performed sequentially in order to capture the impact of one level upon another.According to this framework, the upper level success criteria must be set first.Any business must be performed within a certain context, whether it is a geographical region, a political system, a social environment or an industry.Therefore, the contextual success criteria must be set prior to those at the business level.Any product must serve a certain business and, in order to set the success criteria for this product, the business success criteria must be set first.Any product is an outcome of a project and, in order to set the success criteria for the project, the product success criteria must be set first.This descending order allows the impacts of higher level criteria upon lower levels to be captured.It also aids criteria prioritisation.Moreover, having the criteria linked in this way helps to produce the specific success model for a particular project.Then by using a suitable weighting approach, an operationalized success model can be created for each single project.

The framework's applications
This framework can be applied to the process of defining success for and/or evaluating the success of a particular project.The following cases demonstrate the application of this framework to both tasks.

Application in project success definition
A project is by definition a unique endeavour.Therefore, the meaning of success for a particular project should have some sort of uniqueness.The following two examples demonstrate the application of the framework to the task of success definition.
Project success definition in research about project management in the British aviation industry during World War II This example shows the use of the four-level project success framework to define, evaluate and understand project success in historical research about project management in the British aviation industry during World War II (Howsawi et al 2014).The analysis of the data related to that research shows that the context of the war contains at least six conditions as follows: • Government control over business practices and the market The criteria at context level are derived from these contextual conditions.Failure to deal with any of the above conditions would mean certain failure to a project.Therefore, the success at the context level concerns projects that managed to deal with these contextual conditions successfully.
At the business level, a successful project is one that attracts enough production to keep the business running in the contextual conditions.At the product level, success is achieved by satisfying client requirements given the conditions at the context level.At the project level, success is achieved by producing a satisfactory prototype given the contextual conditions.Therefore, a successful project can be defined as one that manages to deal with the contextual conditions and produces a product that attracts reasonable production orders.This definition can be customised to suit the specific research topic as follows: an aviation project that managed to deal with the circumstances of World War II in the United Kingdom (UK) and produced an aircraft that satisfied the government enough to issue a quantity production order.
This definition encapsulates the criteria of success at all levels in a subjective way.Then an objective or operationalisable indicator is needed to evaluate the project.The production figure can serve this purpose because during World War II no aircraft was authorised to be manufactured unless it satisfied British Government standards.The production quantity was also subject to government authorisation.Moreover, the continuous evaluation of aircraft could result in a halting of production if an aircraft was discovered to be flawed.Consequently, only aircraft with proven efficiency were ever produced in large quantities.For example, the Avro Manchester bomber went into production but when it proved unreliable, production was ceased after only 209 units had been produced.
This indicator illustrates the logic of the four-level project success framework.A project is more likely to reach mass production phase if it managed to deal with the contextual requirements.This was the case in many projects in the British aviation industry.The large production quantities were a good source of revenue to the manufacturing companies, so their businesses succeeded as a result of this revenue.Business success is likely if the business offers a satisfactory product to the customer.This was certainly the case in the UK because the circumstances of the time determined that only satisfactory products would pass into production.All the aircraft that were produced in large quantities -more than 500 units -were put to good use in the Royal Air Force.Given the circumstances of the time, project management processes were likely to produce a good product, within the budget and on time.During World War II in the UK, the circumstances were unforgiving and delays or overruns could cancel a project.For example, the Martin-Baker MB3 aircraft was a good aircraft during tests but it was considered outdated because of delays and late delivery.Eventually that aircraft never progressed to production.
The production figure can also indicate the following: • Ease of production to satisfy the urgency.This means more units can be produced with less effort.For example, the de Havilland Mosquito aircraft could be produced using simple carpentry tools because it was made of wood.Also, it was constructed in modules, so these modules were produced in many small shops.• Client satisfaction.The only requirement was to satisfy the British Government, since it was the sole customer and the only body that could authorise production.• Good utilisation of manpower to overcome the skilled manpower shortage.Good utilisation of manpower means ability to produce more units of the product.
• Versatility to satisfy changing requirements since the same aircraft could serve different roles.Suitability for more roles means more units of the product are needed.• Good utilisation of available material to overcome the disturbance of material supply.Good utilisation of material means ability to produce more units of the product.
This is an example of using the four-level success framework to create a customised definition of success.
Project success definition in research about the management of the Aswan High Dam construction project This is another example of using the four-level project success framework in defining, evaluating and understanding project success within the context of the Aswan High Dam (AHD) project.Success here is defined as satisfying the following criteria: • The product is delivered despite all the difficult and threatening circumstances surrounding the project • The advantages of the project vastly outweigh the disadvantages • The project provides economic benefits to the owner and good revenue to the contractors • The project provides an excellent product to the specifications, within the specified time and budget.Table 2 shows how this definition fits the AHD project.

Table 2. The definition of success to the AHD project Level
Criterion Success indicator

Context level
Delivering the desired product despite all the difficult and threatening circumstances surrounding the project Completed or not?Despite all the complications of the 1960s crises, the project was completed The advantages of the project vastly outweigh the disadvantages The opinion of the majority?
The majority of experts consider the benefits of AHD to be far greater than the disadvantages

Business level
Providing economic benefits to the owner and good revenue to the contractors Measures of economic benefit?On completion, the AHD provided more than 50% of national electrical power, and the company that built the dam is still in business today

Deliverables level
Providing an excellent product to specification Technical specifications?
The AHD is considered by many experts to be one of the finest engineering constructions in the world

Project process level
Providing an excellent product within the specified time and budget Time?Cost?
The project met the time frame of 10 years and the estimated cost of one billion dollars Application in project success evaluation Another use of this framework is to evaluate and analyse projects to reveal the multiple meanings of success in the project and to find the possible factors for success or failure.
Here are some examples to show this application.
The case of the F-20 Tiger Shark fighter aircraft In the early 1960s the Northrop Company privately designed and produced the F-5 Freedom Fighter aircraft as a low-cost, less-sophisticated fighter for export.The aircraft achieved great success in the export market and more than 2000 units were built.The company wanted to build upon this success and, in the early 1980s, started another privately financed project to produce the F-20 Tiger Shark.The company succeeded in producing the aircraft which was one of the best of its day in terms of capabilities and cost efficiency (success at product level) and its project fulfilled the budget, time and quality constraints (success at project process level).Nevertheless, at business level, this project was a loss and not a single aircraft was sold.An analysis of the project attributed the failure mainly to an unexpected change that happened at the context level; a change in the government weapon export policy.The government lifted the export ban of more sophisticated aircraft.This change ruined the basis on which the project was set, which was "less sophisticated aircraft for export".Eventually the project was terminated.The impact of the context level and the resulting failure at the business level (higher levels) influenced the fate of the project despite the success at project and product levels (lower levels).
The case of the de Havilland Mosquito aircraft project The de Havilland DH.98 Mosquito aircraft project in the UK during World War II is an example of a remarkably successful project.The context of the war imposed many demanding restrictions.Among the most important demanding conditions were: the scarcity of material needed for aircraft manufacturing, especially aluminum alloys; the lack of trained labour in metal manufacturing; and the need for quick production.These conditions belong to the context level in the four-level project success framework.The de Havilland Company realised this and adopted the strategy of using innovative and radical ideas to comply with the contextual conditions and to exploit the core expertise of the company -speed.As a result of this strategy the company built a wooden aircraft instead of using metal.This shift in the material used provided several advantages at the project level: • The ease of obtaining work authorisation from the government as the project would not consume precious and scarce material such as aluminium.• The availability of professional wood labourers • The availability of wood • The simplicity in production as wood work was relatively easier than metal work As a result, the company was able to deliver the aircraft in just 11 months after signing the contract.
Several innovative design features also helped the Mosquito to achieve success at the product level.Some of these features were: • Speed: the Mosquito was the fastest aircraft in its class.
• Manufacturability: the aircraft was designed in modules so it could be produced in many small shops.• Versatility: the Mosquito was labelled as "one aircraft for all purposes".Overall, the project was a success at all levels.The Mosquito contributed significantly to the war effort and the company enjoyed huge revenue with more than 7000 aircraft produced.

Discussion
This study produced a four-level project success framework that was then applied to several cases.It was also sent to external experts for evaluation.To avoid the possibility of bias by the authors, the external experts were fully responsible for the evaluation.
Evaluation requests were sent to 91 experts.Seven agreed to take part in the process.These experts have experience in PM ranging from 5 to 35 years and all are certified project management professionals (PMPs).They are from the United States (US), India, Norway, Poland, the United Arab Emirates (UAE) and Germany.They have expertise in information technology (IT), telecommunication, construction and management consultation.
The evaluation request focused on two areas: the knowledge contribution of the framework and its possible future development.Regarding the knowledge contribution of the framework, the discussion with the experts concluded that: • The framework helps significantly to achieve a customised success definition for any particular project.It expands the thinking about success beyond the triple constraints.It is adaptable and easy to modify to any particular situation.This feature is particularly important by definition, a project is a unique endeavour and the definition of success should consider this uniqueness.• The framework articulates the context level, which is not well articulated in the existing literature.It shows the impact of the context level on project success.By considering and reviewing the environment and the wider context of a project the project is better understood and better stakeholder identification can be performed.Moreover, a more detailed risk coverage and assessment of the project can be undertaken as a result of the contextual awareness.The framework facilitates a better understanding of the bigger picture.• The framework combines the levels of project success in a new way that adds to the existing literature.Previously, professionals have been aware of these levels particularly the first three and have used them in isolation, but the four levels have not been available for reference as a single group.The framework provides a systematic method by bringing together the various criteria that are in use in different circumstances and facilitates the addition of more criteria to create a more comprehensive set.
In terms of future development, the experts raised the following points: • Change is often encountered in PM practice; consequently, the definition of success may change.For this reason, developing a strategy to respond to change and incorporating it into the framework will increase the usefulness of the framework in practice.• The framework is not a standalone PM methodology; rather, it is a tool to supplement a PM methodology such as Projects in Controlled Environments, version 2 (PRINCE2).Even though the framework can be used separately, further details are required to include it in existing well-established methodologies.In addition, educational material should be developed to demonstrate how to use this framework with certain methodologies.This may increase the usefulness of the framework.• Despite all projects being unique, there is a level of similarity between them in terms of success criteria.For this reason, the framework could be populated with common criteria at each level; then, the relationship between various criteria at different levels and the measurements of these criteria could be explained further.
In this way, users could choose the common criteria from a menu rather than starting from scratch for every project, making the framework a time saving and effort-reducing tool.These points regarding further development are thought provoking and worth consideration.However, they may be addressed in future research, as the scope of this paper is the presentation of the framework and its applications.

Conclusion
The concept of success is vital in the PM practice and research.This research contributes to the topic of project success by providing a four-level project success framework and, in particular, by highlighting the effect of a project's context on the definition of its success.The framework is generic and it is designed to provide a customised success definition for any particular project.Although the framework is new, it has been validated through several applications and external expert evaluations.The authors encourage other PM researchers and practitioners to apply the framework in order to develop it further.

Figure 1 .
Figure 1.The four-level framework