Construction Economics and Building, Vol. 20, No. 4, December 2020
ISSN 2204-9029 | Published by UTS ePRESS | https://epress.lib.uts.edu.au/journals/index.php/AJCEB
RESEARCH ARTICLE
Identifying success criteria at the post-handover stage for international development projects
Johan Fahri1*, Julien Pollack2 and Duro Kolar2
1 Study Program of Management, Universitas Khairun, North Maluku, Indonesia
2 School of Project Management, Faculty of Engineering, University of Sydney, Australia
*Corresponding Author: Johan Fahri. Jl. Yusuf Abdulrahman, Campus II Gambesi, Ternate, North Maluku, Indonesia. Email: johan.fahri@unkhair.ac.id.
Article history: Received 23/06/2020; Revised 30/08/2020 & 24/09/2020; Accepted 17/10/2020; Published 07/12/2020
Citation: Fahri, J., Pollack, J., and Kolar, D. 2020. Identifying success criteria at the post-handover stage for international development projects. Construction Economics and Building, 20:4, 103-119. http://dx.doi.org/10.5130/AJCEB.v20i4.7289
© 2020 by the author(s). This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) License (https://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.
Abstract
The question of how successful a project is beyond the handover stage is still echoing in the literature on project management and more magnified in international development (ID) projects. In addressing the question, this study aims to demonstrate the importance of time frame in assessing project success, particularly identifying the success criteria at the post-handover stages (outcome and impact stages). This study used a qualitative approach that was rooted in interpretivism, allowing the use of constructivist grounded theory method (CGTM) in an ID project as an example. The study identified eight success criteria: Convenience, Development, Documentation, Maintainability, New Capability, Price of Service or Product, Training, and Usability. The study also found that, first, each participant had different emphases on different success criteria; second, most of the identified success criteria were still under the outcome stage; and lastly, usability received the highest responses from the participants as the most concerned criterion at the post-handover stages. This last finding verified the importance of time frame where the other criteria became less important if the outputs were useful to deliver the institutions’ strategic objectives. Future studies that use more sampled participants and different types of institutions are also encouraged.
Keywords:
Success criteria, post-handover stage, international development projects.
Introduction
Over the last 60 years, organizations around the globe have progressively been using projects and programs to realize their organizational strategic objectives (Bredillet, 2010). Massive funds have been invested in implementing those projects and programs. The World Development Index (World Bank, 2015) highlights that 22% of the world’s $78 trillion gross domestic product (GDP) is gross capital formation, which is nearly totally project based. The benefits of this capital formation will be experienced once projects are successfully delivered.
The delivered benefits are still being questioned, particularly beyond the handover stage. Literature has extensively discussed this topic, particularly the success criteria (Atkinson, 1999; Lim and Mohamed, 1999; Sadeh, Dvir and Shenhar, 2000; Patanakul and Milosevic, 2009). However, these models are unable to differentiate the time frame at which success is assessed. Turner and Zolin’s (2012) project success model stands out from these as it emphasizes the time frame by which projects success is defined throughout the output, outcome, and impact stages. During these post-handover stages, project success will be perceived differently by different stakeholders and will be influenced by several factors that have contributed to the realization of project strategic benefits.
Considering those general success criteria models, this topic is considered more challenging in a specific area, including international development (ID) projects, such as Fiscal Reform and Public Financial Management project (FRPFM) in Jordan, Enhancing the Competitiveness of the Croatian Economy—Development of The Investment Climate, in Croatia, and Technical Assistance in Areas of Domestic Market and Industrial Competitiveness, in Egypt. Success criteria beyond the handover stages become challenging because of the intangibility of project objectives and deliverables, complex network of the many stakeholders involved, lack of defined main clients or beneficiaries, significant political risks, demanding local constraints, risky environment, incompatibility of project management tools and techniques, and monitoring and measurement of achieving project objectives (Youker, 1999; Diallo and Thuillier, 2005; Khang and Moe, 2008; Golini and Landoni, 2013). By using an example of ID project, called the I-MHERE (Indonesia-Managing Higher Education for Relevance and Efficiency), this current study aims to identify success criteria at the post-handover stages (outcome and impact stages) in the context of ID projects.
Literature Review
Project Success – An Evolution
In project management literature, the discussion of project success usually covers two elements: success criteria and success factors (Pinto and Slevin, 1987; 1988; Ika, 2009). Success criteria are the measures on which a project’s success is judged. Success factors are essential elements that contribute to the success of the project. More specifically, Ika (2009) defines success criteria as ‘a group of principles or standards used to determine or judge project success’ (p.8). Söderland et al. (2012) view that the success criteria can be used to determine the success or failure of a project. As defined earlier, success factors contribute to the likelihood of project success. Müller and Jugdev (2012) categorize success factors as independent variables, whereas success criteria as the dependent variables that will be used to measure project success.
In understanding success criteria, a study by Ika (2009) demonstrates an evolution about this in three forms. First, project success criteria are defined by the “triangle virtue” (iron triangle) that generally consists of time, cost, and quality. Second, success criteria are perceived as “virtuous square” that consists of time, cost, quality, and client satisfaction. And lastly, success criteria are broadened and covered time, cost, and quality, the realization of the strategic organization, the satisfaction of end users, and the satisfaction of other stakeholders.
An evaluation is also indicated by Müller and Jugdev (2012) who highlight project success development into three periods. Project success between the 1960s and 1980s focuses on the project implementation and handover period and defined success in terms of the iron triangle (i.e. critical success criteria). Research in the 1980s-1990s concentrated on developing critical success factors (CSFs) lists, employing subjective and single case studies instead of a classification scheme or framework. In the more recent period, research in the 1990s-2000s emphasized—as referred by Bredillet (2008) as—the success school, pertaining to the emergence of integrated frameworks on project success.
Although both studies highlight an evolution in comprehending project success, they are viewed as less specific about the success of a project beyond the delivery stage. In other words, time frame should be taken into consideration in assessing project success. For this purpose, this paper specifically reviewed the literature on success criteria by considering the time frame.
Success Criteria as Project Success Models
The reviews were carried on the following five models and indicated three categories on how project success is measured based on time frame point of view. The first category covers a proposed model by Patanakul and Milosevic (2009). They categorize the measurement of project success into three criteria: organizational criteria, including resource productivity and organizational learning; a project’s criteria, consisting of time-to-market and customer satisfaction; and a personal’s criteria, such as personal growth and personal satisfaction. Patanakul and Milosevic’s model is relevant to the discussion as it indicates how success criteria impact organizations. The impact can be experienced beyond the handover stage.
The next model of project success is offered by Sadeh, Dvir and Shenhar (2000). They divide project success into four dimensions: i) meeting design goals; ii) benefit to the end user; iii) the benefit to the developing organization, and iv) benefit of the technological infrastructure of the country and firms involved in the development process. These four dimensions led the model to be categorized into the first one because no specific indication of when the success is measured. Particularly, the delivered outputs do not only meet the requirements when they are handed over, but they should also deliver the benefits to their end users. These benefits are critical to the organizational or institutional development, such as providing basic infrastructure.
Another study concerning the success criteria is also suggested by Lim and Mohamed (1999) who group the criteria into micro and macro viewpoints. Micro viewpoint consists of completion time, completion cost, completion quality, completion performance, and completion safety. In a broader view, macro-viewpoints cover completion time, completion satisfaction, completion utility, and completion operation. These viewpoints can be viewed as delivery and post-handover stages and indicated a relevance to this paper’s focus where project success occurs beyond the delivery stage.
However, the three models above are perceived insufficient in providing more elaborations on how the time frame is considered to assess project success. As a result, this review groups the models into the first category of project success, considering them as unclear explanations on when the success is measured. Once the outputs of the project start to be used, the success criteria can be perceived differently over the period.
The discussion also defines the second category which separates success into deliver/handover stage and post-delivery/post-handover stage. A seminal project success model by Atkinson (1999) indicates project success that considers the time frame, distinguishing between project success at the delivery and beyond the delivery stage. According to Atkinson (1999), the criteria at the delivery stage still propagate the iron triangle (cost, time, and quality). Beyond it, Atkinson (1999) emphasizes that the criteria broadened the scope by including information system, organizational benefits, and community benefits. In other words, Atkinson’s (1999) model takes into consideration the entire project life cycle and beyond for continuous assessment, because the ‘real’ project success should be measured at the post-handover stage.
In practice, the definitions of the post-handover stage are still ambiguous. This ambiguity has resulted from the time frame that is used to measure long-term success of a project. Additionally, some studies generalize post-handover stage as equal to a longer-term stage (MacDonald et al., 2006; White, 2006; Manzoor Arain and Sui Pheng, 2007; Patton, 2010; Taye, 2013). The clarification of the stages (terms) is a crucial link to the project success model and its detail criteria. In other words, Atkinson’s (1999) and those studies’ divisions of project stages generalize the post-handover stage. Clarifications of this stage are still required.
Due to this need, the study also reviews another model by Turner and Zolin (2012), which clarifies the post-handover stages by dividing them into project outcome stage and impact stage. The outcome stage is defined as the stage where ‘new capabilities that the operation of the new asset gives to the investing organization’ (Turner and Zolin, 2012, p.90). Meanwhile, impact stage is considered to be ‘[t]he long-term performance improvement that it is expected the new capabilities will enable the parent organization to achieve’ (Turner and Zolin, 2012, p.90). This clarification leads this model as the third category and detail the criteria as summarized in Table 1, summarizing dimensions related to project success according to stage categorization. Their model demonstrates that project success is perceived differently by different stakeholders at the different time frame.
The review then concludes three types of category for project success model, as illustrated in Figure 1. Category 1 covers the models that have an undefined time frame. Next, models under Category 2 start to differentiate between the handover stage and the post-handover stage. However, the post-handover stage can still not be differentiated more. The last category has a clearer differentiation of post-handover stages: the outcome and impact stages.
Figure 1 Categories of Project Success Model
To conclude, the basic division for these three categories lies in the time frame where the success of a project is measured. The time frame is one of the emphasized aspects of the evolution of the project success model. Because of the time frame, different stakeholders can define a project success differently at different stages throughout a project’s outputs lifecycle. Based on this context, this thesis views that studies have paid limited attention to the importance of time frame in assessing project success. Time frame is seen as a period from start to finish of a project, as well as how the expected goals are achieved. Importantly, the focus on the time frame will allow a clearer differentiation between project management success and project success. This differentiation can also cover a clear distinction between success criteria and success factors of both, including under the category of certain types of project, such as international development (ID) projects.
Project Success Criteria for International Development (ID) Projects
This study then reviews congruence of Turner and Zolin success model into a more specific context, including International Development (ID) Projects. This type of project is perceived to possess their own characteristics. These include intangibility of project objectives and deliverables, complex network of many stakeholders involved, lack of defined main clients or beneficiaries, significant political risks, demanding local constraints, risky environment, incompatibility of project management tools and techniques, and monitor and measurement of achieving project objectives (Youker, 1999; Diallo and Thuillier, 2005; Khang and Moe, 2008; Golini and Landoni, 2013). Moreover, Khang and Moe (2008, p.74) argue that ID projects mostly concentrate on poverty alleviation and living standards improvement, environment protection, basic human rights protection, assistance for victims of natural or people-caused disasters, capacity building and development of basic physical and social infrastructures.
Considering ID projects’ characteristics and how to measure their success, the criteria in Table 1 cannot be entirely used because the criteria indicate the measurement of mostly commercial and business performance project. Other studies have indicated a lack in the literature regarding success criteria of ID Projects (Palmer, 1986; Youker, 1999; Themistocleous and Wearne, 2000; Kwak and Dewan, 2001; Kwak, 2003; Muriithi and Crawford, 2003; Diallo and Thuillier, 2005). These success criteria should be developed throughout project lifecycle (Ahsan and Gunawan, 2010).
A study by Khang and Moe (2008) identifies success criteria of ID projects for every step of a project lifecycle. They align success criteria according to the PMBOK (Project Management Body of Knowledge) Guide’s project lifecycle. In other words, success criteria are different in different phases of the project. They group the criteria into four phases, as follows:
- Phase I – Conceptualizing. Consisting of addressing relevant needs of the right target group of beneficiaries; identifying the right implementing agency capable and willing to deliver, and matching policy priorities and raising the interests of key stakeholders.
- Phase II – Planning. Success criteria are approval of, and commitment to, the project by the key parties; sufficient resources committed and ready to be disbursed; and core organizational capacity established for PM.
- Phase III – Implementing. Comprising of resources mobilized and used as planned; activities carried out as scheduled; outputs produced to meet the planned specifications and quality; and good accountability of resources utilization.
- Phase IV, Closing or Completing. Success criteria are projected assets transferred, financial settlements completed, and team dissolved to the satisfaction of key stakeholders; project end outputs are accepted and used by target beneficiaries, and project completion report accepted by the key stakeholders.
Although Khang and Moe’s study covers four phases, they are considered to be used up to the handover stage. By using Figure 1 above, a more comprehensive model can be obtained, where, for ID projects, success criteria should be evaluated using Category 3, considering their multi-level stakeholder institutions and are interested in the criteria at different stages. This different interest is illustrated in Figure 2.
Figure 2 Success Model at Different Stages by Different Stakeholders for ID Projects
The figure above emphasizes the discussions earlier. The success achievements of an ID project are the most concerned elements for the project sponsor and the implementing agency. However, this concern is limited to the handover stage. As mentioned earlier, because of high political interests both institutions more focus on the delivery of the project’s outputs in terms of satisfying the financial accountability as one of the success criteria. Once the project hands over its outputs, the success will be at the concerns of the main beneficiaries. These institutions need to ensure that the outputs can be adapted and adopted as the new resources to realize intermediate and long-term strategic objectives. While satisfying financial accountability is one of project management success element—cost, realizing benefits is the ‘real’ project success. Unfortunately, a wider gap is more obvious in the literature to explore project success for ID projects. This gap leads this study’s aim that identifies success criteria at the outcome and the impact stages.
Methodology
The focus of this research was at an ID project called I-MHERE Project (here in called ‘the Project’). The Project was funded by a World Bank’s loan with the total of US$ 80 million and run from 2009 to 2011 in support of Indonesia’s Higher Education Long Term Strategy (HELTS). The project run in 65 state higher education institutions (HEIs) across the country. This current research was aimed at identifying success criteria at the post-handover stages using the model above (Figure 2).
In exploring the area, this study used a qualitative approach rooted from interpretivist paradigm that allowed the use of constructivism in grounded theory. The paradigm aligned with the discussions from the literature earlier that acknowledges different definitions by different stakeholders at different stages. Table 2 summarizes basic assumption in using interpretivist paradigm. The table also indicates the use of constructivism to construct reality in the form of defining success criteria by the users.
| Paradigm | Goals | Theoretical Concerns |
|---|---|---|
| Interpretivist—a more subjectivist view, also with an apparent concern with regulation, or at least a lack of concern with changing the status quo. | To describe and explain in order to diagnose and understand | Social Construction of Reality Reification Process Interpretation |
The constructing process was also grounded from the users (the participants) experiences in using the delivered outputs and the knowledge about their institution. For this aim, grounded theory was chosen as the basic theory for the approach, and in combination with constructivism paradigm, this study used Charmaz (2014) Constructivist Grounded Theory, as the main method. The process of Constructivist Grounded Theory Method (CGTM) consisted of substantive (initial and focused) coding, theoretical coding, and concept generating. Most of the coding processes heavily relied on NVivo qualitative software.
Furthermore, the methodology consisted of two main aspects: data collecting and data analysis. Secondary data, importantly Implementation Completion Report (ICR) published by the World Bank was first gathered to target the potential participating institutions (HEIs, the former Project recipients). By using the ICR, the higher number of HEIs fell under the category of those that used the funding to strengthen their institutional management system. These 28 were then scrutinized and selected based on their financial and physical performances, which resulted in 13 potential participating HEIs and later formally and informally approached. This effort resulted in two HEIs as the participating institutions. From these HEIs, eighteen participants agreed to participate, although two of them refused the interviews to be recorded. The selected participants are highlighted in Figure 3. These participants were mainly selected based on the Project’s outputs that were currently being used. The participants were labeled in four digits numeric. First digit—1 for Site 1 and 2 for Site 2; Second digit—1 for middle management and 2 for lower management; and last-two digit—a sequence of the interview. The selected participants were also considered their involvement of the users in the project—9 pariticpants were consutlted by the IS developer and the Project team; and 7 were purely users after the Project outputs were handed over. The semi-structured interview techniques used in this study based on the most convenient time of the participants. The main questions of the interview were “Are you still using the output produced the I-MHERE project? Why [or Why not?]”?
Figure 3 Participants’ Demography
The transcripts from the sixteen interviewees were the main source of the primary data. These transcripts were coded throughout the initial and focused coding processes. Extant literature, especially related to project success, were used for theoretical coding. As the transcripts were in Indonesia language, since the initial coding, the phrases had been translated into English, as the target language. Theoretical coding was used to identify the success criteria.
Figure 4 Coding Processes (Adapted from Charmaz (2014, p.18))
In conducting the processes, this study also carefully focused on reliability and validity aspects of qualitative research. In terms of reliability, this study referred to Long and Johnson’s (2000, pp.30-31) three types of test for reliability: stability, consistency, and equivalence. For the validity aspect, this study used the method from Whittemore, Chase and Mandle (2001), covering the validity in designing research setting, generating, analysing, and presenting data.
Findings
The findings of this study were focused on three essential aspects. First, the findings indicate different emphasizes on the identified success criteria by the participants (the users) at the outcome and the impact stages. Second, the analyses showed distributions of the success criteria under both stages (outcome and impact); and lastly, the findings demonstrate the overall tendency of the responses provided by the participants in identifying the success criteria.
Different Emphasizes on Identified Success Criteria
Through using the theoretical coding, the findings suggest that at the post-handover stages, the participants, as the users of the delivered outputs, identified eight main success criteria—Convenience, Development, Documentation, Maintainability, New Capability, Price of Service or Product, Training, and Usability. Early seminal studies (Atkinson, 1999; Cooke-Davies, 2002; Turner & Zolin, 2012) allowed the theoretical coding to form these theme as success criteria, differentiated these criteria from the critical factors, which referred from other studies (Diallo & Thuillier, 2005; Dong et al., 2009; Hermano et al., 2013; Ram et al., 2013; Struyk, 2007; Veiga et al., 2014). This process is summarized in Figure 4 above.
The analyses showed different emphasized of the success criteria by different participants (users). By using NVivo software, the analyses generates Figure 4 that shows three participants—1203, 1210, and 2103—have the highest responses compared to other participants. Interestingly, they emphasized the different level of criterion and were from two different managerial levels. While 1203 and 1210, who were at the lower managerial level, responded more on convenience and price of service or product, 2103, who was a middle manager, was more concerned about maintainability. Meanwhile, other participants (2206, 1101, and 1104) provided very low responses that were shaped identifying the criteria.
Figure 5 Variations of Responses by Participants
Distributions of Identified Success Criteria
To examine the distribution of the criteria under two post-handover stages, Turner and Zolin project success model was used. Based on this model (Table 1), the identified success criteria were specifically referred to two types of project stakeholders: consumers and operators/users. This study viewed that while consumers were not necessarily the directs users and could be the middle managers, operators/users were the direct users of the I-MHERE project’s outputs and they tended to be from lower managerial level. While outcome stage reflects the period where the success is measured at the handover stage, outcome and impact stages are the phases where the success is measured at the post-handover stages or beyond the delivery or output stage. However, the findings suggest that the identified factors were distributed under both stages (Table 3).
The findings suggested that although most participants were middle managers, the success criteria were mostly identified by those of lower managerial levels. As stated earlier, the middle managers were more considered as consumers than operators or direct users, who were usually from the lower managerial level. Importantly, most of the criteria were identified under the outcome stage, as summarized in Table 3. This finding suggests that the delivered outputs were considered still at the transition level and fully delivered the benefits in the long-term. The table also shows that new capability, the price of service or product, and development were identified and grouped under the impact stage. Because of these findings, this study tended to perceive the stage beyond the delivery more as post-handover stages than the post-handover stage.
Overall Emphasizes of Identified Success Criteria
Earlier analyses provided evidence that the participants emphasized differently on each success criterion and most of the criteria were found more at the outcome stage than the impact stage. The analyses were also conducted to highlight which of these criteria was the most concerned criteria by the participants at the post-handover stages. The analysis then generated Figure 5 and found that the participants, as the users of the outputs of the I-MHERE project, were more concerned about usability, followed by development, documentation, new capability, training, maintainability, convenience, and price or service or product.
Figure 6 Level of Responses: Identified Success Criteria
Discussions
The findings above elaborated the difference level of responses for each identified criterion, how the criteria were distributed at different stages, and the overall responses from the participants (the users). The discussions are started by focusing on the difference between the level of responses, particularly in identifying usability and convenience. These two criteria have been extensively discussed in the literature, particularly, in the area of an information system or information technology, such as Davis, Bagozzi and Warshaw (1992) with their Technology Acceptance Model (TAM) and TAM 2 by Venkatesh and Davis (2000). While both models used the term ‘ease of use’ in TAM or ‘perceived ease of use (PEU)’ and ‘perceived usefulness (PU)’ in TAM 2 and, this current study used convenience and usability instead, referring to Turner and Zolin’s project success model earlier. Importantly, while in both TAM models, convenience leads to usability, this current study found that at the post-handover stages, the participants were more concerned about usability than convenience.
To explain this tendency, this current study referred to a Badewi et al.’s (2013) study that suggests a separation between System Acceptance and System Success theories. This separation is in line with this current study where system acceptance theories were used explain the project output stage, whereas system success was used to explain the project’s post-handover stages (outcome and impact stages). Figure 6 then illustrates this context.
Figure 7 Focused Process (Reproduced from Badewi, Shehab and Peppard (2013, p.227))
This study considered that up to ‘(2) intention to use’ was the output stage, considering that “[t]he new asset delivered by the project, commissioned at the end of the project” (Turner and Zolin, 2012, p.90). This condition also aligned with System Acceptance, assuming that the outputs had been tested and ready for the implementation. The focus to explain the different level of responses occurred between ‘(2) use’ and ‘(2) user satisfaction’. From this point on, this study considered that the outputs were ready to demonstrate their net benefits. In other words, this was the beginning of the outcome stage.
Because of this stage, a criterion for user satisfaction could be either convenience or usability or both. User satisfaction also implies user acceptance, and according to Venkatesh and Davis (2000), in TAM 2 usability leads to an indirect relationship between convenience and user satisfaction. This relationship assumes that because the users ‘use’ the delivered outputs, they experience the convenience of the outputs. Once they are convenient, they then judge how useful the outputs are. When these two are satisfied, user satisfaction can be achieved.
However, because the I-MHERE’s outputs had been used for some time, the using period—the time frame from output to outcome stages—indicated that the users had moved from the concern about convenience and paid more attention to whether the outputs were useful. The usability of the outputs then became the criterion for user satisfaction before the outputs delivered their net benefits. This condition is illustrated in .
Usability was more concerned by the users than convenience at the outcome stage. In other words, the users (the participants) were more concerned about how useful the delivered outputs to assist them in performing their daily tasks. Satisfying these two—convenience and usability—led to user satisfaction, both to direct and indirect users. When they satisfied, the delivered outputs should start demonstrating their net benefits.
In particular, Figure 8 also implies the reasons for the distributed criteria that most under the outcome stage. Earlier, the outcome stage is defined as the stage where ‘new capabilities that the operation of the new asset gives to the investing organization’ (Turner and Zolin, 2012, p.90). The funding suggested that the capabilities were still questioned or concerned by the users, specifically in term of the outputs’ usability. Once the outputs were useful, both direct and indirect users were satisfied, and then the new capabilities and other benefits could be demonstrated. In other words, this was a crucial point of the delivered outputs to demonstrate their usability. In short, at the outcome stage, it was a crucial point of the deliverables to be adopted and adapted into new resources of the institutions to deliver the strategic objectives. At this point net benefits—as shown in Figure 8—did not only cover new capability, but also ensuring the long term affordability of a produced output in the market as well as how the outputs would be developed, as shown in Table 3. This table is also used to refer how the identified success criteria were assigned under the outcome stage and the impact stage. Table 3, Figure 6, and Figure 7 led to the illustration shown in Figure 8 where net benefits were achieved at the impact stage.
Figure 8 Different Concern of Criteria
Furthermore, the discussions above also implied the findings on the different level of responses of each criterion by each participant (a user). Earlier, the current study showed that a participant, for example, 1203—more concerned about convenience than other criteria. This participant was a direct user of software for inventory management at Site 1. Because of directly using an output, tangible aspect of the output became essential for user satisfaction. Islam et al. (2012, p.6453) found that “user satisfaction is positively related to convenience”.
It was also interesting to note how participants also concerned about the price of service or product. This criterion was identified under the outcome and the impact stages (see Table 3). Appearing under both stages indicated that the participants’ concern about the delivered outputs’ associated costs in mid- and long-term. It has been the basic customer behavior toward prices and costs (Cohen and Whang, 1997; Watanapa and Techanitisawad, 2005). For example, some participants implied that the top management had less priority on the IT investment and tended to look for other affordable options. This tendency was because of the lack of knowledge of an early investment for IT (Ragatz, Handfield and Petersen, 2002). This tendency included anticipation for database management. When the institution grows, database management capacity should also follow (Halaweh, 2013). However, the overall finding indicated a concluding perception that at the post-handover stages, both convenience and price of service or product became less concern as long as the outputs were useful to deliver the institutions’ strategic objectives.
Conclusions
This current study has demonstrated that the importance of time frame in assessing project success, particularly identifying the success criteria. The time frame highlighted a clearer separation of outcome and impact stages. At these post-handover stages, eight success criteria were identified—Convenience, Development, Documentation, Maintainability, New Capability, Price of Service or Product, Training, and Usability. Each of these received a different level of responses from each participant.
As well, most of these criteria were identified under the outcome stage; and a few were at the impact stage. Overall, most participants more concerned about the usability of the outputs. Although some participants provided more responses on convenience and price of service or product, at the post-handover stages, these criteria became less important if the outputs were useful to deliver the institutions’ strategic objectives. The tendency was important in the area of ID projects where, first, the project sponsor and the implementing agency limited their assessment of project success up to the handover or out stage; and second, the main beneficiaries are responsible to ensure the delivery of long-term project’s objectives.
This study has also limitations, particularly the limited numbers of participants and the participating institutions and their identical types. Future studies can generate more solid findings by involving more participant and sampled institutions or organizations from different sectors that received foreign loans to produce their new resources. This is expected to increase the level of external generalizability.
Acknowledgment
This study was funded by Australian Awards Scholarships from Department of Foreign Affairs and Trade, the Government of Australia and the Provincial Government of North Maluku, Indonesia, through Kieraha Scholarship.
References
Ahsan, K. and Gunawan, I., 2010. Analysis of cost and schedule performance of international development projects. International Journal of Project Management, 28(1), pp.68-78. https://doi.org/10.1016/j.ijproman.2009.03.005
Atkinson, R., 1999. Project management: cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria. International Journal of Project Management, 17(6), pp.337-42. https://doi.org/10.1016/s0263-7863(98)00069-6
Badewi, A., Shehab, E. and Peppard, J., 2013, Cranfield, UK.
Bredillet, C.N., 2008. From the Editor. Project Management Journal, 39(1), pp.2-6.
Bredillet, C.N., 2010. Blowing hot and cold on project management. Project Management Journal, 41(3), pp.4-20. https://doi.org/10.1002/pmj.20179
Charmaz, K., 2014. Constructing grounded theory. 2nd ed. London: Sage Publications Ltd.
Cohen, M.A. and Whang, S., 1997. Competing in Product and Service: A Product Life-Cycle Model. Management science, 43(4), pp.535-45. https://doi.org/10.1287/mnsc.43.4.535
Davis, F.D., Bagozzi, R.P. and Warshaw, P.R., 1992. Extrinsic and Intrinsic Motivation to Use Computers in the Workplace1. Journal of Applied Social Psychology, 22(14), pp.1111-32. https://doi.org/10.1111/j.1559-1816.1992.tb00945.x
Diallo, A. and Thuillier, D., 2005. The success of international development projects, trust and communication: an African perspective. International Journal of Project Management, 23(3), pp.237-52. https://doi.org/10.1016/j.ijproman.2004.10.002
Gioia, D.A. and Pitre, E., 1990. Multiparadigm Perspectives on Theory Building. The Academy of Management Review, 15(4), pp.584-602. https://doi.org/10.5465/amr.1990.4310758
Golini, R. and Landoni, P., 2013. International development projects: peculiarities and managerial approaches. [online] Available at: http://www.pmi.org/~/media/PDF/Academic/summaries/International%20Development%20Projects.ashx [Accessed 07 September 2015].
Halaweh, M., 2013. Emerging Technology: What is it? Journal of Technology Management & Innovation, 8(3), pp.108-15.
Ika, L.A., 2009. Project success as a topic in project management journals. Project Management Journal, 40(4), pp.6-19. https://doi.org/10.1002/pmj.20137
Islam, M.A., Yusuf, D.H.M., Yusoff, W.S. and Johari, A.N.B., 2012. Factors affecting user satisfaction in the Malaysian income tax e-filing system. African Journal of Business Management, 6(21), pp.6447-55.
Khang, D.B. and Moe, T.L., 2008. Success criteria and factors for international development projects: A life‐cycle‐based framework. Project Management Journal, 39(1), pp.72-84. https://doi.org/10.1002/pmj.20034
Kwak, Y.H., 2003. Critical Success Factors in International Development Project Management In B.O. Uwakwhe and I.A. Minkarah eds. The organization and management of construction 10th International Symposium, Construction Innovation and Global Competitiveness: CRC Press. 358.
Kwak, Y.H. and Dewan, S., 2001. Risk management in international development projects. In: Proceedings of the Project Management Institute Annual Seminars & Symposium, November 1–10, 2001. Nashville, Tenn., USA.
Lim, C.S. and Mohamed, M.Z., 1999. Criteria of project success: an exploratory re-examination. International Journal of Project Management, 17(4), pp.243-48. https://doi.org/10.1016/s0263-7863(98)00040-4
Long, T. and Johnson, M., 2000. Rigour, reliability and validity in qualitative research. Clinical Effectiveness in Nursing, 4(1), pp.30-37. https://doi.org/10.1054/cein.2000.0106
MacDonald, G., Garcia, D., Zaza, S., Schooley, M., Compton, D., Bryant, T., Bagnol, L., Edgerly, C., & Haverkate, R. (2006). Steps to a HealthierUS Cooperative Agreement Program: foundational elements for program evaluation planning, implementation, and use of findings. Preventing chronic disease, 3(1), A19.
Manzoor Arain, F. and Sui Pheng, L., 2007. Modeling for management of variations in building projects. Engineering, Construction and Architectural Management, 14(5), pp.420-33. https://doi.org/10.1108/09699980710780737
Müller, R. and Jugdev, K., 2012. Critical success factors in projects: Pinto, Slevin, and Prescott – the elucidation of project success. International Journal of Managing Projects in Business, 5(4), pp.757-75. https://doi.org/10.1108/17538371211269040
Muriithi, N. and Crawford, L., 2003. Approaches to project management in Africa: implications for international development projects. International Journal of Project Management, 21(5), pp.309-19. https://doi.org/10.1016/s0263-7863(02)00048-0
Palmer, C.F., 1986. Introduction of special public works projects to Sudan. International Journal of Project Management, 4(4), pp.223-29. https://doi.org/10.1016/0263-7863(86)90007-4
Patanakul, P. and Milosevic, D., 2009. The effectiveness in managing a group of multiple projects: Factors of influence and measurement criteria. International Journal of Project Management, 27(3), pp.216-33. https://doi.org/10.1016/j.ijproman.2008.03.001
Patton, M.Q., 2010. Future trends in evaluation. In: UNICEF. From policies to result: developing capacities for country monitoring and evaluation systems. New York: UNICEF.
Pinto, J.K. and Slevin, D.P., 1987. Critical factors in successful project implementation. Engineering Management, IEEE Transactions on(1), pp.22-27. https://doi.org/10.1109/tem.1987.6498856
Pinto, J.K. and Slevin, D.P., 1988. Critical success factors across the project life cycle. PA, USA: Project Management Institute.
Ragatz, G.L., Handfield, R.B. and Petersen, K.J., 2002. Benefits associated with supplier integration into new product development under conditions of technology uncertainty. Journal of Business Research, 55(5), pp.389-400. https://doi.org/10.1016/s0148-2963(00)00158-2
Sadeh, A., Dvir, D. and Shenhar, A., 2000. The role of contract type in the success of R&D defense projects under increasing uncertainty. Project Management Journal(31), pp.14-22. https://doi.org/10.1177/875697280003100303
Söderland, J., Geraldi, J., Müller, R. and Jugdev, K., 2012. Critical success factors in projects: Pinto, Slevin, and Prescott-the elucidation of project success. International Journal of Managing Projects in Business, 5(4), pp.757-75.
Taye, H., 2013. Evaluating the impact of agricultural extension programmes in sub-Saharan Africa: Challenges and prospects. African Evaluation Journal, 1(1), pp.1-9. https://doi.org/10.4102/aej.v1i1.19
Themistocleous, G. and Wearne, S.H., 2000. Project management topic coverage in journals. International Journal of Project Management, 18(1), pp.7-11. https://doi.org/10.1016/s0263-7863(99)00030-7
Turner, R. and Zolin, R., 2012. Forecasting success on large projects: developing reliable scales to predict multiple perspectives by multiple stakeholders over multiple time frames. Project Management Journal, 43(5), pp.87-99. https://doi.org/10.1002/pmj.21289
Venkatesh, V. and Davis, F.D., 2000. A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management science, 46(2), pp.186-204. https://doi.org/10.1287/mnsc.46.2.186.11926
Watanapa, B. and Techanitisawad, A., 2005. Simultaneous price and due date settings for multiple customer classes. European Journal of Operational Research, 166(2), pp.351-68. https://doi.org/10.1016/j.ejor.2004.02.011
White, H., 2006. Impact evaluation: the experience of the Independent Evaluation Group of the World Bank. [online] Available at: https://mpra.ub.uni-muenchen.de/1111/
Whittemore, R., Chase, S.K. and Mandle, C.L., 2001. Validity in Qualitative Research. Qualitative Health Research, 11(4), pp.522-37. https://doi.org/10.1177/104973201129119299
World Bank, 2015. World Development Indicators [online]. Available at: http://wdi.worldbank.org/table/4.8#.
Youker, R., 1999. Managing international development projects-lessons learned. PM World Journal, IV(II), pp.1-3.