Efficiency Analysis of Project Management Offices for Large-scale Information System Projects : Insights for Construction Megaprojects

In this study, the efficiencies of Project Management Offices (PMOs) in large-scale information system (IS) projects are addressed by using data envelopment analysis. Moreover, the potential improvement levels for each input and output factors of inefficient PMOs are examined. The effects of performance levels of PMO functions on project outcomes with respect to efficiency levels are also analyzed. A total of forty-nine PMOs are analyzed for this study. The result shows that twenty-four PMOs are found to be efficient. As a result of analyzing the impact of efficiency on project performance depending on the functional levels of PMOs, those groups with a high degree of efficiency show higher outcomes compared with the groups with a low degree of efficiency regardless of the functional levels of PMOs. Furthermore, the gap in outcome between the groups with a high degree of efficiency and the groups with a low degree of efficiency is maintained at almost the same level, regardless of the functional levels of PMOs, with the exception of the case of practice management. This indicates that even those groups with a low degree of efficiency could expect high outcomes in terms of schedule and cost compliance if their level of practice management is high.


Introduction
Corporations today are making a lot of effort to improve their competitiveness by constructing information systems (IS) in a rapidly changing business environment.Therefore, they have increased their investment in information systems every year.However, a large number of these IS projects are causing many serious problems such as cost overruns and schedule delays.Subsequently, they are unable to properly achieve the intended outcomes.Information systems are an integrated set of components for collecting, storing, and processing data and for delivering information and knowledge (Encyclopaedia Britannica, 2015).The enterprise resource planning system (ERP), supply chain management system (SCM), and customer relationship management system (CRM) are examples of IS projects.
The Standish Consulting Group (2013) surveyed and published successful project cases in the CHAOS Manifesto 2013 Report.The success of projects was defined based on time compliance, cost compliance and how well the requirements were fulfilled.According to this report, the portion of successful projects had increased from 29 percent in 2004 to 39 percent in 2012.However, 61 percent of all those projects were still deemed as failed projects or projects in progress.Moreover, this report indicated that those challenged projects encountered problems such as cost overruns and schedule delays.As for the number of cases associated with cost overruns, it had increased from 71 percent in 2010 to 79 percent in 2012.In the case of schedule delays, it had increased from 46 percent in 2010 to 59 percent in 2012.According to the Korean software engineering white paper (National IT Industry Promotion Agency of South Korea, 2013), cost compliance rate was 48.3 percent for a total of 203 surveyed projects.Thus, more than half the projects had exceeded the initially planned budget.Among the 105 projects with cost overruns, those projects that exceeded cost by 1 to 10 percent accounted for the highest proportion (84 projects), followed by those projects that exceeded cost by 11 to 20 percent (15 projects) and those projects that exceeded by 20 percent or higher (6 projects).Out of a total of 234 projects, 61 (26.1 percent) projects did not meet the delivery dates.In contrast, the number of the projects having complied with delivery date was 173 (73.9 percent).Among the 61 projects that exceeded the delivery date, those that exceeded by 1 to 10 percent accounted for the highest portion (28 projects), followed by the projects that exceeded 11 to 20 percent (24 projects) and the smallest portion was projects that exceeded 21 percent or more (9 projects).This implies that we need to take some approaches to increase the project success rate while reducing the failure rate and, in the case of large-scale projects, many corporations at home and abroad are implementing PMO as an alternative to them.According to a survey conducted by PM Solutions (2010), 48 percent of all corporations had implemented PMO in 2000.Then, this number increased from 77 percent in 2006 to 88 percent in 2010.
PMO has recently emerged as a new issue and a growing interest is also shown in the domestic project management field.Kim (2013) argued in his study on the Korean software industry that PMO would have a significant impact on project performance in terms of scope, schedule, and integrated management.Moreover, Business Improvement Architects (BIA, 2005) reported that the project success rate could increase by 37 percent within 1 year of implementing PMO, by 62 percent within 2 years and by 65 percent within 5 years.Therefore, BIA (2005) was confident that a project could be successfully managed through PMO operation.However, the aforementioned success rate focused only on output.It did not consider how much input or resources they invested to get those outputs.For this reason it is important to study the efficiency of PMOs in conducting large-scale IS projects.Regarding the scale of the project, Bloch, Blumberg and Laartz (2012) defined an information technology (IT) project at a scale of US$15M or more, as a mega IT project.In Korea, IS projects of over 1 billion Won (or US$1M) are often regarded as large scale projects and we will use this definition throughout this study.As for the IS projects used in this study, the sizes of the organizations, based on the number of people (P), are as follows: 53.1% (P < 500), 10.2% (500 ≤ P < 1,000), 16.3% (1,000 ≤ P < 5,000), 6.1% (5,000 ≤ P < 10,000), and 14.3% (P ≥ 10,000).As for the project budget (B), 55.1% (US$1M ≤ B < US$3M), 8.2% (US$3M ≤ B < US$5M), 36.7% (B ≥ US$5M).In terms of years in business(Y), the statistic shows 61.2% (Y < 1yr.), 20.4% (1yr.≤ Y < 2 yrs.), 8.2% (2yrs.≤Y < 3 yrs.),and 10.2% (Y ≥ 3yrs.)respectively.
Even though the number of corporations and public institutions implementing PMO is on the rise every year, and there is a growing importance in efficiency analysis for PMO, there has not yet been any Korean study that focused on this context.Therefore, this study on the efficiency of PMO should be of great significance.The relative efficiency of PMO would have greater significance since those relatively inefficient PMOs could improve efficiency by benchmarking those PMOs that are relatively efficient.Moreover, it would be possible for corporations to increase the level of PMO maturity while improving project success rate through efficiency improvement.Therefore, this study aims to examine the relative efficiencies of PMOs by leveraging the data envelopment analysis methodology and also proposes directions and amounts of improvement required to become efficient PMOs.

Implications for Construction Projects
In this study, the relative efficiencies of PMOs and the impacts of efficiencies on project performances according to the levels of PMO functions focused on large-scale IS projects are analyzed.Even though construction projects have different application fields from IS projects, they have two common aspects as followings.First, they both have problems in cost and schedule compliance.Second, in order to remedy these problems, they adopted PMOs.Thus, the DEA results and the methodology of this study might be utilized for construction projects.

Theoretical Background Data Envelopment Analysis (DEA) Model
The analysis of the efficiency of PMO (the subject of evaluation in this study) is conducted using the DEA model.The DEA is the nonparametric technique developed for efficiency evaluation.It has been widely utilized in various fields including schools, banks, hospitals and public institutions.The DEA model, first proposed by Charnes, Cooper and Rhodes (1978), was based on the relative efficiency concept of Farrell (1957).The model first identifies the empirical efficiency frontier from the set constructed by observed values of input and output factors of decision making units (DMUs) and then compares them with those of the other DMUs to measure the relative efficiency of those evaluated (Han and Kim, 2008;Park and Yoo, 2013;Koh and Kim, 2014).A DMU refers to the organizational unit which is a subject undergone evaluation and examples of DMUs are such banks, schools, retail stores, and hospitals.In this study, an individual PMO corresponds to a DMU.
There are various models in DEA depending on the purpose of study and the nature of the data.Among them, the CCR (Charnes, Cooper and Rhodes, 1978) model and BCC (Banker, Charnes and Cooper, 1984) models are widely used.CCR model is designed to evaluate other DMUs based on the most efficiently operated DMU without considering economies of scale for DMU.Therefore, CCR model is called CRS (constant returns to scale) model.On the other hand, BCC model considers economies or diseconomies of scale.Thus, BCC model is called VRS (variable returns to scale) model (Baek and Noh, 2013;Joo and Kim, 2014).These CCR and BCC models are divided into input based model and output based model depending on their purposes.The efficiency in input based model is improved by reducing the input level while holding the output level fixed.For the case of output based model, the efficiency is enhanced as the output levels increase while the input levels are fixed.
Assume that there are seven DMUs (A~G) and each of them has one input (X) and one output (Y) as shown in Table 1.In the CRS model, the frontier of efficiency is made by DMU D (efficiency = 1) where the ratio of output to input is the highest as shown in Figure 1.It means that the efficiencies of the remaining DMUs can be obtained based on that of DMU D. For instance, for the case of the output based model, the projection of C onto the CRS frontier line is C 1 .Thus, the efficiency of DMU C becomes (= 8.4/4).In the input based model, the input level of DMU C which has the current efficiency of (= 3.3/7) can be reduced down to having the efficiency of 1.
In the VRS model, the efficiency frontiers are made by B, D, F, and G (efficiency = 1).Accordingly, in the output based model, the output level of DMU C which has the current efficiency of (= 7/4) can be increased to which is on the efficiency frontier line of VRS.For the case of input based model, the current efficiency of DMU C is .Assuming that the number of DMUs to be analyzed is J and, for each , there are M input factors and N output factors .Then the efficiency of can be obtained through the output-based CCR and BCC models represented below.

Output-Based CCR Model
Output-Based BCC Model The CRS (or CCR) model mentioned above is at the optimal scale efficiency state since it is at constant returns to scale.The VRS (or BCC) model, on the other hand, is at the optimal technical efficiency state under the condition of variable returns to scale.Therefore, it is possible to obtain thescale efficiency by dividing the result from the CCR model by that of the BCC model.In cases where the BCC model is not optimal scale-wise, it can be determined whether the variables are at the state of 'economy of scale' or 'diseconomy of scale'.
Meanwhile, Figure 1 shows that DMU D is efficient from the perspective of CRS (or CCR) and has the higher production rate than any other DMUs.On the other hand, from the perspective of VRS, the DMUs on the left side of show higher productivity than that of DMU D. So, for the DMUs on the left side of , it is better to increase the scale until it reaches .The DMUs on the right side of show the lower productivity than that of DMU D.

Project Management Office (PMO)
PMO is an independent organizational entity having full-time staff to provide managerial and administrative services, education and training, consultation and technical services to project teams (Kwak and Dai, 2000).It is also an organization that improves and develops project management capability (Koo, Kwon and Kim, 2006).Adopting a PMO means making the company's project management methodology official (Bates, 1998).
In addition, the Project Management Institute (PMI, 2008) defined PMO as an organization that integrates all projects in progress and manages the project portfolio within a corporation.Its roles include deployment of project methodologies and processes/procedures to control, provide tools, allocate human resources, and conduct education and training.Other prior studies and scholars have also emphasized various functions and roles of PMO.Berry and Parasuraman (1991) highlighted quality control as a role of PMO in their book.Bates (1998) argued that it would be possible to acquire the true effect of adopting a PMO only by expanding its roles to project risk evaluation, project outcome assessment and organizational change management.In a study on organization's project management competence, Crawford (2006) claimed that PMO should have roles of supporting and controlling projects, creating plans, supporting project methodology and management tools, developing project manager competence and work experience, establishing project strategy goals, managing resources, auditing and reviewing, and purchasing and contracting.Desouza and Evarsito (2006) however classified the functions of PMO into the strategic, tactical and operational levels in their case study on PMO archetype.They stressed that PMOs should provide project managers with administrative support and knowledge management of developing best practices and project methodology in order to improve project outcome.They particularly noted that knowledge management applies to all of the three levels previously mentioned.They also classified PMO as supporters, information managers, knowledge managers and coaches.Ayyagari, Henry and Purvis (2006) presented the functions of PMOs in detail, by classifying them into three value domains of knowledge, control, and resource.The knowledge domain includes development of project management methodology, standard template, and checklist.Companywide project management, change management, risk assessment and project outcome measurement belong to the control domain.Resource domain is comprised of project manager selection, team member recruitment, resource allocation and project budget planning.Based on prior studies, Hill (2004) categorized the functions of PMO into the five categories.Hill's classification of PMO's role is known to be one of the most specific and comprehensive.This study, therefore, adopted the categories of PMO functions defined by Hill (2004) in order to evaluate the efficiency of PMO.Hill's classification of the functions of PMO comprises five main domains, as shown in Table 2: practice management, infrastructure management, resource integration, technical support, and business alignment.Hill also subdivided each main domain into four sub-domains, respectively.The total number of sub-domains is twenty.

Research Method Decision Making Units
For this study, the online survey was conducted from Nov. 6, 2014 to Nov. 24, 2014 to the PMO personnel who performed IS projects.Sixty-five personnel responded to the questionnaires.Forty-nine PMOs which supervised IS projects having the budget of US$1M or more have been selected for analysis.The statistical information of PMOs for the analysis is as shown in Table 3.

Selection of Input and Output Factors
To evaluate the relative efficiency of each PMO, five PMO functions defined by Hill (2004) were used as input factors and four project outcomes were selected as output factors.The five functions defined in the study of Hill (2004) comprise practice management, infrastructure management, resource integration, technical support and business alignment.The detailed description of these input factors was as follows.Practice management comprised project management method, project management tools, project management standards and metrics, and project knowledge management.The detailed functions of infrastructure management comprised project governance, project assessment, project implementing organization and structure and project work facilities and equipment support.Resource integration included project resource management, training and education, career development and team development.Technical support consisted of the four detailed sub-functions such as mentoring and consultation, project planning support, project auditing and project recovery.Business alignment also incorporated the following four sub-functions: project portfolio management, project customer relationship, project vendor/contractor relationship and business performance.
Meanwhile, output variables were made up of the following project outcomes presented by the Standish Consulting Group: time compliance, cost compliance, requirement sufficiency, and project performance.A 7-point Likert Scale was used to measure the degree of support and project outcomes from PMO for the functions mentioned above.Each of five functions in PMO had four sub-variables as shown in Table 2.The variables were measured by PMO personnel who conducted the IS project.In addition, the average value of four sub-variables was used for each function.
In DEA, it is desirable to have more DMUs.If the number of DMUs is small, then most of them are more likely to form the efficiency frontiers.This makes the relative efficiency of these DMUs close to 1.In addition, the numbers of input variables and output variables can have the same effect on the determining power as the number of DMUs does.To prevent this, Cooper, Seiford and Zhu (2004) argued that the number of DUMs should be more than three times the sum of the input and output variables.In this study, the number of DMUs is forty-nine in total and the sum of input and output variables is nine.Thus, the condition is satisfied.The descriptive statistics of the selected input variables and output variables are shown in Table 4.

Results of the Relative Efficiencies of DMUs
The results of analysing the relative efficiencies of PMOs are shown in Table 5.The results of the CCR model in Table 5 represent that twelve DMUs are efficient and the remaining thirty-seven DMUs are inefficient.The average efficiency of all DMUs is relatively high at 0.80.From the results of the BCC model, it is found that the number of efficient DMUs is twenty-four, whereas the number of inefficient DMUs is twenty-five.The average efficiency of all DMUs in the BCC model is 0.89.The number of efficient DMUs in the BCC model is twice larger than that found in the CCR model.
The CCR and BCC models' results are used to measure the scale efficiencies of output variables.A total of fifteen DMUs, including DMU3, DMU4, DMU5, DMU9 and DMU10, show scale efficiency.The most inefficient PMO in terms of scale is DMU44.The cause of inefficiency of DMU44 is not related to technical efficiency.Rather, the scale inefficiency generates the inefficiency of DMU44.Therefore, it would be possible to eliminate inefficiency by improving the outcome variables when input variables such as practice management and infrastructure management are given.Among those inefficient DMUs solely associated with technical efficiency, DMU37 is found to be the most inefficient.To reduce the technical inefficiency, various approaches such as project management system improvement and organizational structure realignment for these PMOs are recommended.
In the column of 'returns to scale' in Table 5, the constant return to scale (CRS), decreasing return to scale (DRS) or increasing return to scale (IRS) can be determined by the value of The analysis showed that twelve PMOs experienced CRS or attained the optimal size.Thirty-four PMOs are in the state of DRS in which the production rates of them is less than that of DMU D and three PMOs in the state of IRS.PMOs of DMU16, DMU25, and DMU45 need to increase their scale as they are in the IRS state in which the return rate is higher than that in the optimal scale.It means that as the performance increase rate is higher than the input increase rate for the operation of PMO, it is better to increase the input in order to enhance the scale efficiency.On the other hand, other PMOs need to be reduced in their scale as they are in the DRS state.Twelve organizations such as DMU3, DMU4, DMU5, DMU10, DMU15 and others show the most efficient PMOs as their technical efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE) all show 1.

Potential Improvement Level of Inefficient DMUs
In DEA, the efficiency frontier DMUs are determined and then the other DMUs are compared with the generated efficiency frontiers to measure the relative efficiencies (Han and Kim, 2008;Park and Yoo, 2013;Koh and Kim, 2014).Thus, utilizing the nature of DEA allows us to obtain information which points the inefficient DMU to its benchmark.In other words, it is possible to infer the potential improvement levels of input and output factors of the inefficient DMU.The procedure to obtain the results is as follows.
First, it is necessary to obtain the targeted values of inefficient DMUs in order to become efficient.Equation (3) means that output variables are increased by as much as the amount of slack after reaching the empirical efficiency frontier for the j th DMU.Equation ( 4) means that the input factor of j th DMU is less than the amount of slack.The coordinate ( , ) on the efficiency frontier of the output-based BCC model becomes the target value of j th DMU ( , ).This is also referred to as projected point.Therefore, the potential improvement level of the input factor for the j th DMU becomes , whereas the potential improvement level of output factor becomes .In Table 6, the potential improvement levels of the input and output factors of inefficient DMUs are presented.Inefficient DMUs can become efficient by adjusting input and output levels presented.

Analysis of Influencing Factors
The impact of efficiency on project outcomes depending on the levels of PMO functions (practice management, infrastructure management, resource integration, technical support and business alignment) was analysed.For the analysis, the functional level of PMO was classified into high (H) and low (L) based on the average value of each function (See Table 4).The efficiency value was also classified into two groups of high (H) and low (L) according to their average value (BCC model) (See Table 6).Figures 2(a)~2(c) represent the impact of practice management on time compliance, cost compliance, and requirement sufficiency according to efficiency levels.As shown in Figure 2, those groups with a high degree of efficiency have high outcomes regardless of levels of practice management.Those groups with a low degree of efficiency generally show low outcomes compared to the groups with a high degree of efficiency.However, in the outcomes for schedule and cost compliance, the gaps between the two efficiency groups are substantially reduced when the level of practice management is increased to "high".
From the perspective of customer requirement satisfaction, the gap between these two efficiency groups is barely reduced regardless of the level of practice management.Figure 3 represents the project outcomes for each efficiency group with respect to the level of infrastructure management.The gaps in outcomes between these two efficiency groups are significant and are maintained regardless of the level of infrastructure management.In particular, the gap in customer requirement satisfaction between the two efficiency groups is larger than that of the other two outcomes.This means that efficiency has the largest impact on customer request sufficiency than any other outcomes.Note that impacts of other PMO functions like resource integration and technical support show similar results.
The effects of business alignment on the project outcomes for different efficiency levels are presented in Figure 4.As shown in Figures 4(a) and 4(b), the outcomes are higher when the level of business alignment is higher.Moreover, the outcome of the high-efficiency group is also higher.In particular, the gap between these two groups becomes more significant in customer request sufficiency than in schedule and budget compliance.Similarly, this means the efficiency is the most significant factor for customer request sufficiency than for any other outcomes.

Conclusions
This study analysed the relative efficiencies of PMOs in large-scale IS projects by utilizing DEA.Furthermore, the potential target improvement levels for each input and output factors were presented.The impacts of efficiency on project outcomes according to PMO functions were also studied.
The result from analysing the output-based BCC DEA model, found that the number of PMOs that are efficient (efficiency value=1.00) was twenty-four and the remaining twenty-five PMOs were relatively inefficient with an average of 0.89.The efficiency frontier points for inefficient PMOs were also obtained from the DEA model and used as benchmarks.The distances from the current position of inefficient PMOs to the benchmarks represent the amount of improvement needed in the input and output variables.The results show that the area with the lowest performance is in cost compliance and the current output level of cost compliance should be improved by 85% to become efficient.Time compliance follows requiring a 67% improvement.A lot of effort should be put into improving cost compliance and time compliance to improve efficiency.The causes of the inefficiency of PMOs were identified by utilizing scale efficiency (SE) value.The results show that more than half the inefficient PMOs (twenty-two PMOs) are inefficient in terms of scale.In particular, DMU44 (0.59) was found to be the most inefficient.These inefficient PMOs would be able to eliminate their inefficiency by making improvements in the outcome variables.As for the remaining fifteen PMOs, their inefficiencies are solely related to technical inefficiency.To reduce the technical inefficiency, various approaches such as improving the project management system and restructuring the organization for these PMOs are suggested.Moreover, twelve PMOs experienced constant returns to scale, thirty-four PMOs experienced decreasing returns to scale, and the number of PMOs with increasing returns to scale was three.
The impacts of efficiency on these project outcomes like time compliance, cost compliance, requirement sufficiency according to PMO functions of practice management, infrastructure management, resource integration, technical support and business alignment were analysed.As a result, the group with a high degree of efficiency showed higher outcomes than the group with a low degree of efficiency, regardless of their levels of practice management.However, the gap is substantially reduced with a higher level of practice management in relation to schedule and cost compliance.From the perspective of customer requirement satisfaction, the gap between these two groups showed negligible reduction regardless of the levels of practice management.
As for the outcome for each efficiency group with respect to the levels of infrastructure management, there was a significant difference in outcome between these two groups.The customer request sufficiency depicts the biggest gap.This indicates that efficiency has the largest impact on customer request sufficiency than any other outcomes.Also, the gap remains almost constant regardless of the levels of infrastructure management.The outcomes for each efficiency level of resource integration and technical support also showed similar results to that of infrastructure management.The outcomes for each efficiency group with respect to the level of business alignment also show a similar pattern to results mentioned above, except that the gap between these two groups is more significant in customer request sufficiency than in schedule and budget compliance.This means that the outcome having the largest impact from efficiency is customer request sufficiency.
The contributions of this study are as follows.First, this is the first study in Korea to measure and present the efficiency of domestic PMOs based on large-scale IS projects by utilizing DEA.Moreover, the causes of inefficiency of PMOs are explored.Furthermore, this study establishes the basis for inefficient PMOs to become more efficient by presenting the potential improvement value for each input and output variable.Second, the impact of efficiency on the outcome for each functional level of PMO was identified.As a result, efficiency significantly affects outcome on all levels of PMO functions.This study ultimately provides practical values that establish the direction for effective PMO management.
However, the analysis results may be subject to change depending on the selection of input and output factors, given the intrinsic nature of DEA model.It is therefore necessary to take this into consideration when applying the results of this study in actual PMO management.

Limitations and Future Work
In this study, the input and output elements for DEA were selected based on prior studies on PMO.However, the results of DEA may vary depending on the selection of input elements and output elements due to the intrinsic features of DEA.Accordingly, these points should be considered when they are used for control of PMO.
This study analysed the efficiency of PMOs focusing on large scale IS projects.In the future, more studies are required to find out the efficiency of PMOs by focusing on not only IS projects, but also the projects in other fields.It would be meaningful to analyse the changes in productivity in terms of time, utilizing the Malmquist method.In addition, studies that are conducted to enhance efficiency by finding out the elements which affect the efficiency of PMOs through Tobit analysis would provide PMO personnel with realistic implications.It is also worth noting that if efficiency analysis is conducted by considering the various kinds of internal and external environmental factors which may affect the performance of PMO, more meaningful and helpful results would be obtained.

Figure 2 :
Figure 2: The effects of efficiency on project outcomes in accordance with practice management

Figure 4 :
Figure 4: Project outcomes for each efficiency group in accordance with business alignment

Table 3 :
Statistical information of PMO

Table 4 :
Descriptive statistics for data envelopment efficiency model variables

Table 6 :
Potential improvement levels of input and output factors of inefficient DMUs