Requirements for optimal learning environment for an online project risk management game

Several authors have questioned the effectiveness of using lecture-based teaching to provide students with enough confidence to apply project risk management. Gaming was proposed as a solution. However, despite widespread use of games in teaching project management, it is still not clear what conditions provide optimal learning through games. Another shortcoming with the existing games is oversimplification. This paper addresses these shortcomings and proposes a game design that captures real-life challenges associated with applying the project risk management process; a design that prompts an appreciation for project complexity as well as providing students with the opportunity to experience the consequences of ignoring or following the risk management process. The paper also identifies and elaborates on the requirements for optimum learning, and distinguishes between two types of requirements: 1) learning requirements, and 2) qualitative requirements. Learning requirements identify the learning outcomes of the game. These requirements were identified through structured and semi-structured interviews with senior project managers from several management-consulting firms. The challenges and the corresponding tactics that are adopted in practice in order to manage project risks were thus identified and ranked. These results are also presented in light of supporting literature. The challenges and associated tactics were mapped into a set of eight requirements representing the learning outcomes of the game. These requirements were then mapped to the design using four instructional methods: a briefing lecture, a team-based assignment, an online computer simulation, and a debriefing lecture. All these methods were linked by a real-life project case and executed in a gaming context to improve engagement. Qualitative requirements represent important conditions that must be present for optimal learning. These were identified through structured interviews with continuing education students taking a master’s degree in project management. This empirical study resulted in four qualitative requirements that must be considered in the game design: 1) ownership, 2) relevance, 3) feedback, and 4) adaptation. The paper also presents the evaluation results of the game design. The purpose of the evaluation was to examine the game’s ability to capture the two sets of requirements identified above. Introduction The theoretical foundation of simulation games as a learning/teaching tool is provided by the experiential learning model (Kolb 1984). Experiential learning stresses the importance of direct experience, reflective observation and appropriate feedback in a continuous process of goaldirected action. Games are used to create experimental environments within which learning can 2 occur and be observed (Hussein 2007). While assignments, case studies and role play provide training and experience with reality, the unique characteristics of simulation games as an education tool is the inclusion of a time-line (Basnet 1996). The inclusion of time as an element in the simulation game implies that game participants have to live with their previous decisions as the game evolves. The possibility of including time as a factor makes gaming an excellent tool for teaching project management in general and project risk management in particular. The game artefact itself can be a board, computer, internet, a classroom, and so on. However, most of the reported simulation games in project management are conducted in a computerassisted environment (Rowe et al. 1968; Estes 1974; Deitzler 1978; Harris & Flower 1984; Jakubowski et al. 1984; Pamukcu & Pruett 1985; Cano & Saenz 2003; Prisk & Dunn 2002; Martin 2000; Mario et al. 2005). Others, such as Klassen and Willoughby (2003) and Hood and Hood (2006) reported classroom-based simulation games using artefacts other than computers. The effectiveness of simulation games in management training compared to other instructional methods is still unclear. Basnet (1996) argues that despite the lack of consensus regarding the teaching and grading methods to be used in conjunction with such games, it is clear that well conducted simulation games can provide excellent experiential atmospheres for students of management. Pfahl et al (2003) and Pfahl et al (2004) conducted several experiments in order to evaluate the learning effectiveness of using simulations in software project management education. They concluded that the simulation-based role-play scenario is a very useful approach for learning about issues in software project management. Randel et al (1992) concluded that subject matter areas where very specific content such as mathematics can be targeted are more likely to show beneficial effects for gaming. That is unfortunately not the case in project management. And finally, it goes without saying, that games could be built to be genuinely enjoyable; this feature according to Corti (2006) leads to longer attention spans, improved attentiveness and positive feelings. Teaching project risk management Uncertainty is an inherent characteristic of projects since projects are unique and temporary undertakings based on assumptions and constraints. The project risk management process could be seen as an proactive attempt to understand, assess and manage this uncertain environment (Benta et al 2011). Maytorena et al (2007) indicate that interest in risk management has increased as the size and complexity of projects have grown and as competition between firms has intensified. As a result, numerous best practice standards exist: BS 6079: 3: 2000, AS/NZS 4360: 2004, COSO 2004 or ISO 31000. The Project Management Institute (PMI 2004) has identified 44 processes that fall into five basic process groups and nine generic knowledge areas. Project risk management is one of the nine project management areas, which focuses on describing the processes that are important in order to conduct proper risk management on a project. Pinto (2010) defines project risk management as the art and science of identifying, analysing and responding to risk factors throughout the life of a project and in the best interest of its objectives. The objectives of project risk management are to increase the probability and impact of positive events (Olsson 2007) and to decrease the probability and impact of events adverse to the project. Because of its importance, Jaafari (2001) has suggested expanding the application of project risk management to include business


Introduction
The theoretical foundation of simulation games as a learning/teaching tool is provided by the experiential learning model (Kolb 1984).Experiential learning stresses the importance of direct experience, reflective observation and appropriate feedback in a continuous process of goaldirected action.Games are used to create experimental environments within which learning can occur and be observed (Hussein 2007).While assignments, case studies and role play provide training and experience with reality, the unique characteristics of simulation games as an education tool is the inclusion of a time-line (Basnet 1996).The inclusion of time as an element in the simulation game implies that game participants have to live with their previous decisions as the game evolves.The possibility of including time as a factor makes gaming an excellent tool for teaching project management in general and project risk management in particular.
The game artefact itself can be a board, computer, internet, a classroom, and so on.However, most of the reported simulation games in project management are conducted in a computerassisted environment (Rowe et al. 1968;Estes 1974;Deitzler 1978;Harris & Flower 1984;Jakubowski et al. 1984;Pamukcu & Pruett 1985;Cano & Saenz 2003;Prisk & Dunn 2002;Martin 2000;Mario et al. 2005).Others, such as Klassen and Willoughby (2003) and Hood and Hood (2006) reported classroom-based simulation games using artefacts other than computers.
The effectiveness of simulation games in management training compared to other instructional methods is still unclear.Basnet (1996) argues that despite the lack of consensus regarding the teaching and grading methods to be used in conjunction with such games, it is clear that well conducted simulation games can provide excellent experiential atmospheres for students of management.Pfahl et al (2003) and Pfahl et al (2004) conducted several experiments in order to evaluate the learning effectiveness of using simulations in software project management education.They concluded that the simulation-based role-play scenario is a very useful approach for learning about issues in software project management.Randel et al (1992) concluded that subject matter areas where very specific content such as mathematics can be targeted are more likely to show beneficial effects for gaming.That is unfortunately not the case in project management.And finally, it goes without saying, that games could be built to be genuinely enjoyable; this feature according to Corti (2006) leads to longer attention spans, improved attentiveness and positive feelings.

Teaching project risk management
Uncertainty is an inherent characteristic of projects since projects are unique and temporary undertakings based on assumptions and constraints.The project risk management process could be seen as an proactive attempt to understand, assess and manage this uncertain environment (Benta et al 2011).Maytorena et al (2007) indicate that interest in risk management has increased as the size and complexity of projects have grown and as competition between firms has intensified.As a result, numerous best practice standards exist: BS 6079: 3: 2000, AS/NZS 4360: 2004, COSO 2004 or ISO 31000.
The Project Management Institute (PMI 2004) has identified 44 processes that fall into five basic process groups and nine generic knowledge areas.Project risk management is one of the nine project management areas, which focuses on describing the processes that are important in order to conduct proper risk management on a project.Pinto (2010) defines project risk management as the art and science of identifying, analysing and responding to risk factors throughout the life of a project and in the best interest of its objectives.The objectives of project risk management are to increase the probability and impact of positive events (Olsson 2007) and to decrease the probability and impact of events adverse to the project.Because of its importance, Jaafari (2001) has suggested expanding the application of project risk management to include business objectives.It is now widely accepted that even moderate levels of risk management planning are sufficient to increase the chances of project success (Zwikael & Ahn 2011;Roy et al 2001).
The general consensus from the PMI (2004) and other project management literature, for example; Pinto (2010), Chapman and Ward (2003) and Kerzner (2006) is that the risk management process can be divided into the following basic processes: 1. Risk identification: the process of identifying events or conditions that may occur during project life cycle and could have an impact on at least one project objective.It is considered good practice to use several formal methods for the identification of risks in this stage.2. Risk assessment: the process of assessing the likelihood and the magnitude of consequences of the identified risks on project objectives.According to the PMI ( 2004), this process should be carried out in two stages, namely, qualitative assessment followed by the quantitative establishment of a numerical rating to measure the risk severity on one or several project objectives.Loosemore (2006) suggested however that quantitative risk analysis should only follow on qualitative risk analysis if the latter has exposed important risks that could be analysed with reliable data.van Wyk et al (2008) suggested using semi-quantitative risk analysis to overcome the shortcomings of the subjectivities of the qualitative risk assessment by assigning predetermined values to the probability and impact which, according to them, will result in more precise estimates of risks.3. Risk response planning: the process of identifying measures for dealing with risks.This includes the choice of a strategy to avoid, transfer or mitigate risks when they could be perceived as threats.Loosemore et al (2005) found that most approaches to risk management are not driven or inspired by the opportunities that risk management can offer (the upside of risk), but by the fear of doing something wrong (the downside of risk).It is now widely accepted that risk response should also include identifying strategies to exploit, share or enhance risks if they contribute positively to the project.4. Risk monitoring and control: the process of monitoring, evaluating and updating the risk register.Risk mentoring includes the reassessment of risks or re-examining risk response measures.It is considered good practice to use a risk log or register database system (Patterson & Neailey 2002).This is to facilitate the monitoring, control and evaluation of the risks (Willams 1994).It is also considered best practice to decide the frequency of monitoring the risk and the method of reporting very early in the risk process.
The use of games in project management training may be justified because of the unique characteristics of the skills needed in order to perform the risk management process.First, it is a people-centred process.In other words, people involved in the project form their own subjective perceptions of risk based on their understanding of the context, culture, expectations, experience and skills.Jani (2011) examined the importance of perception of risks and concluded that there is a correlation between accurate perception of risks and the likelihood of success in information technology projects.Second, decisionmaking is largely based on qualitative evaluation, and is dependent on project's context.Reading or thinking about risk management process is, therefore, not enough (Martin 2000).It is a process that must be experienced, reflected upon and guided through feedback and debriefings.
Lectures, assignments and case studies, therefore, do not help students develop an understanding of the difficulties involved in identifying, assessing, planning and monitoring risks.Developing an understanding of these processes requires different types of instructional methods.Taran (2007) questioned the effectiveness of lecture-based teaching in providing students with enough confidence and ability to apply risk management concepts, and pointed out major shortcomings: 1-Lectures do not provide the possibility to experiment with the material being taught.Specific exercises and activities help, but these do not provide an entire "project picture".2-It is difficult to provide students with a way to experience scenarios of following or ignoring risk management practices.Cano and Saenz (2003) pointed out that despite the widespread use of simulation games it is still not clear what conditions have to be provided in order to obtain optimal learning through such games.Another reported shortcoming with the existing games is oversimplification, manifested in the type, timing and even realism of events of the game.Thomas and Mengel (2008) stressed the importance of using methods that prompt the understanding and appreciation of project complexity.A project risk management game should have an adequate level of realism so that it can prompt appreciation for project complexity.
This paper contributes to the literature on gaming for teaching project risk proposes a game design that captures real-life challenges associated with applying project risk management process; a design that prompts appreciation for project complexity as well as providing the students with the opportunity to experience the consequences of ignoring or following the risk management process.The paper also identifies and elaborates on the requirements that must be taken into account in the game design in order to achieve optimum learning.

Goal and scope of the research
The research work included the following tasks.1) Identification of requirements: two sets of requirements were developed: A. Learning requirements: representing the intended learning outcome of the game.The preliminary results of this part of the research are described in (Hussein 2011).This paper revisits and refines these learning requirements.The revised learning requirements are presented in section 3. B. Qualitative requirements, representing important qualitative characteristics the game must have, and that are important to ensure engagement when playing the game.The results of this task are presented in section 4. 2) A proposal for a game design was developed.Each requirement developed was realised in the design by one or several instructional aids and methods.The design is explained in section 5.
3) The final task was to evaluate the final design.The purpose of the evaluation was to examine the game's ability to capture the two sets of requirements identified above.

Learning requirements
Semi-structured interviews were used to collect the challenges and solutions associated with project risk management.After exhausting the questions, informants were offered the opportunity to provide other insights on managing project risks.Each project manager was asked to identify at least two major challenges or issues during each stage of the project risk management process.These interviews yielded a framework of general issues (challenges and tactics).This list of issues was then mapped into a group of eight requirements that the design must embody.The interviews revealed the following results.

Context
The informants stressed the importance of identifying and understanding the project context as a prerequisite for managing project risks.Informants have also pointed out that project context is subject to changes by for example external factors such as changing regulations or objectives.This stage involves, gathering, documenting and distributing updated information on project goals, objectives, constraints, conditions and limitations.It could include time and budget constraints, organisational and resource constraints, laws, ethics or financial and pricing structures.It was stressed that project managers should make sure that the information is made available to and understood by those who will be responsible for managing risks.Similar conclusions were also made by (Kendrick 2009).Couillard (1995) has shown through a field study the correlation between project goal understanding and effective risk management.Mapping these issues into the game gave the following requirements.
R1.The game should show that the availability (with the right quantity and quality) of project information affects the outcome of the project risk management process.R2.The design should reflect the dynamic nature of projects, including changing constraints, stakeholder support, and others.

Risk identification
Informants stressed the importance of including representatives of any stakeholder who has a stake in at least one of the project objectives in order to create ownership of the measures taken.The role of stakeholders in the risk management process is also emphasised by project management literature (Ward & Chapman 2008).
The informants reported that one of the problems they usually encounter during this stage is the lack of interest among the people involved in order to perform this stage in a proper manner.The informants indicated that these individuals fail to prioritise this risk identification process.Moreover, they reported that these people usually expect quicker and less time-consuming approaches to the risk identification phase.This hasty approach usually results in incomplete and shallow understanding of risk factors, which has an impact on the succeeding stages.
The informants also reported that the people involved do not see the value of conducting proper risk management.They believe they can do it in their head, so there is a tendency to ignore or downplay formal risk management methods.The role of ignorance was examined by Kutach and Hall (2010), who suggested defining and adhering to priorities during this phase.Informants recommended the use of structured and formal approaches to the risk identification stage.Informants also indicated that this phase should be used effectively to help the people involved to remain focused on these priorities.
Informants stressed the importance of having the right people with the right experience in the group.The study by Maytorena et al (2007) did not support this view but argued that the role of experience in the risk identification process is much less significant than is commonly assumed.They confirmed, however, that information search style, level of education and risk management training do play a significant role in risk identification.
The informants pointed out that historical information and knowledge from previous projects can help in closing the knowledge gap and hence speed up the identification stage.This might not be true for all types of projects; for instance, the study by Sary et al (2006) questioned the usefulness of past data for technological projects and concluded that historical information about typical risks are less important in case of high technology projects.
In order to deal with these issues, the informants suggested, among other things, using several identification techniques such as brainstorming, cause-effect relations.These techniques are described by the PMI (2004).Lester and Lester (2007) described the advantages and disadvantages of typical risk identification methods such as brainstorming, prompt lists, checklists and other methods.Loosemore (2010) suggested the use of multimedia to better engage, enthuse and stimulate the stakeholders involved in this stage.Mapping these issues into the game gave the following requirements.

Risk assessment
Informants reported that a tough issue during this stage is to agree on the criteria that will be used to prioritise the risk factors.Therefore, common understanding about the project objectives and success criteria among team members is of a paramount importance in order to complete this stage successfully and efficiently.Project risk management practice indicates that it is neither possible nor recommended to mitigate or eliminate all project risks.Monte-Carlo analysis is frequently used to assess the probability of achieving project objectives such as cost and time in the presence of risks (Lester & Lester 2007).Risk prioritisation is performed by grouping risk factors into categories depending on the magnitude of impact and probability of risks (Kendrick 2009).
Almost every informant mentioned lack of experience, indifference, lack of time, bias and prejudice as enemies of the risk assessment phase.This involves the assessment of probability and the impact of risks on project objectives.The strategy identified by informants to tackle this problem involves selecting people with the relevant project-related experience, as well as supporting the assessment with historical data from previous projects.These suggestions are also supported by Chapman and Ward (2003), who stressed the importance of previous experience and familiarity with the risk category as preconditions for completing this stage efficiently.Mapping these issues into the design gives the following requirement.
R6.The game should show that accurate assessment and prioritisation requires access to historical information, data from similar projects and a through understanding of project objectives.

Risk response planning
Risk planning involves selecting the proper measures in order to reduce or mitigate the probability of risks or to reduce their consequences.Risk response planning could also include measures intended to remove the conditions that cause this type of risks.Informants stressed that all agreed measures must have measurable results.Fan et al (2008) confirmed that a proper riskhandling approach should take into account the unique project characteristics, risk situation and implications on project objectives.Acquiring information and improving communication are preconditions for developing proper risk response measures.Similar suggestions are also found in Ward (1999), who emphasised the criticality of considering the feasibility of selected response measures and the time available for them.Mapping these considerations into the design gave the following requirement.

R7:
The game should help learners experience the impact of failing to select proper measures to deal with risks (by using simulation as a forecasting tool to investigate possible risk response strategies for dealing with risks).

Risk monitoring and control
Informants identified that the major challenges in this stage as information gathering and distribution, the availability of new information about changing project conditions and information about the results from risk response planning.Mapping these considerations into the design gave the following requirement.
R8.The game should illustrate the importance of information gathering about project objectives status and changing context in order to make informed decisions.

Qualitative requirements
It is clear that well conducted games can provide excellent experiential environments for students of management.Kiili (2005) discussed factors important for developing games that are engaging and that would result in increased learning.These include control, playability, game-fullness, focused attention, feedback, clear goals and challenges that are matched to players' skill levels.
In order to identify factors that will contribute to optimum learning in the game, I conducted two test sessions for a risk management game prototype for a group of 30 players.These players were students taking continuing education course in project management.The sex, educational background, type of work and experience profile of the respondents were diverse.Some students were taking the course to satisfy an obligatory requirement for a competence-based master's degree in organisation and leadership.Others were enrolled in order to gain a greater insight of project management methods or in pursuit of new career opportunities in project management.All had some project management experience, either as participants or as managers.The test session was followed by a workshop and an open debate.The central issue of these discussions was the conditions for optimum learning in the game.Table 1 shows examples of the type of industry and job titles of informants.The workshop participants were asked to identify characteristics of the prototype that contributed to a better understanding of the complexity of the project risk management process.They were also instructed to identify characteristics that could have been built into the game design.The discussions revealed the importance of the following requirements.

R9. Relevance:
Participants stressed the importance of relevance of the case used in the game to their background and needs.The one-size-fits-all approach was not preferred.They recommended developing risk management games with multiple cases from several domains; for example, information systems, construction, modification, product development and organisational change, so that the players themselves can select the case that best fit their needs.Taran (2007) also came to a similar conclusion, and pointed out that realism and relevance contributes to better learning.

R10. Ownership.
Informants suggested developing a game that could provide players with the possibility of experiencing the entire project risk management process, including identification, assessment, response planning and monitoring.This condition is very important in order to enhance the level of involvement in the game, which will turn give players a sense of ownership of the entire simulation.Importance of involvement has been heavily emphasised in project management literature.For instance, Loosemore (2010) argues that the benefits of involvement in projects include building trust, wider ownership of the decisions take, better understanding of constraints and greater collective responsibility in managing risks.

R11. Feedback.
Informants suggested the use of visual aids to provide status information about cost, time and other project objectives.Aids such as the S-curve and progression map to show instantly the consequences of failing to prioritise or mitigate risks on project objectives.Debriefing sessions and use of a moderator during the game was also mentioned as effective measures for providing feedback.According to Peters and Vissers (2004: 4), debriefing can be considered the phase in which the game's learning objectives are made evident.In debriefing, participants are asked to explore possible connections between the experiences they had while playing the game and experiences in real-life situations; i.e., what they have learnt from the game.Kiili (2005) discussed feedback as one of the most important factors that are important to develop games that are engaging and would result in increased learning.Other factors according to the same reference include clear goals and challenges that are matched to players' skill levels.

R12. Adaptable.
Informants advised that the game should be developed in such a way that it can function as an adaptable incubator or knowledge base for the type of risks, their consequences and possible measures to counter these types of risks for each project case.The extent of this knowledge should increase as more players play the game.They pointed out that game adaptability is important to show that the actual purpose of the risk management process is to learn about the project rather than just using it as a decision-making tool.
In gaming literature, adaptation is used for two purposes (Bakkes et al 2009) and (Ram et al 2007): 1) Enabling the game to change its behaviour in order to meet the user capacity level; 2) Ensuring two-directional knowledge transfer between players and the game.Adaptive games are dynamic; the game can be played several times without having the same results and challenges every time.This feature will ensure engagement and better learning.It will also transform the game into a knowledge-sharing exercise because it will learn from experienced users and transfer this new knowledge to new users.

Game design
The design itself uses a combination of four instructional methods (briefing lecture, team-based assignment, online game and debriefing lecture).These methods are implemented in order to realise the requirements identified in the previous section.In the following sections, we provide a brief description of each method used in the design.
A web-based interface allows the instructor to plan the project when it is played for the first time.Planning involves defining the type of project, number of work packages, relationships between these work packages and expected workload of each work package.The instructors also set up the template of the risk register.The risk register is a table that summarises risk factors and assesses the impact and likelihood of these factors.It also includes possible measures to mitigate these risks.In addition, the instructor defines other parameters such as project success criteria and availability of resources for each work package.The instructor can include any related background information such as scope, goal and objectives and historical data about similar projects.

Briefing lecture
The aim of the briefing lecture is to introduce the underlying project risk management theory and processes.The concepts of risk register; risk matrix, brainstorming, qualitative and quantitative risk management, risk planning and risk monitoring techniques are explained.At the end of the briefing lecture, project teams (of 45 people) are established and a project definition document (PDD) is distributed to the teams.The PDD includes information about project scope, product description, cost and time constraints, project success criteria as defined by the project owner and other assumptions and requirements.Two project cases were developed that student groups could select from; an information system project, and a construction project.Teams were also instructed to decide for themselves the roles and responsibilities of each member at each stage of the game.

Team-based assignment
Based on the description given in the PDD, teams are instructed to identify and develop a complete risk register for each work package in the selected project.At the end of the assignment, the instructor collects and reviews the completed risk registers from each team.Risk factors are then categorised and duplicates omitted.The final edited list is then fed into the database of a simulation environment where actual project execution will take place.The aim of this team-based assignment is twofold.First, it provides students with hands-on training in identification, assessment and mitigation.Second, it prepares each team for the next phase of the game (computer simulation) when they will have to assess, prioritise and select mitigation measures for far more risk factors than those that each team managed to identify and assess on its own.

Online game simulation
The simulation itself takes place online.The simulation environment contains an updateable risk register database of the risk factors that may occur and that could affect the project's objectives.At the start of the simulation, the simulation engine randomly selects 3-5 risk factors from the database for each work package.The list of risks includes two categories: 1) risk factors identified by the teams during the preceding assignment; and 2) risk factors identified in previous experiments by other classes.The database can, therefore, be seen as an incubator of risk factors for each class of projects.
During simulation, teams use their own experiences as well as the information provided to assess the likelihood and consequences of each risk factor displayed before them.Alternative risk mitigation measures are also displayed next to each risk factor.Failing to select the proper measure might trigger the occurrence of additional risk factors in later project stages.The price or time needed to implement each measure is also given.Failing to assess risks correctly will ultimately result in failing to respond to critical risks.This might result in severe delays, penalties by authorities, slow progression and so on.Visual effects such as video clips, sounds and images are used to illustrate the consequences of failing to assess risks.At the same time, teams should not mitigate all the risks listed.If they do so, actual costs will overrun the budget.Teams must thus mitigate only those risk factors that are critical or significant.Information provided in the leaflet is meant to assist the teams to prioritise risks and select measures appropriately.
The simulation environment also contains two types of visual aids:  Progression map.During project execution, teams will be able to get updates and information about the status of the project.The information shown in the map includes work packages completed, percentage completion, number of days passed, money remaining, person-hours used and simulation time.The maximum allowed time to complete the project in the simulation is set to 60 minutes.After 60 minutes, a fine will be imposed for each additional minute. S-curve.This is a graph showing the accumulated actual costs, planned costs and earned value.The S-curve is updated after each turn in the simulation.This graphical aid should help teams visualise the consequences of their decisions on time and money instantly.A reflective analysis of the information offered by the S-curve should help teams think carefully during the risk assessment and prioritisation of subsequent work packages.

Debriefing lecture
The debriefing lecture takes places at the end of the computer simulation.It evaluates the performances of each team, revisits and discusses execution strategies and explains how teams distributed the roles and responsibilities.The efficiency of communication in the team and reflections about conformity, dominance, bias and indifference are also discussed with teams by linking the results to the project management theory and identifying lessons learned.
Table 2 shows an overview of the identified requirements and describes how each requirement was embodied in the complete design.

Requirement
How the requirement was realised in the design R1.The game should show that the availability (with the right quantity and quality) of project information affects the outcome of the project risk management process.
The briefing lecture.Information is prepared in advance, stored and made accessible to all teams.
R2.The design should reflect the dynamic nature of projects, including changing constraints, stakeholder support, and others.
Contextual risk factors are changed randomly.
R3.The design should demonstrate the importance of competence and experience in the actual project domain.
Briefing lecture.Team formation is not random.
R4.The design should demonstrate the importance of using various formal techniques to identify risks that could occur in projects.
Team-based assignment.Teams were given time to try to use the formal techniques to identify, assess and mitigate project risks.R5.The design should take into account challenges in real project situations such as time limitations and human factors.
Briefing.Players were not permitted to play individually.Simulation was time limited.Debriefing/discussion with teams about the human aspects encountered during the simulation.R6.The design should show that accurate risk assessment requires access to historical information and data from similar projects.
Team-based assignment.Information containing historical data and statistics from similar projects distributed.R7: The design should help learners experience the impact of failing to select proper measures to deal with risks Simulation/failing to select the proper measure could trigger the occurrence of risks in a later project stage.R8.The design should illustrate the importance of effective communication between participants in order to make informed decisions.
Briefing/after forming the teams, students were instructed to decide for themselves the roles and responsibilities of each team member.

R9. Relevance
Players can select what type of project they want to use in the simulation.

R10. Ownership
During the team-based assignment, players will be able to identify, assess and plan project risks.During simulation, they will be able to simulate some the risks they have identified during the team-based assignment.R11.Feedback Debriefing, S-curve and progression map are built in the simulation to provide information to teams about progression and earned value.

R12. Adaptable
The list of risk factors, measures and possible impact in the database will increase as more players play the game.
Table 2. Requirements and methods used to realise each requirement

Evaluation results
A total of 24 respondents took part in the evaluation.The forms were distributed right after the game.It was emphasised that the results of the evaluation were important for further development of the game.There were 20 valid responses.The players were students taking a continuing education course in project management at master's level.Around 30% of these participants reported that they had knowledge of and experience in project risk management.The rest of the population identified themselves as having no former experience in project risk management.
The questionnaires were used to assess if the design satisfied the requirements that were developed and presented in the previous sections.Respondents were asked to identify to what degree they believed the design satisfied these requirements.The scale given was from 1 to 6, where 6 meant strongly agree; 5, agree; 4, tend to agree; 3, tend to disagree; 2, disagree; and 1, meant strongly disagree.The target was for over 50% of the respondents to agree or strongly agree.Percentile statistics was used to present the results as shown in table 3. Table 4 shows the interpretation of the results.

R1
71% of the respondents agreed or strongly agreed that the game succeeded in showing the importance of information as an important tool in risk management.R2 63% of the respondents agreed or strongly agreed that the events or contextual risk factors that triggered during the course of the game succeeded in showing that projects are not static entities but are subject to continuous changes.R3 71 % of the respondents agreed or strongly agreed that the design demonstrated the importance of competence in the actual project domain.R4 24% of the respondents agreed or strongly agreed that the design demonstrated the importance of using various formal techniques to identify risks.Debriefing and discussions with respondents revealed that more time is needed to give players the chance to train in formal techniques.They suggested splitting the game into two phases to give more time for learning formal methods.R5 32% of the respondents agreed or strongly agreed that the design managed to take into account challenges in real project situations such as time limitations and human factors.Discussions with respondents revealed that effect of human factors was not evident in the design.

R6
58% of the respondents agreed or strongly agreed that the design managed to show the importance of availability of information from similar projects as precondition for proper risk assessment.R7 46% of the respondents agreed or strongly agreed that the design managed to show the impact of failing to select proper measures to mitigate project risks.R8 16% of the respondents agreed or strongly agreed that game managed to illustrate the importance of effective communication between participants in order to make informed decisions.R9 76% of the respondents agreed or strongly agreed that game was relevant and realistic.R10 69% of the respondents agreed or strongly agreed that game satisfied this condition.R11 77% of the respondents agreed or strongly agreed that game satisfied this condition.R12 Was not tested.

Table 4. Interpretation of results
The results show that that the design largely satisfied the intended requirements with one main exception, namely, R8.This suggests that the design should give focus more on group dynamics and communication in teams.The evaluation results indicate that the design needs to be calibrated and adjusted by: 1-Increasing the duration of the lesson to allocate more time to risk identification as well as to assessment and response planning activities; 2-Ensuring that the simulation manifests the consequences of failing to select the proper risk response measures; 3-Highlighting the effect of human factors and creating awareness about this factor during the briefing session.

Conclusions
This paper addressed several shortcomings and challenges in the use of games in teaching project management, and proposed and evaluated an online game design that captures real-life challenges associated with project risk management.The game prompts an appreciation for project complexity as well as providing the students with the opportunity to experience the consequences of ignoring or following risk management.The paper identified eight learning requirements that must be embedded in the game design.It also identified the qualitative requirements for achieving optimum learning in project risk management.These requirements are: 1) ownership, 2) relevance, 3) feedback, and 4) adaptation.
The uniqueness and strength of the design comes from its ability to engage students actively in the entire risk management process as well as using real-life project cases.The design also provides students with the ability to simulate some of the risks they identified during the team assignment.This gave the students a feeling of ownership of the risk management process during the simulation.The game can also be seen as an adaptable incubator or knowledge base for the type of risks, their consequences and possible measures to counter these types of risks.The amount of knowledge grows as more players play the game.The game, therefore, managed to show that the main purpose of conducting project risk management is for learning more about the project (a tool for learning, rather than decision making).The game also showed that decisions concerning risks must be based on an adequate analysis of both risks and project context.
R3.The game should demonstrate the importance of competence and experience in the actual project domain.R4.The game should demonstrate the importance of using various formal techniques to identify risks that could occur in projects.R5.The game should take into account challenges in real project situations such as time limitations and human factors.

Table 1 .
Examples of types of industry and job titles

Table 3 .
Evaluation results of the design.