Discussion forum rubrics: using rubrics to enhance and increase project management students’ participation in online discussions

This paper examines the impact of using a discussion forum grading rubric to encourage and enhance discussions (posts) in an online introduction to project management course. The study consists of 53 Master of Business Administration (MBA) students enrolled in three sections of the course from October 2010 to May 2011. A control group of 20 students was compared to two experimental groups of 16 and 17 students respectively. The two experimental groups were given the specific grading rubric and instructions designed to encourage online conversation; the control group was not. The results indicate that there was a statistical difference in average postings per student between the two experimental groups and the control group but no statistical difference between the two experimental groups. The results suggest that the use of a rubric specifically designed to engage the student and expand the student’s learning experience can increase online classroom participation and, as a consequence, enhance the educational experience for the online project management student and strengthen the university’s MBA program.


Background
It can be difficult to move online students to a formal rubric-based grading system for discussion forums, but the benefits are effective and measurable.Therefore, the purpose of the research was to examine the impact of using a discussion forum rubric to enhance discussion posts submitted by students in the virtual project management classroom.The analysis involved evaluating the first section of the virtual project management classroom in which the protocols and evaluation criteria were minimal or loosely defined, as well as the remaining sections of the course in which the opposite was true.The evaluation involved how the addition of facilitator guidelines influenced meaningful discourse, how the posting of these guidelines influenced meaningful discourse, and how the addition of an evaluation rubric influenced meaningful discourse.
The students often go kicking and screaming into the formal grading process.The following quote from a course evaluation demonstrates this: I ABSOLUTELY HATED THE DISCUSSION QUESTION REQUIREMENT to search for and provide a response with references from the text and an outside source PLUS [c]ite in APA format.That said it was probably the most beneficial part of the course.BUT I STILL HATE IT.
In teaching online project management classes, the grading rubric adds yet another dimension, as demonstrated by this student's insightful comments to the other learners in the class: As a PM I think that the harsh discussion requirements are right in tune with reality and that the difficulties are a learning opportunity in and of themselves.When you are managing a project and disagree with the customer on a requirement (that you agreed upon by signing), you are rarely going to see the results you expect.Especially if nothing has been said right up until the time it became difficult for you to execute … Let's look at this course as if we were project managers.
• Project initiation stage -We sign up for the course and the instructor agrees to teach the course.• Project planning and design stage -We prepare for the course and the instructor outlines the syllabus.• Project execution and construction stage -This would be week one.You can still get a return on your investment if you decide the course is not for you.
You have a chance to review the course materials (requirements) and to question them.By saying nothing, signing the statement of understanding and remaining in the course you have agreed to the project terms.• Project monitoring and controlling systems -We continue with the course work.
• Project completion -Final Exam and hopefully, the taking from this course a wealth of knowledge that will prove useful in your career.

Introduction
The explosion of online learning presents opportunities and challenges for universities to maintain quality in education while increasing satisfaction among learners.As business schools compete to capture market share in this competitive arena, they must also maintain the highest standards of excellence and the best learning environment for their learners (Blount 2011: 390-404).To do this, many institutions have developed online learning courses for the convenience of their students.The purpose of this research was to examine the impact of using a discussion forum rubric to enhance discussion posts submitted by students in the virtual project management classroom.
This need to develop high-quality online learning experiences requires instructional methods that ensure online discussion forums to provide a sense of community, interaction and social presence and thus foster learning and learner satisfaction.Student satisfaction surveys for Masters of Business Administration (MBA) degrees reveal that traditional online discussion forum questions do not encourage critical thinking compared with lively in-class interactions (Strang 2011: 80-92).The ability and necessity to develop a sense of community with virtual learners in the online environment is dependent upon the instructors and other learners sharing insights and knowledge (Menkhoff et al 2011: 132-151).Research shows that students who feel a sense of community in the online environment are more likely to continue in an online program than students who feel alone or alienated (Rovai 2003: 89-107).This, in turn, leads to an increase in high-quality e-learning and course completion rates and to the success of the academic program.
Current research suggests that an interactive, collaborative learning model is appropriate for online MBA programs due to the prior business experience of the learners and the convenient, asynchronous learning environment (Arbaugh & Rau 2007).Menkhoff et al's (2011: 132-151) study indicates that social presence is positively related to the students' desire to learn.Owston (1997) states that, in his experience teaching graduate-level courses, online discussions are often more thoughtful and reflective, with quiet or shy students more eager to interact than in the traditional classroom.Students have also reflected on the online learning process as a successful medium.Langan (1997: 129) writes of her online experiences, stating: "The online environment is much richer in resources and discussions founded in sound research and critical thinking than I have found in a local classroom environment".Berge and Collins (1995) concur, stating that online learning encourages an active environment of social learning in which learners can explore their own interests, interact with peers beyond their own local network, and communicate with peers in a learning atmosphere.
The importance of student-to-student interaction is essential to the establishment of the online learning community.Dawson et al (2006: 127-139) posit that increased student interaction is an indicator of the development of the community, and quantitative measures of student-tostudent interactions occurring online in discussion forums are useful in benchmarking and analysing online classes.In a quasi-experiment comparing two instructional approaches for an online MBA business strategy class, Strang (2011: 80-92) explored how discussion questions might be more effective in online MBA classes.He hypothesised and proved that more online experiential knowledge interactions had a positive effect on the students' grades and the number of online interactions.Menkhoff et al (2011: 132-151) hypothesised that the frequency of online interaction stimulated the desire to learn, and that student-to-student interaction stimulated interest in the topic.Their study of 138 online undergraduate business management students indicated a positive relationship between the e-learning experience and the frequency of online interaction.Arbaugh (2005: 135-149), in his study of 44 MBA online classes between 1998 to 2001, hypothesised that interaction between class participants would be positively associated with perceived learning and satisfaction with the MBA classes.His results strongly supported these hypotheses and indicated that encouraging participation and collaboration may enhance online learning.Swan (2001: 306-331) looked at 1,108 online students in 73 SUNY Learning Network courses in the spring semester of 1999.She found that students with higher levels of activity in the courses and high levels of student-to-student interaction reported higher levels of satisfaction and higher levels of learning from the classes.That is, the "greater the percentage of the course grade that was based on discussion, the more satisfied the students were, the more they thought they learned from the course, and the more interaction they thought they had with the instructor and with their peers" (Swan 2001:306-331).
The challenge facing the online instructor is in involving learners in the online discussion forums and engaging the students in quality discussions (Gilbert & Dabbagh 2005: 5-18).There appears to be an absence of a comprehensive, theory-driven metric that can assess quality and effectiveness of the discussion forum in online classrooms (Kay 2006: 761-783).
The amount of quality interaction between learners in online classes is a well-established indicator of virtual course effectiveness and it also positively predicts student learning and satisfaction (Arbaugh & Rau 2007).With a practice-based subject like project management, the design, development and facilitation of discussion forums becomes a critical component to the success of the educator's curriculum.The asynchronous environment allows for careful reflection and thoughtfulness in posts and an equal playing field for all students (Fauske & Wade 2004: 137-153).Through robust online discussions, virtual learners collaborate to share practical ideas, best practices, tools and techniques, as well as expand their learning experience and, therefore, increase their satisfaction with the curriculum.A student's use of terms in a post or increase in verbal activity alone may not indicate an increase in learning (Fauske & Wade 2004: 137-153); therefore, a consistent, fair and equitable means of assessing the interaction and quality of the posts, such as the use of a rubric, is necessary.Current research literature rarely discusses the kind and degree of structure that can be effective in developing quality student-to-student interactions and discussions.
A rubric is a scoring tool for a piece of work and can be a helpful tool for both the learner and the instructor (Goodrich 1996: 14-17), providing a guide to evaluate the quality of students' constructed responses using three features -evaluative criteria, quality definitions and a scoring strategy (Popham 1997: 72-75).In his study of 262 graduate-level students in 18 online courses, Rovai (2003: 89-107) found that grading strategies influence online discussions and that these discussions relate directly to the students' sense of community.
The rubric criteria in his study included the number of posts each week, the number of accesses to the discussion board weekly, and the content of the posts.His results showed that courses in which discussions were graded had significantly more discussions per student per week and higher levels of a sense of community, and also students who felt more satisfied with their education and the online experience.Benefits of using rubrics include: (a) efficiency in grading assignments, (b) validity and accuracy of grading, (c) similarity of grading tools used in the workplace, and (d) clarity of expectations for the student (Czaplewski 2009: 29-36).In addition, rubrics can clarify expectations for the instructor, help students focus on the work and provide unbiased feedback without excessive instructor grading (Goodrich Andrade 2005: 27-30).The result of using rubrics is often improvement in the quality of work and the learning experience for the student (Goodrich 1996: 14-17).
Several problems are cited in the literature regarding the design and use of rubrics.Among those listed by Czaplewski (2009: 29-36) are: (a) rubrics that are too complicated or detailed, (b) the learners' inability to agree with the grade, and (c) rubrics are often misused as a substitute for valuable teaching and mentoring opportunities.Other problems with rubrics can be that they are not entirely self-explanatory, they cannot replace good instruction, and students often are not good at self-assessment and, therefore, cannot properly use the rubric (Goodrich Andrade 2005: 27-30).Popham (1997: 72-75) suggests that flaws of rubrics include criteria that are not relevant or too general, missing the key attributes of the learning being assessed.
According to Czaplewski (2009: 29-36), a rubric must clearly outline the criteria and how each will be graded.Weltzer-Ward's (2011: 56-74) meta-analysis of online classroom discussions yielded several categories appropriate for development of the rubric for this introductory project management class.Some of her rubric categories included level of participation, interactivity, focus on or thoroughness of understanding the discussion prompt, professional tone, clarity and mechanics.The meta-analysis of online classroom discussions also suggests that the consensus of researchers is that the post (initial post responding to the discussion question and response posts to other learners) is the most appropriate and valid unit for analysis.Czaplewski (2009: 29-36) suggests comments can be written ahead by the professor and then copied and pasted into the feedback comments to increase the efficiency of the use of the rubric.
The discussion grading rubric and clear communication by the facilitator of the criteria in the rubric allows for a means of just assessment and evidence of the learner's interaction levels and quality of discussion posts.This study examines the impact of using such a discussion forum grading rubric to encourage and enhance discussions (posts) in the virtual project management classroom which, in turn, may assist educational facilities providing exceptional virtual project management classes and prompt the growth of the institution's online learning programs.

Comparable study
This study is comparable to and attempts to repeat that of Gilbert and Dabbagh's (2005: 5-18) work that examined the impact of online discussion protocols and evaluation rubrics on meaningful discourse in the online classroom.Their study involved 87 participants from four sections of a graduate course in instructional technology foundations and learning theory from spring 1999 to spring 2001.In the first section, protocols and evaluation criteria were minimal or loosely defined.In the remaining sections, the opposite was true.They reviewed how the addition of facilitator guidelines influenced meaningful discourse, how the posting of these guidelines influenced meaningful discourse, and how the addition of an evaluation rubric influenced meaningful discourse.Their analysis found that providing guidelines and using a rubric improved not only the facilitation and evaluation of the online discussions, but also increased the number and quality of the students' posts, improving them by students' abilities to: (a) promote and relate new knowledge to prior knowledge and experience, (b) interpret content, (c) analyse, synthesise and evaluate other students' posts, and (d) make inferences.Therefore, Gilbert and Dabbagh's (2005) study suggested that using facilitator guidelines, posting protocols explaining discussion board expectations relating to the quantity and quality of posts, and following evaluation criteria improved students' posts.
The current study is similar in that facilitator guidelines, posting protocols and evaluation criteria were also used although this study used different guidelines, protocols and evaluation criteria.The current study is set in a project management course whereas Gilbert and Dabbagh's (2005) study was set in an instructional technology foundations and learning theory course.However, the largest difference is that Gilbert and Dabbagh's (2005) study took place in a course in which students led the discussion board question and responses to other students, as this was a task in their syllabus as part of their grade.Students were required to form groups of two and each pair was assigned one week of leading the discussion board efforts.Thus, a different student team performed the facilitation of the discussion board each week.In the current study, the discussion board task was instructorled, being fully facilitated by the same instructor every week for all three sections.

Participants
This study consisted of 53 MBA students enrolled in three sections of an introduction to project management course from October 2010 to May 2011.The course is the first of four project management courses students are required to take in a series.MBA students are generally striving toward excellence in the workplace, and are willing to study to advance their careers and increase their opportunities (Li 2011: 201-220).They are commonly parttime students, looking not only to advance their careers, but also to understand theory and best practices (Elgort 2011: 97-108).
A control group of 20 students was compared to two experimental groups of 16 and 17 students respectively.The participants of the study, a deliberate convenience non-random sample, were students in three separate classes.They were considered to be a random sample, as they were not assigned to any of the three classes specifically by the university, and all students must fulfil identical requirements to be admitted into the MBA program and the course.

Rubric
The rubric, which included instructions designed to encourage online conversation, was used in the two experimental groups but not in the control group.The rubric was simplified to include five columns and the grading for each column was either 0 or 5, indicating that the learner either did not complete the task successfully (0) or completed the task successfully (5).This eliminated the learners' inability to agree with the grade.Issues with the rubric not being entirely self-explanatory were resolved by providing the rubric as an Excel spreadsheet.
The learners are able to self-evaluate the quality of their posts and responses, and the totals will sum automatically to show the score by discussion question and by the total week score.This also mitigated the potential problem that students may not be able to properly use the rubric.The criteria used in the rubric were carefully constructed to be relevant to the key attributes of the learning being assessed.For example, sharing personal experiences and expertise is important for learning at the MBA level.The personal experiences that the learner shares in posts are considered under the Example/Experience section of the rubric (5 points) and also under the Insight/Depth section of the rubric (5 points), so they contribute twice the weight in grading the discussion questions.
To compensate for the possible lack of professor feedback to help the learner understand the requirements for posts or misuse of the rubric as a substitute for valuable teaching and mentoring opportunities, the rubric scores were presented along with comments to help the learners.Below are a few examples of the comments used for this project management class: Great job this week!So much of what happens in project management is sharing ideas, best practices, tools and techniques.Thank you for making this happen in our discussion boards!In Week 4, go the extra mile and respond to more than just the minimum number (2) of learners for an outstanding week (-5 DQ#2 Response).
I did not see that you used the book as a reference.It's important to use our book as a resource for the other learners' reference purposes and as a basic denominator for our class.Be sure to include our text(s) as a reference in your response(s) (-5 DQ #1 Initial Post).
I noticed that you did not include outside references.It's important to use outside resource(s) for the other learners' reference purposes and to expand your own knowledge on the topic.I'd like to see you use some outside resources to round out your thinking on our topics (-5 DQ#1 Response, -5 DQ#2 Initial Post).
The rubric for this study scores the initial discussion post (in which the student answers a question posed by the facilitator) content on five levels: (1) if it is an insightful, thorough, and interesting analysis of material, whether or not the post includes new and useful  (Strang 2011: 80-92).
It also scores the content of the response posts written to other students on five levels.The first level (Insight/Depth) scores the response on the following: the response must be more than just agreement, it must answer another student's question(s) or lead that learner to a resource that will help answer his or her question(s); it must include in-depth evaluation or discussion, use personal stories, provide new resources, share examples or experiences that relate to the topic; and it must ask questions to encourage conversation and/or provide a contrasting view.It scores if the responder used a reference to the text and used additional academic references.Finally, the rubric scores the required number of responses to other learners, and increases the score (five points) when the learner responds to more than the minimum required number of learners.(See Appendix A for the rubric used in the two experimental groups.) Detailed instructions for the two experimental groups for the grading process and the rubric were presented to the participants the first week of class (see Appendix B).Virtual students often self-advise, misunderstand instructions and, therefore, become less committed (Menkhoff, et al. 2011: 132-151), so clear instructions were provided for the rubric.Each student was required to respond to the post that included the instructions and the copy of the rubric as an Excel worksheet by posting a response that showed that he or she had read the instructions and rubric.

Qualitative responses
To substantiate the quantitative results, student evaluation comments were captured that related directly to the use of the rubric in the project management class and related comments from the control group were also captured.These comments can be found in Appendix C. The comments from the experimental groups suggest that although hard work was required to get a good grade for the discussion board posts, students felt that it forced them to learn more by being held to a higher standard and by having to interact more with others in the class.The control group evaluation comments consisted of only two comments but these addressed appreciation for feedback in the grading process in terms of how to improve the quality of a discussion board post and thus improve a student's grade.The comments also suggest the need for more interaction resulting in more learning in the control group as well, which was the group without the rubric.

Statistical analysis
A statistical analysis was done comparing the average number of responses per student per discussion question among the three groups.Because the relative sample sizes were small and normality was not ensured, nonparametric statistics were used to compare the groups.First, a comparison was made on the average number of responses per question between the two experimental groups.The two groups were compared question by question (matching) over the entire semester of work.A matched pairs Wilcoxon Signed-Rank test was conducted because this test is used when comparing samples to assess whether their population mean ranks differ.No significant statistical difference was found between the two experimental groups (p = 0.272).Since no difference existed between the two experimental groups, they were combined and a comparison was made between the combined experimental group and the control group.A statistically significant difference (p = 0.000) was found using the Wilcoxon Rank-Sum Test because this test is used to assess whether one of two samples of independent observations tends to have larger values than the other.
Statistical analysis of the average of the four quiz grades for the students was also done using nonparametric statistics.First, the two experimental groups were compared using a Wilcoxon Rank-Sum Test, which revealed that there was no statistical difference between the two experimental groups at the p = 0.6013 level.Because the two experimental groups were not statistically different, they were combined into one experimental group.When comparing the combined experimental groups to the control group, there was also no significant difference if one is using α = 0.05.However, there would be a difference at the α = 0.10 level.

Instructor evaluations
The instructor evaluations for the control group and experimental groups were reviewed.
Histograms were created to display the student feedback collected at the end of each of the three sections, control group and two experimental groups.The histograms reflect the responses to those questions on the evaluation form that relate to any aspect of the discussion board questions.

Statistical analysis
The results indicate that the ratio of posts per discussion question to learners is considerably higher in the combined experimental group.The average number of responses over all questions over the semester for the combined experimental group was 3.88 while the average number of responses for the control group was 2.77.Table 1 shows that it is clear that the two groups are markedly different on all descriptive statistics except for their standard deviations.

Instructor evaluations
In reviewing the histograms presented in the results section, the experimental groups' instructor evaluations scored higher than the control group when combing percentages for the strongly agree and agree categories together, and the very effective to effective categories together.For instance, in combining the strongly agree with the agree categories, 84.2% of students in the control group found the course to be intellectually stimulating compared with 87.6% in the first experimental group and 94.12% in the second experimental group.And in combining the strongly agree with the agree categories, 73.7% of the students in the control group found the message board discussion to be valuable in their learning experience compared with 87.5% in the first experimental group and 76.47% in the second experimental group.Furthermore, the histograms show how students perceived their overall value of the course to be higher in the experimental groups than in the control group.When combining the very effective and the effective categories and using the average experimental percentage scores, students perceived both the quality of instruction and value of the course to be higher than those in the control group.The amount of time students spent on the discussion board was significantly higher in the experimental groups than in the control groups.In the control group, 31.6% of the students spent three or more hours per week posting on the discussion board.In the experimental groups, students spent 43.8% in the first experimental group and 70.59% in the second experimental group in terms of spending three or more hours on the discussion board content.

Implications and recommendations
This study is significant because it provides researched guidelines and a rubric that instructors can use to improve the quantity and quality of online discussions in project management or business class discussion forums.The research suggests that specific grading activities, such as the rubric, promote better quality discussions and a higher sense of community.Additionally, student-to-student interaction can provide online instructors with a strategy to increase student satisfaction in terms of their perceived value and quality of instruction that could aid in retention.The results suggest that the use of a rubric specifically designed to engage the student and expand the student's learning experience can increase online classroom participation and, as a consequence, enhance the educational experience of the online project management student, bolster the quality of the online program and enhance the reputation of the business school.The research outcome is important as the inclusion of effective grading rubrics for discussion forums in online project management courses represents a movement toward higher standards in the classroom, increased satisfaction and learning for the students, and improved quality and quantity of online discussions leading to an enhanced online project management program.Another example of excellence in the discussion classroom is stepping forward to answer another student's question(s) on a topic or leading that learner to a resource that will help answer his or her question(s).In addition, those who go beyond just doing the required number of responses to other students' postings show that their abilities are above average." Here is some additional information on how the 25% for each part is broken down even further in the grading rubric.(Also, see attached sample student rubric.) 25% DQ#1 post, 25% DQ#1 responses, 25% DQ#2 post, 25% DQ#2 responses.
Discussion posts should be insightful, thorough and interesting.They are insightful through your analysis of the material (5%) and thorough in your complete response (5%) to the question.An excellent discussion post demonstrates that you understand the topic and, if possible, shares personal examples and experiences on the subject (5%).Your post should be interesting by including new and useful information for the readers.The excellent post uses two or more examples from the textbook (5%) and other resources (journal articles, books, white papers, professional/academic web sites, etc.) (5%).Everyone in the class has the textbook, and bringing new knowledge and new references to the discussion distinguishes the higher-level posting.
Your responses to other students should demonstrate substantive analysis and/or evaluation of their discussion post.A response should be more than agreement and be an in-depth evaluation or discussion of the post (5%).Two responses to other learners are required for each discussion question (5%).(You are encouraged to continue conversations with your professor, but your responses to the professor will not count as responses to other students.)An excellent response includes personal stories, new resources, examples or experiences that relate to the topic, or asks questions regarding the response post to encourage conversation.Providing a contrasting view and developing your case for it using academic references is an example of an excellent response post.Providing new resources that relate to the response can do this (5%).Again, everyone has the text (5%); so providing information from the text is expected for a good response.
Another example of excellence in the discussion classroom is stepping forward to answer another student's question(s) on a topic or leading that learner to a resource that will help answer his or her question(s).In addition, those who go beyond just doing the required number of responses to other students' postings show that their abilities are above average (5%).
If you respond to more than the required minimum of two other learners, your additional posts (response posts #3, 4, 5, etc.) do not have to cite Portny or an outside source.You should, of course, substantiate your thoughts, so citations are encouraged.However, just normal conversation and sharing of ideas, tips and experiences is also acceptable for posts beyond the required minimum of two other learners.Two of your response postings are required to reference Portny and an outside source (see the grading rubric).Our goal in BUS 5661 is to learn from each other and our discussion forums are the way we can do this!Please let me know if you have any questions.We are working toward excellence in our postings in BUS 5661 and the sharing of information/resources/references is vital to the project management profession.

Figure 1 .
Figure 1.Student responses to the question "Hard work is required to get good grades in this course" 1--2 hr 2--3 hr 3-- YOU WILL NEED TO CONFIRM THAT YOU HAVE READ THIS DOCUMENT BY REPLYING TO THIS POSTING IN THE MESSAGE BOARD.Thank you.
(5) if the learner shares personal examples and experiences on the subject.These categories encourage the students to consider additional perspectives and deep learning, versus just cursory discussion information, and if the learner demonstrates understanding of the topic; (2) if it includes examples from the text; (3) if it includes examples of other academic resources; (4) completeness of response; and

Table 1 . Statistical analysis
Dr Deborah S. Carstens PMP®, joined Florida Tech in fall 2003 where she instructs both on campus and online courses.She is an Associate Professor in Information Systems and the Academic Chair for the Project Management Track in the Online MBA Program.Previously, she worked for NASA Kennedy Space Center (KSC) from 1992 through 2003.In her career both at Florida Tech and NASA, she has managed over a million dollars of funded research.She instructs undergraduate and graduate courses in MIS, human-computer interaction and project management.She has had an opportunity to work on funded research grants from the Small Business Administration, Florida Department of Education, Department of Health & Human Services, NASA and other locally funded studies in task analysis, human error, patient safety, usability, learning effectiveness and government accountability.She is currently working on a project management tools and techniques textbook being published in 2013.Email: carstens@fit.eduDr Linda C. Malone is a Professor Emeritus in Industrial Engineering from the University of Central Florida.She got her PhD degree in statistics from Virginia Tech after having gotten her BS and MS degrees in mathematics (at Emory and Henry College and University of Tennessee respectively).Her primary research interests include Response Surface Analysis and Quality.Malone has published 94 refereed articles, supervised 26 PhD student dissertations and authored one statistics textbook.She was an associate editor of the Journal of Statistical Computation and Simulation for over 25 years and was a founding co-editor of the STATS Magazine.She was awarded the honor of Fellow of the American Statistical Association.