Measuring Mastery Behaviors at Scale: The Persistence, Effort, Resilience and Challenge-Seeking Task (PERC)

Main Article Content

Tenelle Porter
Diego Catalán Molina
Lisa Blackwell
Sylvia Roberts
Abigail Quirk
Angela Lee Duckworth
Kali Trzesniewski

Abstract

Mastery behaviours — seeking out challenging tasks and continuing to work on them despite difficulties — are integral to achievement but difficult to measure with precision. The current study reports on the development and validation of the computer-based persistence, effort, resilience, and challenge-seeking (PERC) task in two demographically diverse samples of adolescents (total N = 3,188). We present evidence for convergent validity with self-reported mastery behaviours and learning mindsets, discriminant validity from theoretically unrelated constructs, and incremental predictive validity for grade point average (GPA). Results suggest that the PERC task may be a useful tool for assessing mastery behaviours at scale.

Article Details

How to Cite
Porter, T., Catalán Molina, D., Blackwell, L., Roberts, S., Quirk, A., Lee Duckworth, A., & Trzesniewski, K. (2020). Measuring Mastery Behaviors at Scale: The Persistence, Effort, Resilience and Challenge-Seeking Task (PERC). Journal of Learning Analytics, 7(1), 5–18. https://doi.org/10.18608/jla.2020.71.2
Section
Special Section: Beyond Cognitive Ability: Enabling Assessment of 21C Skills

References

Abrahams, L., Pancorbo, G., Primi, R., Santos, D., Kyllonen, P., John, O. P., & De Fruyt, F. (2019). Social-emotional skill assessment in children and adolescents: Advances and challenges in personality, clinical, and educational contexts. Psychological Assessment, 31(4), 460–473. https://dx.doi.org/10.1037/pas0000591

Arendasy, M., Sommer, M., Herle, M., Schützhofer, B., & Inwanschitz, D. (2011). Modeling effects of faking on an objective personality test. Journal of Individual Differences, 32, 210–218. https://dx.doi.org/10.1027/1614-0001/a000053

Aron, A., Aron, E. N., & Smollan, D. (1992). Inclusion of other in the self scale and the structure of interpersonal closeness. Journal of Personality and Social Psychology, 63, 596–612. https://dx.doi.org/10.1037/0022-3514.63.4.596

Bettinger, E., Ludvigsen, S., Rege, M., Solli, I. F., & Yeager, D. (2018). Increasing perseverance in math: Evidence from a field experiment in Norway. Journal of Economic Behavior & Organization, 146, 1–15. https://dx.doi.org/10.1016/j.jebo.2017.11.032

Blackwell, L., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246–263. https://dx.doi.org/10.1111/j.1467-8624.2007.00995.x

Brown, E. D. (2009). Persistence in the face of academic challenge for economically disadvantaged preschool children. Journal of Early Childhood Research, 7(2), 175–186. https://dx.doi.org/10.1177/1476718X09102650

Burnette, J. L., O’Boyle, E. H., VanEpps, E. M., Pollack, J. M., & Finkel, E. J. (2013). Mind-sets matter: A meta-analytic review of implicit theories and self-regulation. Psychological Bulletin, 139(3), 655–701. https://dx.doi.org/10.1037/a0029531

Caspi, A., Elder, G. H., & Bem, D. J. (1988). Moving away from the world: Life-course patterns of shy children. Developmental Psychology, 24(6), 824–831. https://dx.doi.org/10.1037/0012-1649.24.6.824

Choudhury, N., & Gorman, K. S. (1999). The relation between reaction time and psychometric intelligence in a rural Guatemalan adolescent population. International Journal of Psychology, 34(4), 209–217. https://dx.doi.org/10.1080/002075999399855

Clark, L. A., & Watson, D. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412–1427. https://dx.doi.org/10.1037/pas0000626

Cohen, P., Cohen, J., Aiken, L. S., & West, S. (1999). The problem of units and the circumstance for POMP. Multivariate Behavioral Research. 34(3), 315–346. https://dx.doi.org/10.1207/S15327906MBR3403_2

Costenbader, V., & Ngari, S. M. (2001). A Kenyan standardization of the Raven’s coloured progressive matrices. School Psychology International, 22(3), 258–268. https://dx.doi.org/10.1177/0143034301223003

Cutumisu, M., Blair, K. P., Chin, B., & Schwartz, D. L. (2015). Posterlet: A game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2(1), 49–71. https://dx.doi.org/10.18608/jla.2015.21.4

Diener, C. I., & Dweck, C. S. (1978). An analysis of learned helplessness: Continuous changes in performance, strategy, and achievement cognitions following failure. Journal of Personality and Social Psychology, 36(5), 451–462. https://dx.doi.org/10.1037/0022-3514.36.5.451

Diener, C. I., & Dweck, C. S. (1980). An analysis of learned helplessness: II. The processing of success. Journal of Personality and Social Psychology, 39(5), 940–952. https://dx.doi.org/10.1037/0022-3514.39.5.940

Diener, E., Ingelhart, R. F., & Tay, L. (2013). Theory and validity of life satisfaction scales. Social Indicators Research, 112(5), 497–527. https://dx.doi.org/10.1007/s11205-012-0076-y

Duckworth, A. L., & Kern, M. L. (2011). A meta-analysis of the convergent validity of self-control measures. Journal of Research in Personality, 45(3), 259–268. https://dx.doi.org/10.1016/j.jrp.2011.02.004

Duckworth, A. L., & Yeager, D. S. (2015). Measurement matters: Assessing personal qualities other than cognitive ability for educational purposes. Educational Researcher, 44(4), 237–251. https://doi.org/10.3102/0013189X15584327

Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. New York: Psychology Press, Taylor & Francis Group. https://doi.org/10.4324/9781315783048

Dweck, C. S., Chiu, C., & Hong, Y. (1995). Implicit theories and their role in judgments and reactions: A world from two perspectives. Psychological Inquiry, 6(4), 267–285. https://dx.doi.org/10.1207/s15327965pli0604_1

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation and personality. Psychological Review, 95(2), 256–273. https://dx.doi.org/10.1037/0033-295X.95.2.256

Eisinga, R., te Grotenhuis, M., & Pelzer, B. (2013). The reliability of a two-item scale: Person, Cronbach, or Spearman-Brown? International Journal of Public Health, 58(4), 637–642. https://dx.doi.org/10.1007/s00038-012-0416-3

Elliot, A., McGregor, H., & Gable, S. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 91, 549–563. https://dx.doi.org/10.1037/0022-0663.91.3.549

Enkavi, Z. A, Eisenberg, I. W., Bissett, P. G., Mazza, G. L., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019). Large-scale analysis of test–retest reliabilities of self-regulation measures. Proceedings of the National Academy of Sciences of the United States of America, 116(12), 5472–5477. https://dx.doi.org/10.1073/pnas.1818430116

Funder, D. C., & Ozer, D. J. (2019). Evaluating effect sizes in psychological research: Sense and nonsense. Advances in Methods and Practices in Psychological Science, 2(2), 156–168. https://dx.doi.org/10.1177/2515245919847202

Galla, B. M., Plummer, B. D., White, R. E., Meketon, D., D’Mello, S. K., & Duckworth, A. L. (2014). The academic diligence task (ADT): Assessing individual differences in effort on tedious but important schoolwork. Contemporary Education Psychology, 39(4), 314–325. https://dx.doi.org/10.1016/j.cedpsych.2014.08.001

Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the discovery of learner metacognition from reflective writing. Journal of Learning Analytics, 3(2), 22–36. https://dx.doi.org/10.18608/jla.2016.32.3

Gottfried, A. E. (1985). Academic intrinsic motivation in elementary and junior high school students. Journal of Educational Psychology, 77(6), 631–645. https://dx.doi.org/10.1037/0022-0663.77.6.631

Haimovitz, K., & Dweck, C. S. (2016). Parents’ views of failure predict children’s fixed and growth intelligence mind-sets. Psychological Science, 27(6), 859–869. https://dx.doi.org/10.1177/0956797616639727

Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavioral Research Methods, 50(3), 1166–1186. https://dx.doi.org/10.3758/s13428-017-0935-1

Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3), 61–83. https://dx.doi.org/10.1017/S0140525X0999152X

Joksimović, S., Gašević, D., Loughin, T. M., Kovanović, V., & Hatala, M. (2015). Learning at distance: Effects of interaction traces on academic achievement. Computers & Education, 87, 204–217. https://dx.doi.org/10.1016/j.compedu.2015.07.002

Knafo, A., Israel, S., Darvasi, A., Bachner-Melman, R., Uzefovsky, F., Cohen, L., …, Ebstein, R. P. (2007). Individual differences in allocation of funds in the dictator game associated with length of the arginine vasopressin 1a receptor RS3 promoter region and correlation between RS3 length and hippocampal mRNA. Genes Brain Behavior, 7(3), 266–275. https://dx.doi.org/10.1111/j.1601-183X.2007.00341.x

Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the black box of time-on-task estimation, Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK ’15), 16–20 March 2015, Poughkeepsie, NY (pp. 184–193). New York: ACM. https://dx.doi.org/10.1145/2723576.2723623

Kraft, M. A. (2019). Interpreting effect sizes of education interventions. EdWorkingPaper No. 19-10. https://dx.doi.org/10.26300/8pjp-2z74

Krumm, A. E., Beattie, R., Takahashi, S., D’Angelo, C., Feng, M., & Cheng, B. (2016). Practical measurement and productive persistence: Strategies for using digital learning system data to drive improvement. Journal of Learning Analytics, 3(2), 116–138. https://dx.doi.org/10.18608/jla.2016.32.6

Licht, B. G., & Dweck, C. S. (1984). Determinants of academic achievement: The interaction of children’s achievement orientations with skill area. Developmental Psychology, 20(4), 628–636. https://dx.doi.org/10.1037/0012-1649.20.4.628

Lucas, R. E., & Donnellan, B. M. (2012). Estimating the reliability of single-item life satisfaction measures: Results from four national panel studies. Social Indicators Research, 105(3), 323–331. https://dx.doi.org/10.1007/s11205-011-9783-z

Mangels, J. A., Butterfield, B., Lamb, J., Good, C., & Dweck, C. (2006). Why do beliefs about intelligence influence learning success? A social cognitive neuroscience model. Social Cognitive and Affective Neuroscience, 1(2), 75–86. https://dx.doi.org/10.1093/scan/nsl013

Marcus, B. (2006). Relationships between faking, validity, and decision criteria in personnel selection. Psychology Science, 48(3), 226–246. https://psycnet.apa.org/record/2006-21801-003

Martin, C. K., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 37–87. https://dx.doi.org/10.18608/jla.2016.32.4

Meindl, P., Yu, A., Galla, B., Quirk, A., Haeck, C., Parker, G. J., …, Duckworth, A. L. (2019). A brief behavioral measure of frustration tolerance predicts academic achievement immediately and two years later. Emotion, 19(6), 1081–1092. https://dx.doi.org/10.1037/emo0000492

Miller, R., Greene, B., Montalvo, G., Ravindran, B., & Nichols, J. (1996). Engagement in academic work: The role of learning goals, future consequences, pleasing others, and perceived ability. Contemporary Educational Psychology, 21, 388–422. https://dx.doi.org/10.1006/ceps.1996.0028

Mischel, W., Shoda, Y., & Peake, P. K. (1988). The nature of adolescent competencies predicted by preschool delay of gratification. Journal of Personality and Social Psychology, 54, 687–696. https://dx.doi.org/10.1037/0022-3514.54.4.687

Moser, J. S., Schroder, H. S., Heeter, C., Moran, T. P., & Lee, Y. (2011). Mind your errors: Evidence for a neural mechanism linking growth mind-set to adaptive posterror adjustments. Psychological Science, 22(12), 1484–1489. https://dx.doi.org/10.1177/0956797611419520

Mueller, C. M., & Dweck, C. S. (1998). Praise for intelligence can undermine children’s motivation and performance. Journal of Personality and Social Psychology, 75(1), 33–52. https://dx.doi.org/10.1037/0022-3514.75.1.33

Raven, J., Raven, J. C., & Court, J. H. (1998). Manual for Raven’s progressive matrices and vocabulary scales. Oxford, U.K.: Oxford Psychologists Press.

Robins, R. W., & Pals, J. L. (2002). Implicit self-theories in the academic domain: Implications for goal orientation, attributions, affect, and self-esteem change. Self and Identity, 1(4), 313–336. https://dx.doi.org/10.1080/15298860290106805

Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://dx.doi.org/10.1177/0002764213498851

Soland, J., Zamarro, G., Cheng, A., & Hitt, C. (2019). Identifying naturally occurring direct assessments of social-emotional competencies: The promise and limitations of survey and assessment disengagement metadata. Educational Researcher, 48(7), 466–478. https://dx.doi.org/10.3102/0013189X19861356

Vazire, S., & Mehl, M. R. (2008). Knowing me, knowing you: The accuracy and unique predictive validity of self-ratings and other-ratings of daily behavior. Journal of Personality and Social Psychology, 95(5), 1202–1216. https://dx.doi.org/10.1037/a0013314

Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29, 2568–2572. https://dx.doi.org/10.1016/j.chb.2013.06.033

Ventura, M., Shute, V., & Zhao, W. (2013). The relationship between video game use and a performance-based measure of persistence. Computers & Education, 60(1), 52–58. https://dx.doi.org/10.1016/j.compedu.2012.07.003

Walton, G. M., & Cohen, G. L. (2007). A question of belonging: Race, social fit, and achievement. Journal of Personality and Social Psychology, 92(1), 82–96. https://dx.doi.org/10.1037/0022-3514.92.1.82

Willerman, L., Turner, R. G., & Peterson, M. (1976). A comparison of the predictive validity of typical and maximal personality measures. Journal of Research in Personality, 10(4), 482–492. https://dx.doi.org/10.1016/0092-6566(76)90063-5

Yeager, D. S., & Bryk, A. (2015). Practical measurement. Unpublished manuscript, University of Texas at Austin.

Yeager, D. S., Romero, C., Paunesku, D., Hulleman, C. S., Schneider, B., Hinojosa, C., …, Dweck, C. S. (2016). Using design thinking to improve psychological interventions: The case of the growth mindset during the transition to high school. Journal of Educational Psychology, 108(3), 374–391. https://dx.doi.org/10.1037/edu0000098

Ziegler, M., Schmidt-Atzert, L., Bühner, M., & Krumm, S. (2007). Fakability of different measurement methods for achievement motivation: Questionnaire, semi-projective, and objective. Psychology Science, 49, 291–307. https://psycnet.apa.org/record/2008-02884-001

DB Error: Unknown column 'Array' in 'where clause'