Modelling and Using Response Times in Online Courses

Main Article Content

Ilia Rushkin
Isaac Chuang
Dustin Tingley

Abstract

Each time a learner in a self-paced online course seeks to answer an assessment question, it takes some time for the student to read the question and arrive at an answer to submit. If multiple attempts are allowed, and the first answer is incorrect, it takes some time to provide a second answer. Here we study the distribution of such “response times.” We find that the log-normal statistical model for such times, previously suggested in the literature, holds for online courses. Users who, according to this model, tend to take longer on submits are more likely to complete the course, have a higher level of engagement, and achieve a higher grade. This finding can be the basis for designing interventions in online courses, such as MOOCs, which would encourage “fast” users to slow down.

Article Details

How to Cite
Rushkin, I., Chuang, I., & Tingley, D. (2019). Modelling and Using Response Times in Online Courses. Journal of Learning Analytics, 6(3), 76–89. https://doi.org/10.18608/jla.2019.63.10
Section
Research Papers

References

Baker, F. B., & Kim, S. H. (2004). Item response theory: Parameter estimation techniques. Boca Raton, FL: CRC Press. http://dx.doi.org/10.1201/9781482276725

Beck, J. E. (2005). Engagement tracing: Using response times to model student disengagement. Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, 125, 88.

Bertling, M., & Chuang, I. (2015). Response time as a measure of motivation and applied effort in MOOCs. Unpublished manuscript.

Cetintas, S., Si, L., Xin, Y. P. P., & Hord, C. (2009). Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Transactions on Learning Technologies, 3(3), 228–236. http://dx.doi.org/10.1109/TLT.2009.44

Dai, Y. H., & Yuan, Y. (2001). An efficient hybrid conjugate gradient method for unconstrained optimization. Annals of Operations Research, 103(1–4), 33–47. http://dx.doi.org/10.1023/A:1012930416777

Goldstein, H. (2011). Multilevel statistical models (Vol. 922). Hoboken, NJ: John Wiley & Sons. http://dx.doi.org/10.1002/9780470973394

Grabe, M., & Sigler, E. (2002). Studying online: Evaluation of an online study environment. Computers & Education, 38(4), 375–383. http://dx.doi.org/10.1016/S0360-1315(02)00020-9

Grimmett, G., & Stirzaker, D. (2001). Probability and random processes. Oxford, UK: Oxford University Press.

Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory (Vol. 2). Thousand Oaks, CA: Sage.

Ho, A., Reich, J., Nesterko, S., Seaton, D., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses, fall 2012–summer 2013. HarvardX and MITx Working Paper No. 1. http://dx.doi.org/10.2139/ssrn.2381263

Lin, C., Shen, S., & Chi, M. (2016). Incorporating student response time and tutor instructional interventions into student modeling. Proceedings of the 24th Conference on User Modeling, Adaptation and Personalization (UMAP 2016), 13–16 July 2016, Halifax, Nova Scotia, Canada (pp. 157–161). New York: ACM. http://dx.doi.org/10.1145/2930238.2930291

Maris, E. (1993). Additive and multiplicative models for gamma distributed random variables, and their application as psychometric models for response times. Psychometrika, 58(3), 445–469. http://dx.doi.org10.1007/BF02294651

Roskam, E. E. (1987). Toward a psychometric theory of intelligence. In E. E. Roskam & R. Suck (Eds.), Progress in mathematical psychology, 1 (pp. 151–174). New York: Elsevier Science. https://psycnet.apa.org/record/1987-98116-009

Roskam, E. E. (1997). Models for speed and time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 187–208). New York: Springer. http://dx.doi.org/10.1007/978-1-4757-2691-6_11

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. http://dx.doi.org/10.2307/3002019

Scheiblechner, H. (1979). Specifically objective stochastic latency mechanisms. Journal of Mathematical Psychology, 19(1), 18–38. http://dx.doi.org/10.1016/0022-2496(79)90003-8

Scheiblechner, H. (1985). Psychometric models for speed-test construction: The linear exponential model. Test design: Developments in psychology and psychometrics, 219–244. http://dx.doi.org/10.1016/B978-0-12-238180-5.50012-4

Schnipke, D. L., & Scrams, D. J. (1999). Representing response-time information in item banks. Law School Admission Council Computerized Testing Report. LSAC Research Report Series.

Thissen, D. (1983). Timed testing: An approach using item response theory. In D. J. Weiss (Ed.), New horizons in testing: Latent trait test theory and computerized adaptive testing (pp. 179–203). Cambridge, MA: Academic Press. http://dx.doi.org/10.1016/B978-0-12-742780-5.50019-6

van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204. http://dx.doi.org/10.3102/10769986031002181

van der Linden, W. J., Scrams, D. J., & Schnipke, D. L. (1999). Using response-time constraints to control for differential speededness in computerized adaptive testing. Applied Psychological Measurement, 23(3), 195–210. http://dx.doi.org/10.1177/01466219922031329

Verhelst, N. D., Verstralen, H. H., & Jansen, M. G. H. (1997). A logistic model for time-limit tests. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 169–185). New York: Springer. http://dx.doi.org/10.1007/978-1-4757-2691-6_10

Wang, S., Zhang, S., Douglas, J., & Culpepper, S. (2018). Using response times to assess learning progress: A joint model for responses and response times. Measurement: Interdisciplinary Research and Perspectives, 16(1), 45–58. http://dx.doi.org/10.1080/15366367.2018.1435105

Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. http://dx.doi.org/10.1111/bmsp.12114