Moving Forward with Learning Analytics: Expert Views

Main Article Content

Rebecca Ferguson
Doug Clow
Dai Griffiths
Andrew Brasher

Abstract

Learning analytics involve the measurement, collection, analysis, and reporting of data about learners and their contexts, in order to understand and optimise learning and the environments in which it occurs. Since emerging as a distinct field in 2011, learning analytics has grown rapidly, and early adopters around the world are already developing and deploying these new tools. This paper reports on a study that investigated how the field is likely to develop by 2025, in order to make recommendations for action to those concerned with the implementation of learning analytics. The study used a Policy Delphi approach, presenting a range of future scenarios to international experts in the field and asking for responses related to the desirability and feasibility of these scenarios, as well as actions that would be required. Responses were received from 103 people from 21 countries. Responses were coded thematically, inter-rater reliability was checked using Cohen’s kappa coefficient, and data were recoded if kappa was below 0.6. The seven major themes that were identified within the data were power, pedagogy, validity, regulation, complexity, ethics, and affect. The paper considers in detail each of these themes and its implications for the implementation of learning analytics.

Article Details

How to Cite
Ferguson, R., Clow, D., Griffiths, D., & Brasher, A. (2019). Moving Forward with Learning Analytics: Expert Views. Journal of Learning Analytics, 6(3), 43–59. https://doi.org/10.18608/jla.2019.63.8
Section
Research Papers
Author Biographies

Rebecca Ferguson, The Open University

Rebecca is a senior lecturer in the Institute of Educational Technology (IET) and a senior fellow of the Higher Education Academy. Her primary research interests are educational futures and how people learn together online.

Doug Clow, The Open University

Dr Doug Clow is a Senior Lecturer in the Institute of Educational Technology. He was Head of the Centre for the Study of Educational Technologies (CSET) between 2004 and 2008, and has over 25 years’ experience as a manager, contributor and Principal or Co-Investigator on technology-enhanced learning projects in science and other topic areas.

Dai Griffiths, The University of Bolton

Dai Griffiths is a professor in the School of Education and Psychology. His background is in the arts and in education, and he has taught at many levels including primary and secondary education, higher education and continuing education, and in industry.

Andrew Brasher, The Open University

Andrew works in the Learning and Teaching Development team in the Institute of Educational Technology at the Open University. His role is to get involved in a wide variety of research and development projects, with the aim of determining new tools and practices which will benefit the university's teaching and learning strategies.

References

Adams Becker, S., Freeman, A., Giesinger Hall, C., Cummins, M., & Yuhnke, B. (2016). NMC/CoSN Horizon Report: 2016 K-12 Edition. Austin, TX: NMC. https://www.nmc.org/publication/nmc-cosn-horizon-report-2016-k-12-edition/.

ADL. (2018). An organized list of ADL’s resources available on github. http://adlnet.github.io/.

ADL. (2017) xAPI-Spec. https://github.com/adlnet/xAPI-Spec.

Apereo. (2016). Apereo Open Analytics at a Glance. https://www.apereo.org/communities/learning-analytics-initiative/apereo-open-analytics-glance.

Arnold, K. E., & Pistilli, M. (2012). Course Signals at Purdue: Using learning analytics to increase student success. Paper presented at LAK ’12 (30 April–2 May), Vancouver, Canada. New York, USA: ACM. http://dx.doi.org/10.1145/2330601.2330666

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. http://dx.doi.org/10.1191/1478088706qp063oa

Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1

Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: a new tool for a new era. Educause Review, 42(4), 40–57. https://er.educause.edu/articles/2007/7/academic-analytics-a-new-tool-for-a-new-era

Clow, D., Cross, S., Ferguson, R. and Rienties, B. (2014). Evidence Hub Review. Milton Keynes: LACE Project. http://www.laceproject.eu/deliverables/d2-5-evidence-hub/

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, XX(1), 37–46. https://doi.org/10.1177/001316446002000104

Cormack, A. (2016). A data protection framework for learning analytics. Journal of Learning Analytics, 3(1), 91–106. https://doi.org/10.18608/jla.2016.31.6

Dalkey, N. C. (1969). The Delphi Method: An Experimental Study of Group Opinion. Santa Monica, CA: Rand Corporation. https://www.rand.org/pubs/research_memoranda/RM5888.html.

Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S. (2014). Current state and future trends: A citation network analysis of the learning analytics field. Paper presented at LAK ’14 (24–28 March), Indianapolis, IN. New York, USA: ACM. http://dx.doi.org/2567574.2567585

Drachsler, H., & Greller, W. (2012). The pulse of learning analytics: Understandings and expectations from stakeholders. Paper presented at LAK ’12 (29 April–2 May, 2012), Vancouver, Canada. New York, USA: ACM. http://dx.doi.org/10.1145/2330601.2330634

Drachsler, H., & Greller, W. (2016). Privacy and analytics: It’s a DELICATE issue. A checklist for trusted learning analytics. Paper presented at LAK ’16 (25–29 April, 2016), Edinburgh, UK. New York, USA: ACM. http://dx.doi.org/10.1145/2883851.2883893

Dringus, L. P. (2012). Learning analytics considered harmful. Journal of Asynchronous Learning Networks, 16(3), 87–100.

Engelfriet, A., Manderveld, J., & Jeunink, E. (2017). Learning Analytics Under the Dutch Data Protection Act. SURFnet. https://www.surf.nl/handreiking-learning-analytics-onder-de-wet-bescherming-persoonsgegevens

Essa, A. (2019) Is data dark? Lessons from Borges’s ‘Funes the Memorius’. Journal of Learning Analytics, 6(3), 35–42. http://dx.doi.org/10.18608/jla.2019.63.7

European Parliament and Council. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data. http://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679.

Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning (IJTEL), 4(5/6), 304–317.

Ferguson, R. (2019). Ethical challenges for learning analytics. Journal of Learning Analytics, 6(3), 25–30. http://dx.doi.org/10.18608/jla.2019.63.5

Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy — Study. Joint Research Centre Science for Policy Report. Seville, Spain: EUR 28294. http://dx.doi.org/10.2791/955210

Ferguson, R., Cooper, A., Drachsler, H., Kismihok, G., Boyer, A., Tammets, K., & Martinez Mones, A. (2015). Learning analytics: European perspectives. Paper presented at LAK ’15 (16–20 March, 2015), Poughkeepsie, NY, USA. New York, USA: ACM. http://dx.doi.org/10.1145/2723576.2723637

Ferguson, R., Hoel, T., Scheffel, M., & Drachsler, H. (2016). Guest editorial: Ethics and privacy in learning analytics. Journal of Learning Analytics, 3(1), 5–15. https://doi.org/10.18608/jla.2016.31.2

Ferguson, R., Macfadyen, L. P., Clow, D., Tynan, B., Alexander, S., & Dawson, S. (2015). Setting learning analytics in context: Overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120–144. https://doi.org/10.18608/jla.2014.13.7

Goodyear, P., Banks, S., Hodgson, V., & McConnell, D. (Eds.). (2004). Advances in Research on Networked Learning. Norwell, Massachusetts, USA: Kluwer Academic Publishers. http://dx.doi.org/10.1007/1-4020-7909-5

Griffiths, D., Brasher, A., Clow, D., Ferguson, R., & Yuan, L. (2016). Visions of the future: Horizon report. http://www.laceproject.eu/deliverables/d3-2-visions-of-the-future-2/

Hernández Leo, D., Rodríguez-Triana, M., Inventado, P., Mor, Y. (Eds.). 2017. Special issue of IxD&A Connecting Learning Design and Learning Analytics. http://ixdea.uniroma2.it/inevent/events/idea2010/index.php?s=10&a=10&link=ToC_33_P

Hoel, T., & Chen, W. (2016). Privacy-driven design of learning analytics applications: exploring the design space of solutions for data sharing and interoperability. Journal of Learning Analytics, 3(1), 139–158.

Hoel, T., Griffiths, D., & Chen, W. (2017). The influence of data protection and privacy frameworks on the design of learning analytics systems. Paper presented at LAK ’17, Vancouver, British Columbia, Canada. New York, USA: ACM. https://doi.org/10.1145/3027385.3027414

IMS Global Learning Consortium. (2018). Caliper Analytics Specification, version 1.1. https://www.imsglobal.org/sites/default/files/caliper/v1p1/caliper-spec-v1p1/caliper-spec-v1p1.html

Jisc. (n.d.). Learning Analytics. Harness the Power of Your Data. https://www.jisc.ac.uk/learning-analytics

Jiscdev. (n.d.). Jisc Learning Analytics Statement Templates. https://github.com/jiscdev/xapi

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Hall, C. (2016). NMC Horizon Report: 2016 Higher Education Edition. Austin, Texas, USA: NMC. https://www.nmc.org/publication/nmc-horizon-report-2016-higher-education-edition/.

Knight, S., Buckingham Shum, S., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://doi.org/10.18608/jla.2014.12.3

Kurzweil, R. (2005). The Singularity Is Near. London, UK: Duckworth.

LACE. (n.d.). Ethics and Privacy in Learning Analytics. http://www.laceproject.eu/ethics-privacy-learning-analytics/

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. http://dx.doi.org/10.2307/2529310

Linstone, H. A., & Turoff, M. (2002). The Delphi Method: Techniques and Applications. Boston, Massachusetts, USA: Addison-Wesley.

Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 31–40. https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education

Macfadyen, L., & Dawson, S. (2012). Numbers are not enough. Why e-learning analytics failed to inform an institutional strategic plan. Educational Technology & Society, 15(3), 149–163. https://www.jstor.org/stable/jeductechsoci.15.3.149

Macfadyen, L., Dawson, S., Pardo, A., & Gašević, D. (2014). Embracing Big Data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9(Winter), 17–28. https://eric.ed.gov/?id=EJ1062692

McIntosh, N. E. (1979). Barriers to implementing research in higher education. Studies in Higher Education, 4(1), 77–86. http://dx.doi.org/10.1080/03075077912331377121

National Academy of Education. (2017). Big Data in Educaton, Balancing the Benefits of Educational Research and Student Privacy. http://naeducation.org/wp-content/uploads/2017/05/NAEd_BD_Booklet_FINAL_051717_3.pdf

Orwell, G. (1949). Nineteen Eighty-Four. London: Secker & Warburg.

Perry, M. R., et al. (2018). SPEC Kit 360. https://publications.arl.org/Learning-Analytics-SPEC-Kit-360/.

Prinsloo, P. (2019). Commentary on Neil Selwyn’s LAK18 keynote address. Journal of Learning Analytics, 6(3), 20–24. http://dx.doi.org/10.18608/jla.2019.63.4

Prinsloo, P., & Slade, S. (2013). An evaluation of policy frameworks for addressing ethical considerations in learning analytics. Paper presented at LAK ’13 (8–12 April 2013), Leuven, Belgium. New York, USA: ACM. http://dx.doi.org/10.1145/2460296.2460344

Prinsloo, P., & Slade, S. (2017). Ethics and learning analytics: Charting the (un)charted. In Lang, C. et al. (Eds.), Handbook of Learning Analytics (pp. 49–57). SoLAR. https://doi.org/10.18608/hla17

Rosé, C. P. (2019). Monolith, multiplicity, or multivocality: What do we stand for and where do we go from here? Journal of Learning Analytics, 6(3), 31–34. http://dx.doi.org/10.18608/jla.2019.63.6

Sclater, N., & Mullan, J. (2017). Learning Analytics and Student Success — Assessing the Evidence. Bristol, UK: Jisc. https://repository.jisc.ac.uk/6560/1/learning-analytics_and_student_success.pdf.

Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. http://dx.doi.org/10.18608/jla.2019.63.3

Sharples, M. (2000). The design of personal mobile technologies for lifelong learning. Computers & Education, 34(3-4), 177–193. https://dx.doi.org/10.1016/S0360-1315(99)00044-5

Sharples, M., de Roock, R., Ferguson, R., Gaved, M., Herodotou, C., Koh, E., Kukulska-Hulme, A., Looi, C-K., McAndrew, P., Rienties, B., Weller, M. & Wong, L. H. (2016). Innovating Pedagogy 2016: Open University Innovation Report 5. Milton Keynes, UK: The Open University. https://iet.open.ac.uk/file/innovating_pedagogy_2016.pdf.

Siemens, G. (2005). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1).

Siemens, G., Gašević, D., Haythornthwaite, C., Dawson, S., Buckingham Shum, S., Ferguson, R., Duval, E., Verbert, K., & Baker, R. S. J. d. (2011). Open Learning Analytics: An Integrated and Modularized Platform (Concept Paper). https://solaresearch.org/core/open-learning-analytics-an-integrated-modularized-platform/

Slade, S., & Prinsloo, P. (2015). Student perspectives on the use of their data: Between intrusion, surveillance and care. European Journal of Distance and E-learning, 18(1). http://www.eurodl.org/index.php?p=special&sp=articles&inum=6&abstract=672&article=679

Tsai, Y. S., Moreno-Marcos, P. M., Tammets, K., Kollom, K., & Gašević, D. (2018, March). SHEILA policy framework: Informing institutional strategies and policy processes of learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 320–329). ACM. http://dx.doi.org/10.1145/3170358.3170367

Turoff, M. (1970). The Design of a Policy Delphi, reprinted in The Science of Public Policy, vol. IV, Policy Analysis II (2002), T. Miyakawa (Ed.), pp. 488–517. https://dx.doi.org/10.1016/0040-1625(70)90161-7

Turoff, M. (2002). The Policy Delphi. In H. A. Linstone & M. Turoff (Eds.), The Delphi Method: Techniques and Applications (pp. 80–96). Boston, Massachusetts, USA: Addison-Wesley.