Early-warning analytics with LLM intervention rationales for student retention decisions: Classroom interaction modeling with xAPI-edu-data and dropout/success prediction
DOI:
https://doi.org/10.64268/inspire.v2i1.117Keywords:
Early-Warning Analytics, Educational Decision Support, Learning Analytics, Student Retention, XapiAbstract
Background: Student-retention early-warning systems have substantially improved predictive performance, yet their outputs often remain limited to risk scores that provide little guidance for educational intervention.
Aims: This study proposes and empirically evaluates a reproducible two-stage framework that integrates classroom interaction modeling, institutional retention prediction, and structured LLM-ready intervention rationales for student-retention decision support.
Methods: The framework was evaluated using two complementary benchmark datasets. xAPI-Edu-Data was used for classroom interaction modeling, whereas Predict Students' Dropout and Academic Success was used for institutional retention prediction. The datasets were analyzed independently rather than merged. Stratified train-test splits, ensemble machine learning models, feature ablation, feature-importance analysis, probability calibration, and rationale-quality evaluation were employed.
Result: LightGBM achieved the highest Macro-F1 (0.7775) on the xAPI-Edu-Data benchmark, while XGBoost produced the best overall performance on the institutional retention dataset, achieving a multiclass accuracy of 0.7672, a Macro-F1 of 0.6964, and a ROC-AUC of 0.8889. In binary dropout prediction, XGBoost achieved a ROC-AUC of 0.9360 and an Average Precision of 0.9101. Behavioral engagement, attendance, academic progression, and tuition-related variables consistently emerged as the most informative predictors. The structured rationale layer achieved complete evidence alignment, actionability, and monitoring specificity while generating 392 unique intervention rationales.
Conclusion: The proposed framework demonstrates that early-warning analytics can move beyond risk prediction by integrating predictive analytics, explainable AI, and structured intervention rationale generation into a transparent, evidence-grounded decision-support workflow for improving student-retention decisions in higher education.
References
ADL Initiative. (2013). Experience API (xAPI) specification (Version 1.0.1). Advanced Distributed Learning Initiative.
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138–52160. https://doi.org/10.1109/ACCESS.2018.2870052
Amrieh, E. A., Hamtini, T., & Aljarah, I. (2015). Preprocessing and analyzing educational data set using X-API for improving student's performance. In 2015 IEEE Jordan Conference on Applied Electrical Engineering and Computing Technologies (AEECT). IEEE. https://doi.org/10.1109/AEECT.2015.7360581
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). ACM. https://doi.org/10.1145/2330601.2330666
Baker, R. S., & Siemens, G. (2014). Educational data mining and learning analytics. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 253–274). Cambridge University Press. https://doi.org/10.1017/CBO9781139519526.016
Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17. https://doi.org/10.5281/zenodo.3554657
Bowers, A. J. (2010). Grades and graduation: A longitudinal risk perspective to identify student dropouts. The Journal of Educational Research, 103(3), 191–207. https://doi.org/10.1080/00220670903382970
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., ... Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review, 42(4), 40–57.
de Oliveira, C. F., Sobral, S. R., Ferreira, M. J., & Moreira, F. (2021). How does learning analytics contribute to prevent students' dropout in higher education: A systematic literature review. Big Data and Cognitive Computing, 5(4), Article 64. https://doi.org/10.3390/bdcc5040064
Dekker, G. W., Pechenizkiy, M., & Vleeshouwers, J. M. (2009). Predicting students dropout: A case study. In Proceedings of the 2nd International Conference on Educational Data Mining (pp. 41–50).
Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv. https://arxiv.org/abs/1702.08608
Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5–6), 304–317. https://doi.org/10.1504/IJTEL.2012.051816
Ifenthaler, D., & Yau, J. Y.-K. (2020). Utilising learning analytics to support study success in higher education: A systematic review. Educational Technology Research and Development, 68, 1961–1990. https://doi.org/10.1007/s11423-020-09788-z
Jayaprakash, S. M., Moody, E. W., Lauria, E. J. M., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47. https://doi.org/10.18608/jla.2014.11.3
Kasneci, E., Sessler, K., Kuchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Gunnemann, S., Hullermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., ... Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
Khosravi, H., Buckingham Shum, S., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gasevic, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, Article 100074. https://doi.org/10.1016/j.caeai.2022.100074
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.
Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an early warning system for educators: A proof of concept. Computers & Education, 54(2), 588–599. https://doi.org/10.1016/j.compedu.2009.09.008
OpenAI. (2023). GPT-4 technical report. arXiv. https://arxiv.org/abs/2303.08774
Rahmani, A. M., Groot, W., & Rahmani, H. (2024). Dropout in online higher education: A systematic literature review. International Journal of Educational Technology in Higher Education, 21, Article 19. https://doi.org/10.1186/s41239-024-00450-9
Realinho, V., Machado, J., Baptista, L., & Martins, M. V. (2022). Predicting student dropout and academic success. Data, 7(11), Article 146. https://doi.org/10.3390/data7110146
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Why should I trust you? Explaining the predictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations (pp. 97–101). Association for Computational Linguistics. https://doi.org/10.18653/v1/N16-3020
Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 40(6), 601–618. https://doi.org/10.1109/TSMCC.2010.2053532
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40.
Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125. https://doi.org/10.3102/00346543045001089
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). University of Chicago Press.
Wong, B. T.-M., & Li, K. C. (2020). A review of learning analytics intervention in higher education (2011–2018). Journal of Computers in Education, 7(1), 7–28. https://doi.org/10.1007/s40692-019-00143-7
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., & Gasevic, D. (2024). Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, 55(1), 90–112. https://doi.org/10.1111/bjet.13370
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Qi Xin
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
