From Prediction to Personalized Intervention: An Explainable, Lightweight SHAP-Driven Ensemble Framework for Early Academic Failure Risk Assessment
Article Sidebar
Main Article Content
Abstract
Early identification of students at risk remains one of the most critical problems in e-learning learning, and most of the prediction models sacrifice accuracy to be interpretable, and are expensive to implement using deep learning architectures that are not feasible in the educational systems with limited resources. This paper proposes a model to interpret the model globally and instance-level using SHapley Additive exPlanations (SHAP) along with three machine learning algorithms (Random Forest (RF), XGBoost, and Gradient Boosting (GB)) in a lightweight, easy to explain academic performance prediction framework. We use three publicly available datasets to test the framework: the UCI Student Performance Dataset (649 records, 33 features), Open University Learning Analytics Dataset — OULAD — (32,593 records, 23 features), and the Kaggle Student Habits Dataset (1,000 records, 20 features). Data leakage is prevented by the Synthetic Minority Oversampling Technique (SMOTE) used during training folds and hyperparameters are tuned using RandomizedSearchCV with five-fold stratified cross-validation. The proposed ensemble achieves 94.7% accuracy, 0.943 F1 score and 0.971 AUC-ROC in 1.8 seconds of CPU training time (470× faster than the state of the art deep learning models LSTM and CNN+LSTM, and without requiring any GPU hardware). At the population level, SHAP is a measure of the most important predictive features of the model and automatically provides actionable intervention recommendations for each at-risk student. The Wilcoxon signed-rank test is used to establish the statistical significance (p<0.01) and an ablation study is used to measure the contribution of each component of the system. It should be deployable on common laptop computers and be based on open and free tools, accessible for institutions with restricted computational resources, and explored in relation to the fairness, privacy and trustworthy-AI requirements for educational deployment.
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
[1] UNESCO, "Education in a post-COVID world: Nine ideas for public action," UNESCO, Paris, France, Rep., 2023.
[2] L. P. Macfadyen and S. Dawson, "Mining LMS data to develop an 'early warning system' for educators," Comput. Educ., vol. 54, no. 2, pp. 588–599, Feb. 2010, doi: 10.1016/j.compedu.2009.09.008.
[3] C. Romero and S. Ventura, "Educational data mining and learning analytics: An updated survey," Wiley Interdiscip. Rev. Data Min. Knowl. Discov., vol. 10, no. 3, p. e1355, May 2020, doi: 10.1002/widm.1355.
[4] A. Adadi and M. Berrada, "Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)," IEEE Access, vol. 6, pp. 52138–52160, 2018, doi: 10.1109/ACCESS.2018.2870052.
[5] S. M. Lundberg and S.-I. Lee, "A unified approach to interpreting model predictions," in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), Long Beach, CA, USA, 2017, pp. 4765–4774.
[6] P. Cortez and A. Silva, "Using data mining to predict secondary school student performance," in Proc. 5th Future Bus. Technol. Conf. (FUBUTEC), Porto, Portugal, 2008, pp. 5–12.
[7] S. B. Kotsiantis, C. Pierrakeas, and P. E. Pintelas, "Predicting students' performance in distance learning using machine learning techniques," Appl. Artif. Intell., vol. 18, no. 5, pp. 411–426, 2004, doi: 10.1080/08839510490442058.
[8] T. Chen and C. Guestrin, "XGBoost: A scalable tree boosting system," in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., San Francisco, CA, USA, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
[9] S. A. Hussain, M. A. Dahan, F. A. Ba-Alwi, and N. Ribata, "Educational data mining and analysis of students' academic performance using WEKA," Indonesian J. Electr. Eng. Comput. Sci., vol. 9, no. 2, pp. 447–459, 2018, doi: 10.11591/ijeecs.v9.i2.pp447-459.
[10] L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5–32, Oct. 2001, doi: 10.1023/A:1010933404324.
[11] F. Dalipi, A. S. Imran, and Z. Kastrati, "MOOC dropout prediction using machine learning techniques: Review and research challenges," in Proc. IEEE Global Eng. Educ. Conf. (EDUCON), Santa Cruz de Tenerife, Spain, 2018, pp. 1007–1014.
[12] M. Hlosta, Z. Zdrahal, and J. Zendulka, "Ouroboros: Early identification of at-risk students without models based on legacy data," in Proc. 7th Int. Learn. Anal. Knowl. Conf. (LAK), Vancouver, BC, Canada, 2017, pp. 6–15.
[13] M. T. Ribeiro, S. Singh, and C. Guestrin, "“Why should I trust you?” Explaining the predictions of any classifier," in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., San Francisco, CA, USA, 2016, pp. 1135–1144, doi: 10.1145/2939672.2939778.
[14] Y. Choi, C. S. Yeo, and J. J. Jung, "Attention-based LSTM for student dropout prediction in MOOC," in Proc. IEEE Int. Conf. Big Data Smart Comput. (BigComp), Shanghai, China, 2019, pp. 1–5.
[15] S. M. Lundberg et al., "From local explanations to global understanding with explainable AI for trees," Nat. Mach. Intell., vol. 2, pp. 56–67, Jan. 2020, doi: 10.1038/s42256-019-0138-9.
[16] J. Kuzilek, M. Hlosta, and Z. Zdrahal, "Open University Learning Analytics dataset," Sci. Data, vol. 4, art. 170171, Nov. 2017, doi: 10.1038/sdata.2017.171.
[17] A. Aryan, "Student Habits and Academic Performance Dataset," Kaggle, 2023. [Online]. Available: https://www.kaggle.com/datasets/aryan208/student-habits-and-academic-performance-dataset. [Accessed: Jul. 14, 2026].
[18] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "SMOTE: Synthetic minority over-sampling technique," J. Artif. Intell. Res., vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953.
[19] High-Level Expert Group on AI, "Ethics guidelines for trustworthy AI," European Commission, Brussels, Belgium, 2019.
[20] R. Conijn, C. Snijders, A. Kleingeld, and U. Matzat, "Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS," IEEE Trans. Learn. Technol., vol. 10, no. 1, pp. 17–29, Jan. 2017, doi: 10.1109/TLT.2016.2616312.
[21] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, "A survey on bias and fairness in machine learning," ACM Comput. Surv., vol. 54, no. 6, art. 115, 2021, doi: 10.1145/3457607.
[22] S. Batool, J. Rashid, M. W. Nisar, J. Kim, H.-Y. Kwon, and A. Hussain, "Educational data mining to predict students’ academic performance: A survey study," Educ. Inf. Technol., vol. 28, no. 1, pp. 905–971, 2023, doi: 10.1007/s10639-022-11152-y.
[23] A. Khan and S. K. Ghosh, "Student performance analysis and prediction in classroom learning: A review of educational data mining studies," Educ. Inf. Technol., vol. 26, pp. 205–240, 2021, doi: 10.1007/s10639-020-10230-3.
[24] A. Asselman, M. Khaldi, and S. Aammou, "Enhancing the prediction of student performance based on the machine learning XGBoost algorithm," Interact. Learn. Environ., vol. 31, no. 6, pp. 3360–3379, 2023, doi: 10.1080/10494820.2021.1928235.
[25] R. Alamri and B. Alharbi, "Explainable student performance prediction models: A systematic review," IEEE Access, vol. 9, pp. 33132–33143, 2021, doi: 10.1109/ACCESS.2021.3061368.
[26] G. Deeva, J. De Smedt, and J. De Weerdt, "Educational sequence mining for dropout prediction in MOOCs: Model building, evaluation, and benchmarking," IEEE Trans. Learn. Technol., vol. 15, no. 6, pp. 720–735, 2022, doi: 10.1109/TLT.2022.3215598.
[27] A. Joshi, P. Saggar, R. Jain, M. Sharma, D. Gupta, and A. Khanna, "CatBoost—an ensemble machine learning model for prediction and classification of student academic performance," Adv. Data Sci. Adapt. Anal., vol. 13, no. 3&4, art. 2141002, 2021, doi: 10.1142/S2424922X21410023.
[28] S. A. Priyambada, T. Usagawa, and E. Mahendrawathi, "Two-layer ensemble prediction of students’ performance using learning behavior and domain knowledge," Comput. Educ. Artif. Intell., vol. 5, art. 100149, 2023, doi: 10.1016/j.caeai.2023.100149.
[29] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, "LightGBM: A highly efficient gradient boosting decision tree," in Proc. 31st Conf. Neural Inf. Process. Syst. (NeurIPS), Long Beach, CA, USA, 2017, pp. 3146–3154.
[30] J. Bergstra and Y. Bengio, "Random search for hyper-parameter optimization," J. Mach. Learn. Res., vol. 13, pp. 281–305, Feb. 2012.
[31] D. Chicco and G. Jurman, "The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation," BMC Genomics, vol. 21, art. 6, 2020, doi: 10.1186/s12864-019-6413-7.
[32] A. B. Arrieta, N. Diaz-Rodriguez, J. Del Ser, et al., "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI," Inf. Fusion, vol. 58, pp. 82–115, 2020, doi: 10.1016/j.inffus.2019.12.012.
[33] Z. Yin et al., "FairAIED: Navigating fairness, bias, and ethics in educational AI applications," arXiv:2407.18745, 2024.
[34] Z. Xu, H. Yuan, and Q. Liu, "Student performance prediction based on blended learning," IEEE Trans. Educ., vol. 64, no. 1, pp. 66–73, 2020, doi: 10.1109/TE.2020.3008751.
[35] G. Feng, M. Fan, and Y. Chen, "Analysis and prediction of students’ academic performance based on educational data mining," IEEE Access, vol. 10, pp. 19558–19571, 2022, doi: 10.1109/ACCESS.2022.3151652.
[36] Y. Chen and L. Zhai, "A comparative study on student performance prediction using machine learning," Educ. Inf. Technol., vol. 28, no. 9, pp. 12039–12057, 2023, doi: 10.1007/s10639-023-11672-1.