RCKT: Difference-aware Calibration-driven Representation Consistency Knowledge Tracing
-
摘要: 知识追踪旨在对学习者与教学系统的交互序列进行建模, 以刻画其知识状态的动态演化并预测后续作答表现, 进而支持个性化干预. 现有深度模型虽然预测性能优异, 但通常隐含一个假设, 即技能标签、题目表征和时间上下文等异构信号能够在统一嵌入空间中天然兼容. 然而, 当知识粒度不一致、标注存在噪声或数据分布发生偏移时, 这一假设常常不成立, 异构信号之间容易产生表示冲突. 这些表示冲突还会在注意力机制沿时间维度聚合信息的过程中被进一步放大, 使得模型预测置信度与真实不确定性之间出现失配, 从而损害知识状态估计的可靠性和可解释性. 针对上述问题, 本文提出一种表示一致性知识追踪框架RCKT. 该框架首先采用差异检测驱动的门控校准机制, 在表示传播之前修正技能表示与题目表示之间的冲突; 随后在传播阶段引入可靠性感知的自适应融合与组内一致性正则化, 以抑制不可靠信息的级联扩散. 在多个公开数据集上的实验结果表明, RCKT在保持具有竞争力的预测精度的同时, 有效改善了表示传播的稳定性, 并降低了校准误差.Abstract: Knowledge tracing aims to model the interaction sequences between learners and instructional systems, so as to characterize the dynamic evolution of learners' knowledge states and predict subsequent answering performance, thereby supporting personalized intervention. Although existing deep models achieve superior prediction performance, they typically imply an assumption that heterogeneous signals such as skill labels, item representations, and temporal context are naturally compatible in a unified embedding space. However, when knowledge granularity is inconsistent, annotations contain noise, or data distributions shift, this assumption often does not hold, and representation conflicts can easily arise among heterogeneous signals. These representation conflicts are further amplified when the attention mechanism aggregates information along the temporal dimension, causing a mismatch between prediction confidence and true uncertainty, thereby impairing the reliability and interpretability of knowledge state estimation. To address the above issues, this paper proposes a representation-consistency knowledge tracing framework, RCKT. The framework first adopts a difference-detection-driven gated calibration mechanism to correct the conflicts between skill representations and item representations before propagation, and then introduces reliability-aware adaptive fusion and intra-group consistency regularization during the propagation stage to suppress the cascading diffusion of unreliable information. Experimental results on multiple public datasets show that RCKT maintains competitive prediction accuracy while effectively improving the stability of representation propagation and reducing calibration error.
-
Key words:
- knowledge tracing /
- representation inconsistency /
- feature calibration /
- robustness /
- sequence modeling
-
图 1 表示不一致性的传播累积效应示意图
Fig. 1 Schematic of the cumulative propagation effect of representation inconsistency
表 1 关键符号和含义
Table 1 Key symbols and meanings
符号 含义 $ {\cal{S}} $ 技能(概念)集合 $ {\cal{P}} $ 题目集合 $ d $ 表示维度 $ d_g $ 组织特征维度 $ d_{hidden} $ MLP隐层维度 $ H $ 注意力头数 $ d_k $ 单头键/查询维度 $ \mathbb{I}(\cdot) $ 独热指示向量 $ [\cdot;\cdot] $ 向量拼接 $ \odot $ 逐元素乘积 $ \sigma(\cdot) $ Sigmoid函数 $ \text{Mean}(\boldsymbol{A}_t) $ 时间步$ t $的注意力分布的摘要特征 
下载: 导出CSV
表 2 数据集版本与统计信息
Table 2 Dataset versions and statistical information
数据集 交互数 学生数 知识点数 ASSISTments 2009 约 346860 约 4217 约124 ASSISTments 2015 约 708601 约 19840 约100 ASSISTments 2017 约 942816 约 1709 约102 Junyi 公开版本及筛选口径存在差异 39 EdNet-KT1 未单独公布 约 784309 293
下载: 导出CSV
表 3 本文实验中的$ {\boldsymbol{g}}_t $构造规则
Table 3 Construction rules of $ {\boldsymbol{g}}_t $ in the experiments of this paper
数据集/场景 标量来源 分组规则 解释方式 ASSISTments 2009 / ASSISTments 2017 ASSISTments $ 0 \rightarrow 1,\;\ 1 \rightarrow 2,\;\ \geq2 \rightarrow 3 $ 行为计数代理组 Junyi 无对应辅助字段 常数组1 退化为无group设定 EdNet-KT1 无同口径辅助计数字段 常数组1 不引入额外group先验 其他缺少可复现辅助字段的数据 无 常数组1 不引入额外group先验
下载: 导出CSV
表 6 ASSISTments 2017分层性能分析
Table 6 Stratified performance analysis on ASSISTments 2017
分桶类型 桶名称 样本数 AUC ACC AP 冲突分桶 low 6673 0.7692 0.7270 0.6065 mid 6733 0.7892 0.7514 0.6381 high 6594 0.8103 0.7407 0.7320 长尾分桶 tail 761 0.7090 0.6518 0.6877 mid 3218 0.7703 0.7082 0.6891 head 16021 0.7973 0.7502 0.6601
下载: 导出CSV
表 4 对比实验结果(%)
Table 4 Comparative experiment results (%)
模型 ASSISTments 2009 ASSISTments 2017 Junyi EdNet-KT1 AUC ACC AUC ACC AUC ACC AUC ACC BKT 66.05 ± 0.17 64.50 ± 0.23 63.65 ± 0.21 62.10 ± 0.17 66.68 ± 0.18 65.30 ± 0.21 65.13 ± 0.21 62.30 ± 0.19 DKT 76.53 ± 0.34 72.46 ± 0.38 71.25 ± 0.13 69.07 ± 0.08 73.90 ± 0.26 70.15 ± 3.15 69.83 ± 0.26 66.88 ± 0.38 DKVMN 75.44 ± 0.52 72.39 ± 0.33 68.90 ± 0.37 68.52 ± 0.50 76.92 ± 0.44 71.40 ± 0.32 73.17 ± 0.44 71.12 ± 0.33 SAKT 76.26 ± 0.33 72.06 ± 0.36 68.26 ± 0.36 66.94 ± 0.29 77.45 ± 0.43 74.98 ± 0.43 75.25 ± 0.36 72.87 ± 0.36 AKT 74.68 ± 0.36 72.72 ± 0.33 71.18 ± 0.33 70.80 ± 0.33 77.55 ± 0.15 74.04 ± 0.35 67.36 ± 0.33 62.53 ± 0.33 IEKT 74.58 ± 0.52 73.72 ± 0.31 75.60 ± 0.28 71.55 ± 0.33 76.49 ± 0.48 73.56 ± 0.38 69.64 ± 0.37 68.90 ± 0.33 LPKT 73.78 ± 0.23 72.11 ± 0.36 74.06 ± 0.52 73.72 ± 0.34 79.47 ± 0.20 76.03 ± 2.32 74.47 ± 0.29 71.01 ± 0.34 GKT 75.49 ± 0.27 72.52 ± 0.37 69.16 ± 0.40 68.85 ± 0.41 78.06 ± 0.38 74.47 ± 0.15 66.03 ± 0.38 66.57 ± 0.37 GIKT 78.20 ± 0.44 73.20 ± 0.34 75.80 ± 0.41 74.20 ± 0.39 81.61 ± 0.46 80.09 ± 0.18 72.73 ± 0.44 70.41 ± 0.34 DyGFormer 76.18 ± 0.45 74.33 ± 0.29 77.54 ± 0.31 76.20 ± 0.17 77.11 ± 0.20 75.42 ± 0.65 72.73 ± 0.31 62.59 ± 0.17 DyGKT 78.91 ± 0.35 76.50 ± 0.26 80.54 ± 0.47 78.40 ± 0.34 79.88 ± 0.52 77.37 ± 0.15 74.82 ± 0.47 72.94 ± 0.26 HiSACKT 74.17 ± 0.33 71.14 ± 0.29 74.80 ± 0.31 72.15 ± 0.27 79.56 ± 0.24 81.89 ± 0.22 65.22 ± 0.41 61.33 ± 0.38 UKT 78.85 ± 0.42 76.11 ± 0.35 73.98 ± 0.65 71.67 ± 0.32 70.13 ± 0.58 64.13 ± 0.61 74.45 ± 0.49 70.21 ± 0.44 RCKT 79.93 ± 0.15 76.90 ± 0.04 78.31 ± 0.11 77.37 ± 0.09 82.24 ± 0.54 78.87 ± 1.01 75.27 ± 0.65 73.14 ± 0.47
下载: 导出CSV
表 5 四个数据集上的校准指标对比(ECE/Brier/NLL)
Table 5 Calibration metrics comparison on four datasets (ECE/Brier/NLL)
模型 ASSISTments 2009 ASSISTments 2017 Junyi EdNet-KT1 ECE$ \downarrow $ Brier$ \downarrow $ NLL$ \downarrow $ ECE$ \downarrow $ Brier$ \downarrow $ NLL$ \downarrow $ ECE$ \downarrow $ Brier$ \downarrow $ NLL$ \downarrow $ ECE$ \downarrow $ Brier$ \downarrow $ NLL$ \downarrow $ AKT 0.0402 0.1955 0.5791 0.0208 0.2254 0.6422 0.0546 0.1899 0.5533 0.0203 0.2143 0.6172 DKT 0.0391 0.2046 0.5982 0.0228 0.2216 0.6319 0.0332 0.1841 0.5420 0.0206 0.2154 0.6200 IEKT 0.0485 0.1992 0.5875 0.0237 0.2116 0.6109 0.0289 0.1818 0.5368 0.0146 0.2190 0.6280 DyGKT 0.0259 0.1901 0.5680 0.0630 0.1989 0.5793 0.0217 0.1707 0.4845 0.0127 0.2007 0.5856 RCKT 0.0211 0.1722 0.5157 0.0241 0.1611 0.4823 0.0204 0.1645 0.4922 0.0147 0.1826 0.5425
下载: 导出CSV
表 7 RCKT消融实验结果(%)
Table 7 Ablation experiment results of RCKT (%)
数据集 RCKT w/o RC_1 (表示校准) w/o RC_2 (自适应融合) w/o RC_3 (一致性正则) ASSISTments 2009 79.3 $ \pm $ 0.15 76.6 $ \pm $ 0.21 77.4 $ \pm $ 0.18 74.8 $ \pm $ 0.24 ASSISTments 2017 78.3 $ \pm $ 0.11 73.6 $ \pm $ 0.19 75.9 $ \pm $ 0.16 77.6 $ \pm $ 0.14 Junyi 82.2 $ \pm $ 0.54 79.1 $ \pm $ 0.65 80.6 $ \pm $ 0.62 81.5 $ \pm $ 0.58 EdNet-KT1 76.3 $ \pm $ 0.65 72.3 $ \pm $ 0.75 73.2 $ \pm $ 0.72 75.4 $ \pm $ 0.68
下载: 导出CSV
表 8 ASSISTments 2009跨随机种子路由稳定性
Table 8 Cross-seed routing stability on ASSISTments 2009
模型 Attention JSD$ \downarrow $ Top-5 overlap$ \uparrow $ Entropy std$ \downarrow $ Gate std$ \downarrow $ w/o RC_3 0.0281 $ \pm $0.0075 0.2084 $ \pm $0.0359 0.2290 0.0680 RCKT 0.0122 $ \pm $0.0011 0.2711 $ \pm $0.0638 0.1272 0.0608
下载: 导出CSV
表 9 ASSISTments 2017上组输入依赖性控制实验
Table 9 Controlled experiment of group-input dependency on ASSISTments 2017
输入方式 AUC$ \uparrow $ ACC$ \uparrow $ ECE$ \downarrow $ original group 0.7831 0.7737 0.0241 w/o group 0.7806 0.7733 0.0276 zero-group 0.7818 0.7742 0.0314 random-group 0.7829 0.7741 0.0252
下载: 导出CSV
表 10 前置与后置校准对比(ASSISTments 2009 / ASSISTments 2017)
Table 10 Comparison of pre-calibration and post-hoc calibration on ASSISTments 2009/ASSISTments 2017
数据集 Full w/o w/o AUC Full ECE RC_1 AUC RC_1 ECE RC_1+TS AUC RC_1+TS ECE ASSISTments 2009 0.7993 0.0211 0.7660 0.0312 0.7812 0.0267 ASSISTments 2017 0.7831 0.0241 0.7364 0.0416 0.7541 0.0388
下载: 导出CSV
-
[1] 王宇, 朱梦霞, 杨尚辉, 陆雪松, 周傲英. 深度知识追踪模型综述和性能比较. 软件学报, 2023, 34(3): 1365−1395Wang Y, Zhu M X, Yang S H, Lu X S, Zhou A Y. Review and performance comparison of deep knowledge tracing models. Journal of Software, 2023, 34(3): 1365−1395 [2] Corbett A T, Anderson J R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 1994, 4(4): 253−278 [3] Lord F M. Applications of item response theory to practical testing problems. Routledge, 2012 [4] Piech C, Bassen J, Huang J, Ganguli S, Sahami M, Guibas L J, et al. Deep knowledge tracing. Advances in Neural Information Processing Systems, 2015, 28: 1−9 [5] Yeung C K, Yeung D Y. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In: Proceedings of the Fifth Annual ACM Conference on Learning at Scale (L@S). London, United Kingdom: ACM, 2018. 1−10 [6] Zhang J, Shi X, King I, Yeung D Y. Dynamic key-value memory networks for knowledge tracing. In: Proceedings of the 26th International Conference on World Wide Web (WWW). Perth, Australia: International World Wide Web Conferences Steering Committee, 2017. 765−774 [7] Pandey S, Karypis G. A self-attentive model for knowledge tracing. arXiv: 1907.06837, 2019 [8] Choi Y, Lee Y, Cho J, Sohn K, Chang J, Gweon G. Towards an appropriate query, key, and value computation for knowledge tracing. In: Proceedings of the Seventh ACM Conference on Learning@ Scale (L@S). Virtual Event, USA: ACM, 2020. 341−344 [9] Ghosh A, Heffernan N, Lan A S. Context-aware attentive knowledge tracing. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD). Virtual Event, CA, USA: ACM, 2020. 2330−2339 [10] Pandey S, Srivastava J. RKT: Relation-aware self-attention for knowledge tracing. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM). Virtual Event, Ireland: ACM, 2020. 1205−1214 [11] Nakagawa H, Iwasawa Y, Matsuo Y. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence (WI). Thessaloniki, Greece: ACM, 2019. 156−163 [12] Yang Y, Shen J, Qu Y, Liu Y, Wang K, Zhu Y, et al. GIKT: A graph-based interaction model for knowledge tracing. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD). Ghent, Belgium: Springer, 2020. 299−315 [13] Cheng K, Peng L, Wang P, Ma C, Liu Z, Sun M. DyGKT: Dynamic graph learning for knowledge tracing. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). Barcelona, Spain: ACM, 2024. 409−420 [14] Elman J L. Finding structure in time. Cognitive Science, 1990, 14(2): 179−211 [15] Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation, 1997, 9(8): 1735−1780 [16] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, et al. Attention is all you need. Advances in Neural Information Processing Systems, 2017, 30: 5998−6008 [17] Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. arXiv: 1609.02907, 2016 [18] Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[J]. arXiv preprint arXiv: 1710.10903, 2017 [19] Gilmer J, Schoenholz S S, Riley P F, Vinyals O, Dahl G E. Neural message passing for quantum chemistry. In: Proceedings of the 34th International Conference on Machine Learning (ICML). Sydney, Australia: PMLR, 2017. 1263−1272 [20] 张凯, 刘月, 覃正楚, 秦心怡. 概念表示增强的知识追踪模型. 计算机应用研究, 2022, 39(11): 3309−3314Zhang K, Liu Y, Qin Z C, Qin X Y. Concept representation enhanced knowledge tracing. Application Research of Computers, 2022, 39(11): 3309−3314 [21] Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples. arXiv: 1412.6572, 2014 [22] Miyato T, Maeda S, Koyama M, Ishii S. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(8): 1979−1993 [23] Zhang H, Cisse M, Dauphin Y N, Lopez-Paz D. mixup: Beyond empirical risk minimization. arXiv: 1710.09412, 2017 [24] Arjovsky M, Bottou L, Gulrajani I, Lopez-Paz D. Invariant risk minimization. arXiv: 1907.02893, 2019 [25] Sun B, Saenko K. Deep coral: Correlation alignment for deep domain adaptation. In: Proceedings of the European Conference on Computer Vision (ECCV). Amsterdam, The Netherlands: Springer, 2016. 443−450 [26] Gretton A, Borgwardt K M, Rasch M J, Schölkopf B, Smola A. A kernel two-sample test. The Journal of Machine Learning Research, 2012, 13(1): 723−773 [27] Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, et al. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 2016, 17(59): 1−35 [28] Sagawa S, Koh P W, Hashimoto T B, Liang P. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv: 1911.08731, 2019 [29] Guo C, Pleiss G, Sun Y, Weinberger K Q. On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML). Sydney, Australia: PMLR, 2017. 1321−1330 [30] Platt J. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 1999, 10(3): 61−74 [31] Hardt M, Price E, Srebro N. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 2016, 29: 3315−3323 [32] Dwork C, Hardt M, Pitassi T, Reingold O, Zemel R. Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS). Cambridge, MA, USA: ACM, 2012. 214−226 [33] Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML). Haifa, Israel: Omnipress, 2010. 807−814 [34] Ba J L, Kiros J R, Hinton G E. Layer normalization. arXiv: 1607.06450, 2016 [35] Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014, 15(1): 1929−1958 [36] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016. 770−778 [37] Glenn W B. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 1950, 78(1): 1−3 [38] Student. The probable error of a mean. Biometrika, 1908, 6(1): 1−25 [39] Wilcoxon F. Individual comparisons by ranking methods. Biometrics Bulletin, 1945, 1(6): 80−83 [40] Long T, Liu Y, Shen J, Zhang W, Yu Y. Tracing knowledge state with individual cognition and acquisition estimation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Virtual Event, Canada: ACM, 2021. 173−182 [41] Shen S, Liu Q, Chen E, Huang Z, Huang W, Yin Y, et al. Learning process-consistent knowledge tracing. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD). Virtual Event, Singapore: ACM, 2021. 1452−1460 [42] Yu L, Sun L, Du B, Liu C, Lv W, Xiong H. Towards better dynamic graph learning: New architecture and unified library. Advances in Neural Information Processing Systems, 2023, 36: 67686−67700 [43] Baig D, Nurbakova D, Mbaye B, Calabretto S. Combining hierarchical self-attention and skill clustering to enhance knowledge tracing. In: Proceedings of the 24th IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT). 2025 [44] Cheng W, Du H, Li C, Ni E, Tan L, Xu T, et al. Uncertainty-aware knowledge tracing. In: Proceedings of the 39th AAAI Conference on Artificial Intelligence (AAAI). 2025 -
下载:
计量
- 文章访问数: 1
- HTML全文浏览量: 2
- 被引次数: 0
