2.624

2020影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

一种同伴知识互增强下的序列推荐方法

胡开喜 李琳 吴小华 解庆 袁景凌

胡开喜, 李琳, 吴小华, 解庆, 袁景凌. 一种同伴知识互增强下的序列推荐方法. 自动化学报, 2022, 48(x): 1−15 doi: 10.16383/j.aas.c220347
引用本文: 胡开喜, 李琳, 吴小华, 解庆, 袁景凌. 一种同伴知识互增强下的序列推荐方法. 自动化学报, 2022, 48(x): 1−15 doi: 10.16383/j.aas.c220347
Hu Kai-Xi, Li Lin, Wu Xiao-Hua, Xie Qing, Yuan Jing-Ling. A sequential recommendation method enhanced by peer knowledge. Acta Automatica Sinica, 2022, 48(x): 1−15 doi: 10.16383/j.aas.c220347
Citation: Hu Kai-Xi, Li Lin, Wu Xiao-Hua, Xie Qing, Yuan Jing-Ling. A sequential recommendation method enhanced by peer knowledge. Acta Automatica Sinica, 2022, 48(x): 1−15 doi: 10.16383/j.aas.c220347

一种同伴知识互增强下的序列推荐方法

doi: 10.16383/j.aas.c220347
基金项目: 国家自然科学基金(62276196, 61602353), 湖北省重点研发计划项目(2021BAA030), 国家留学基金委(202106950041, 留金美[2020] 1509号)和安康市科学技术研究发展计划(AK2020-GY-08)资助
详细信息
    作者简介:

    胡开喜:武汉理工大学计算机与人工智能学院博士研究生. 2018年获得重庆大学控制工程硕士学位. 主要研究方向为序列预测. E-mail: issac_hkx@whut.edu.cn

    李琳:武汉理工大学计算机与人工智能学院教授. 2009年获得日本东京大学博士学位. 主要研究方向为信息检索, 推荐系统. 本文通信作者. E-mail: cathylilin@whut.edu.cn

    吴小华:武汉理工大学计算机与人工智能学院博士研究生. 2019年获得西北大学计算机科学与技术硕士学位. 主要研究方向为可解释机器学习. E-mail: xhwu@whut.edu.cn

    解庆:武汉理工大学计算机与人工智能学院副教授. 2013年获得澳大利亚昆士兰大学博士学位. 主要研究方向为流数据挖掘与模式分析. E-mail: felixxq@whut.edu.cn

    袁景凌:武汉理工大学计算机与人工智能学院教授. 2004年获得武汉理工大学博士学位. 主要研究方向为分布式并行计算. E-mail: yjl@whut.edu.cn

A Sequential Recommendation Method Enhanced by Peer Knowledge

Funds: Supported by National Natural Science Foundation of China (62276196, 61602353), Key Research and Development Program of Hubei Province (2021BAA030), China Scholarship Council (202106950041, LiuJinMei[2020]1509) and Ankang Municipal Science and Technology Bureau (AK2020-GY-08)
More Information
    Author Bio:

    HU Kai-Xi Ph. D. candidate at the School of Computer Science and Artificial Intelligence, Wuhan University of Technology. He received his master degree in control engineering from Chongqing University in 2018. His main research interest is sequential prediction

    LI Lin Professor at the School of Computer Science and Artificial Intelligence, Wuhan University of Technology. She received her Ph. D. degree from University of Tokyo, Japan, in 2009. Her research interest covers information retrieval and recommender systems. Corresponding author of this paper

    WU Xiao-Hua Ph. D. candidate at the School of Computer Science and Artificial Intelligence, Wuhan University of Technology. He received his master degree in computer science and technology from Northwest University in 2019. His main research interest is explainable machine learning

    XIE Qing Associate professor at the School of Computer Science and Artificial Intelligence, Wuhan University of Technology. He received his Ph. D. degree from University of Queensland, Australia, in 2013. His research interest covers streaming data mining and pattern analysis

    YUAN Jing-Ling Professor at the School of Computer Science and Artificial Intelligence, Wuhan University of Technology. She received her Ph. D. degree from Wuhan University of Technology, in 2004. Her main research interest is parallel distributed computing

  • 摘要: 序列推荐(Sequential recommendation, SR)旨在建模用户序列中的动态兴趣, 预测下一个行为. 现有基于知识蒸馏的多模型集成方法通常将教师模型预测的概率分布作为学生模型样本学习的软标签, 不利于关注低置信度序列样本中的动态兴趣. 提出了一种同伴知识互增强下的序列推荐方法(Sequential recommendation enhanced by peer knowledge, PeerRec), 使多个具有差异的同伴网络按照人类由易到难的认知过程进行两阶段的相互学习. 在第一阶段知识蒸馏的基础上, 第二阶段的刻意训练通过动态最小组策略协调多个同伴从低置信度样本中挖掘出可被加强训练的潜在样本. 然后, 受训的网络利用同伴对潜在样本预测的概率分布调节自身对该样本学习的权重, 从解空间中探索更优的兴趣表示. 三个公开数据集上的实验结果表明, 提出的PeerRec方法相比于最新的基线方法在基于Top-k的指标上不仅获得了更佳的推荐精度, 且具有良好的在线推荐效率.
    1)  1 一种基于最优传输理论衡量两个分布间距离的度量方式, 目前只在一维分布、高斯分布等少数几种分布上存在闭式解.
  • 图  1  用户动态兴趣在潜在空间中的表示与推断

    Fig.  1  The representation and inference of dynamic interests in latent representation spaces

    图  2  PeerRec模型的网络结构

    Fig.  2  The architecture of our proposed PeerRec

    图  3  基于刻意训练的互相学习

    Fig.  3  An illustration of deliberate practice based learning strategy

    图  4  PeerRec变体在HR@1指标上的对比

    Fig.  4  The comparison between the variants of PeerRec in terms of HR@1

    图  5  不同同伴网络数量的性能对比

    Fig.  5  The evaluation on different number of peers

    图  6  Batch大小设置为256时, 模型每秒迭代次数

    Fig.  6  The running speed of different models with batch size of 256

    图  7  超参数敏感性分析

    Fig.  7  Sensitivity analysis of hyper-parameters

    表  1  实验数据统计表

    Table  1  Statistics of dataset

    ML-1mLastFMToys
    用户数量(个)6 0401 09019 412
    行为类别数量(个)3 4163 64611 924
    最长序列的行为数量(个)2 275897548
    最短序列的行为数量(个)1633
    序列的平均行为数量(个)163.5046.216.63
    序列行为数量的方差192.53 77.698.50
    下载: 导出CSV

    表  2  与基线模型在标准推荐数据集上的精度对比

    Table  2  The comparison with baselines in terms of accuracy based metrics

    数据集模型HR1HR@5HR@10NDCG@5NDCG@10MRR
    ML-1mPOP0.04070.16030.27750.10080.13830.1233
    BERT4Rec[17]0.36950.68510.78230.53750.56900.5108
    S3-Rec[1]0.28970.65750.79110.45570.52660.4535
    HyperRec[9]0.31800.66310.77380.50140.53750.4731
    R-CE[11]0.39880.64780.74040.53270.56270.5179
    STOSA[14]0.32220.65460.78440.49670.53890.4716
    PeerRec(同伴1)0.42500.71970.81410.58430.61500.5600
    PeerRec(同伴2)0.42520.72250.81410.58600.61570.5610
    LastFMPOP0.02020.09080.17800.05440.08250.0771
    BERT4Rec[17]0.10910.32940.46140.22270.26480.2266
    S3-Rec[1]0.11560.28440.42290.20030.24520.2148
    HyperRec[9]0.11460.31470.46880.21500.26460.2241
    R-CE[11]0.06510.18350.28620.12430.15700.1397
    STOSA[14]0.07520.21650.34120.14580.18600.1556
    PeerRec(同伴1)0.12940.34950.47890.23390.27550.2341
    PeerRec(同伴2)0.12480.33580.48350.23180.27960.2378
    ToysPOP0.02600.10460.18480.06520.09090.0861
    BERT4Rec[17]0.13900.33790.45960.24090.28020.2444
    S3-Rec[1]0.09900.30230.43930.20210.24630.2081
    HyperRec[9]0.11470.28750.39090.20310.23650.2087
    R-CE[11]0.11300.31890.45290.21790.26110.2233
    STOSA[14]0.18380.35870.45500.27490.30590.2732
    PeerRec(同伴1)0.17940.37030.47850.27850.31340.2810
    PeerRec(同伴2)0.17820.37060.47780.27810.31270.2803
    下载: 导出CSV

    表  3  知识蒸馏与刻意训练对比

    Table  3  The comparison between knowledge distillation and deliberate practice

    数据集HR@1NDCG@5MRR
    知识蒸馏[39]ML-1m0.39520.56560.5386
    LastFM0.11190.23010.2314
    Toys0.16930.27610.2767
    刻意训练PeerRecML-1m0.42510.58520.5605
    LastFM0.12710.23290.2360
    Toys0.17880.27830.2807
    下载: 导出CSV

    表  4  PeerRec模型采用不同初始化的性能对比

    Table  4  The performance comparison between different initializations of our PeerRec

    数据集初始化方式HR@1NDCG@5MRR
    ML-1mTND0.42510.58520.5605
    Xavier0.42630.58520.5600
    Kaiming0.42780.59110.5652
    LastFMTND0.12710.23290.2360
    Xavier0.12940.23970.2424
    Kaiming0.12470.22570.2342
    ToysTND0.17880.27830.2807
    Xavier0.17750.27940.2811
    Kaiming0.18060.27760.2804
    下载: 导出CSV
  • [1] Zhou K, Wang H, Zhao W, Zhu Y, Wang S, Zhang F, et al. S3-Rec: Self-supervised learning for sequential recommendation with mutual information maximization. In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM). New York, USA: ACM, 2020. 1893−1902
    [2] 饶子昀, 张毅, 刘俊涛, 曹万华. 应用知识图谱的推荐方法与系统. 自动化学报, 2021, 47(9): 2061-2077 doi: 10.16383/j.aas.c200128

    Rao Zi-Yun, Zhang Yi, Liu Jun-Tao, Cao Wan-Hua. Recommendation methods and systems using knowledge graph. Acta Automatica Sinica, 2021, 47(9): 2061-2077 doi: 10.16383/j.aas.c200128
    [3] Li X, Liang J, Liu X, Zhang Y. Adversarial filtering modeling on long-term user behavior sequences for click-through rate prediction. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). New York, USA: ACM, 2022. 2671−2679
    [4] 汤文兵, 任正云, 韩芳. 基于注意力机制的协同卷积动态推荐网络. 自动化学报, 2021, 47(10): 2438-2448 doi: 10.16383/j.aas.c190820

    Tang Wen-Bing, Ren Zheng-Yun, Han Fang. Attention—based collaborative convolutional dynamic network for recommendation. Acta Automatica Sinica, 2021, 47(10): 2438-2448 doi: 10.16383/j.aas.c190820
    [5] 郭磊, 李秋菊, 刘方爱, 王新华. 基于自注意力网络的共享账户跨域序列推荐. 计算机研究与发展, 2021, 58(11): 2524-2537 doi: 10.7544/issn1000-1239.2021.20200564

    Guo Lei, Li Qiu-Ju, Liu Fang-Ai, Wang Xin-Hua. Shared-account cross-domain sequential recommendation with self-attention network. Journal of Computer Research and Development, 2021, 58(11): 2524-2537 doi: 10.7544/issn1000-1239.2021.20200564
    [6] Rao X, Chen L, Liu Y, Shang S, Yao B, Han P. Graph-flashback network for next location recommendation. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). New York, USA: ACM, 2022. 1463−1471
    [7] 孟祥武, 梁弼, 杜雨露, 张玉洁. 基于位置的移动推荐系统效用评价研究. 计算机学报, 2019, 42(12): 2695-2721 doi: 10.11897/SP.J.1016.2019.02695

    Meng Xiang-Wu, Liang Bi, Du Yu-Lu, Zhang Yu-Jie. A survey of evaluation for location-based mobile recommender systems. Chinese Journal of Computers, 2019, 42(12): 2695-2721 doi: 10.11897/SP.J.1016.2019.02695
    [8] Hu K, Li L, Liu J, Sun D. Duronet: A dual-robust enhanced spatial-temporal learning network for urban crime prediction. ACM Transactions on Internet Technology, 2021, 21(1): 1-24
    [9] Wang J, Ding K, Hong L, Liu H, Caverlee J. Next-item recommendation with sequential hypergraphs. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). New York, USA: ACM, 2020. 1101−1110
    [10] 陈聪, 张伟, 王骏. 带有时间预测辅助任务的会话式序列推荐. 计算机学报, 2021, 44(9): 1841-1853 doi: 10.11897/SP.J.1016.2021.01841

    Chen Cong, Zhang Wei, Wang Jun. Session-based sequential recommendation with auxiliary time prediction. Chinese Journal of Computers, 2021, 44(9): 1841-1853 doi: 10.11897/SP.J.1016.2021.01841
    [11] Wang W, Feng F, He X, Nie L, Chua T. Denoising implicit feedback for recommendation. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM). New York, USA: ACM, 2021. 373−381
    [12] Neupane K P, Zheng E, Yu Q. MetaEDL: Meta evidential learning for uncertainty-aware cold-start recommendations. In: Proceedings of the 2021 IEEE International Conference on Data Mining (ICDM). New York, USA: IEEE, 2021. 1258−1263
    [13] Fan Z, Liu Z, Wang S, Zheng L, Yu P. Modeling sequences as distributions with uncertainty for sequential recommendation. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (CIKM). New York, USA: ACM, 2021. 3019−3023
    [14] Fan Z, Liu Z, Wang A, Nazari Z, Zheng L, Peng H, et al. Sequential recommendation via stochastic self-attention. In: Proceedings of the 30th ACM Web Conference (WWW). New York, USA: ACM, 2022. 2036−2047
    [15] Jiang J, Yang D, Xiao Y, Shen C. Convolutional gaussian embeddings for personalized recommendation with uncertainty. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI). Macao, China: IJCAI, 2019. 2642−2648
    [16] Zhou X, Liu H, Pourpanah F, Zeng T, Wang X. A survey on epistemic (model) uncertainty in supervised learning: Recent advances and applications. Neurocomputing, 2022, 489: 449-465 doi: 10.1016/j.neucom.2021.10.119
    [17] Sun F, Liu J, Wu J, Pei C, Lin X, Ou W, et al. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). New York, USA: ACM, 2019. 1441−1450
    [18] Snoek J, Ovadia Y, Fertig E, Lakshminarayanan B, Nowozin S, Sculley S, et al. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS). Vancouver, Canada: Curran Associates, 2019. 13969−13980
    [19] Fort S, Hu H, Lakshminarayanan B. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: 1912.02757, 2019.
    [20] Lakshminarayanan B, Pritzel A, Blundell C. Simple and scalable predictive uncertainty estimation using deep ensembles. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS). Long Beach, USA: Curran Associates, 2017. 6402−6413
    [21] Renda A, Barsacchi M, Bechini A, Marcelloni F. Comparing ensemble strategies for deep learning: An application to facial expression recognition. Expert Systems With Applications, 2019, 136: 1-11 doi: 10.1016/j.eswa.2019.06.025
    [22] Deng D, Wu L, Shi B. Iterative distillation for better uncertainty estimates in multitask emotion recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). New York, USA: IEEE, 2021. 3557−3566
    [23] Reich S, Mueller D, Andrews N. Ensemble distillation for structured prediction: Calibrated, accurate, fast—choose three. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, USA: ACL, 2020. 5583−5595
    [24] Jiao X, Yin Y, Shang L, Jiang X, Chen X, Li L, et al. TinyBERT: Distilling BERT for natural language understanding. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, USA: ACL, 2020. 4163−4174
    [25] Zhu J, Liu J, Li W, Lai J, He X, Chen L, et al. Ensembled CTR prediction via knowledge distillation. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM). New York, USA: ACM, 2020. 2941−2958
    [26] Kang S, Hwang J, Kweon W, Yu H. De-rrd: A knowledge distillation framework for recommender system. In: Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM). New York, USA: ACM, 2020. 605−614
    [27] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. In: Proceedings of the 28th Conference on Neural Information Processing Systems (NeurIPS). Montreal, Canada: Curran Associates, 2014. 1−9
    [28] Shen Z, Liu Z, Xu D, Chen Z, Cheng K, Savvides M. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In: Proceedings of the International Conference on Learning Representations (ICLR). Virtual Event, Austria: OpenReview, 2020. 1−17
    [29] Furlanello T, Lipton Z, Tschannen M, Itti L, Anandkumar A. Born again neural networks. In: Proceedings of the International Conference on Machine Learning (ICML). Stockholm, Sweden: PMLR, 2018. 1607−1616
    [30] Romero A, Ballas N, Kahou S, Chassang A, Gatta C, Bengio Y. Fitnets: Hints for thin deep nets. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA, USA: OpenReview, 2015. 1−13
    [31] Lin T, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). New York, USA: IEEE, 2017. 2999−3007
    [32] Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning (ICML). New York, USA: ACM, 2009. 41−48
    [33] Ericsson K A. Deliberate practice and acquisition of expert performance: A general overview. Academic Emergency Medicine, 2008, 15(11): 988-994 doi: 10.1111/j.1553-2712.2008.00227.x
    [34] Song W, Shi C, Xiao Z, Duan Z, Xu Y, Zhang M, et al. Autoint: Automatic feature interaction learning via self-attentive neural networks. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM). New York, USA: ACM, 2019. 1161−1170
    [35] Qin Y, Wang P, Li C. The world is binary: Contrastive learning for denoising next basket recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). New York, USA: ACM, 2021. 859−868
    [36] 黄震华, 杨顺志, 林威, 倪娟, 孙圣力, 陈运文, 等. 知识蒸馏研究综述. 计算机学报, 2022, 45(3): 624-653 doi: 10.11897/SP.J.1016.2022.00624

    Hung Zhen-Hua, Yang Shun-Zhi, Lin Wei, Ni Juan, Sun Sheng-Li, Chen Yun-Wen, et al. Knowledge distillation: A survey. Chinese Journal of Computers, 2022, 45(3): 624-653 doi: 10.11897/SP.J.1016.2022.00624
    [37] 潘瑞东, 孔维健, 齐洁. 基于预训练模型与知识蒸馏的法律判决预测算法. 控制与决策, 2022, 37(1): 67-76 doi: 10.13195/j.kzyjc.2020.0985

    Pan Rui-dong, Kong Wei-jian, Qi Jie. Legal judgment prediction based on pre-training model and knowledge distillation. Control and Decision, 2022, 37(1): 67-76 doi: 10.13195/j.kzyjc.2020.0985
    [38] Zhao B, Cui Q, Song R, Qiu Y, Liang J. Decoupled knowledge distillation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, USA: IEEE, 2022. 11953−11962
    [39] Zhang Y, Xiang T, Hospedales T, Lu H. Deep mutual learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York, USA: IEEE, 2018. 4320−4328
    [40] Zhao H, Yang G, Wang D, Lu H. Deep mutual learning for visual object tracking. Pattern Recognition, 2021, 112: 107796-107808 doi: 10.1016/j.patcog.2020.107796
    [41] Chen D, Mei J, Wang C, Feng Y, Chen C. Online knowledge distillation with diverse peers. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). New York, USA: AAAI, 2020. 3430−3437
    [42] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, et al. Attention is all you need. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS). Long Beach, USA: Curran Associates, 2017. 5998−6008
    [43] Toneva M, Sordoni A, Combes R, Trischler A, Bengio Y, Gordon G. An empirical study of example forgetting during deep neural network learning. In: Proceedings of the 7th International Conference on Learning Representations (ICLR). New Orleans, Louisiana, USA: OpenReview, 2019. 1−19
    [44] Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS). Sardinia, Italy: JMLR, 2010. 249−256
    [45] He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). New York, USA: IEEE, 2015. 1026−1034
  • 加载中
计量
  • 文章访问数:  45
  • HTML全文浏览量:  16
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-04-28
  • 录用日期:  2022-09-13
  • 网络出版日期:  2022-10-26

目录

    /

    返回文章
    返回