A Cascade Reinforcement Learning Strategy for Efficiently Enhancing the Reasoning Ability of Multimodal Large Language Models
-
摘要: 强化学习在提升多模态大语言模型的推理能力上展现出巨大潜力, 逐渐成为模型训练过程中的关键步骤. 然而, 在线强化学习算法需要策略模型在训练过程中实时采样, 且收敛速度较慢, 因此训练成本昂贵; 离线强化学习算法虽然整体成本更低, 但是也牺牲了性能上限. 本文尝试将离线强化学习训练成本低的特性与在线强化学习性能上限高的优势相结合, 提出一种新的训练策略——级联强化学习. 这套训练策略包含离线强化学习和在线强化学习两个训练阶段, 其中离线强化学习阶段用于加速模型收敛并提升后续训练的稳定性; 在线强化学习阶段对模型进行更精细的训练, 进一步提升其性能上限. 本文通过一系列定量分析实验证明了相比单一的在线强化学习算法, 级联强化学习可以通过一半的训练成本达到更高的性能上限. 这种简单有效的训练策略将InternVL3.5-8B-Instruct和InternVL3.5-241B-A28B-Instruct在七个多模态推理评测基准上的平均准确率分别提升了6.7%和6.5%, 证明了这一策略的有效性和可扩展性.Abstract: Reinforcement learning has demonstrated substantial potential in enhancing the reasoning ability of multimodal large language models and has become a critical step during model training. However, online reinforcement learning algorithms require real-time sampling from the policy model during training and typically suffer from slow convergence speed, resulting in high training cost; Offline reinforcement learning algorithms offer lower overall cost but often sacrifices performance ceiling. In this paper, we propose a novel training strategy, termed cascade reinforcement learning, which integrates the low training cost of offline reinforcement learning with the superior performance potential of online reinforcement learning. This training strategy consists of two stages: An offline reinforcement learning stage that accelerates model convergence and improves subsequent training stability, followed by an online reinforcement learning stage that performs more fine-grained training to further enhance the model＇s performance ceiling. Through extensive quantitative analysis, we demonstrate that our cascade reinforcement learning achieves higher performance ceiling than standalone online reinforcement learning algorithms while requiring only half of the training cost. Using this simple yet effective training strategy, we improve the average accuracy of InternVL3.5-8B-Instruct and InternVL3.5-241B-A28B-Instruct by 6.7% and 6.5%, respectively, across seven multimodal reasoning evaluation benchmarks, validating the effectiveness and scalability of the proposed strategy.
-
图 2 模型训练过程中关于起始模型的正负响应的生成概率
Fig. 2 Generation probabilities of positive and negative responses relative to the initial model during model training
图 3 级联强化学习各阶段对模型推理性能的影响
Fig. 3 Impact of different CascadeRL stages on model reasoning performance
图 5 不同算法训练过程中模型关于起始模型的正负响应的生成概率
Fig. 5 Generation probabilities of positive and negative responses relative to the initial model during training with different algorithms
表 1 不同算法的训练结果对比
Table 1 Comparison of training results for different algorithms
模型 训练响应数量 显卡小时数 得分 基线模型 InternVL3.5-4B-Instruct — — 57.5 有监督微调 SFT (Half) ~ 59.2 K 5.3 64.4 SFT (Half × 2) ~ 118.4 K 8.0 66.6 SFT (Full) ~ 124.3 K 8.0 67.0 离线强化学习算法 DPO (Half) ~ 64.5 K 4.5 67.3 DPO (Half × 2) ~ 129.1 K 8.5 70.2 DPO (Full) ~ 128.2 K 8.5 69.5 MPO (Half) ~ 64.5 K 4.6 68.0 MPO (Half × 2) ~ 129.1 K 8.5 70.8 MPO (Full) ~ 128.2 K 8.5 70.8 在线强化学习算法 GRPO (Half) ~ 62.9 K 65.2 63.9 GRPO (Half × 2) ~ 125.8 K 141.4 69.8 GRPO (Full) ~ 125.8 K 144.4 71.5 GSPO (Half) ~ 62.9 K 65.4 63.4 GSPO (Half × 2) ~ 125.8 K 159.3 67.3 GSPO (Full) ~ 125.8 K 168.6 72.0 
下载: 导出CSV
表 2 相同数据成本下不同算法的训练结果
Table 2 Training results of different algorithms under the same data cost
模型 训练响应数量 显卡小时数 得分 SFT (Half × 2) ~ 118.4 K 8.0 66.6 SFT (Full) ~ 124.3 K 8.0 67.0 MPO (Half × 2) ~ 129.1 K 8.5 70.8 MPO (Full) ~ 128.2 K 8.5 70.8 GSPO (Half × 2) ~ 125.8 K 159.3 67.3 GSPO (Full) ~ 125.8 K 168.6 72.0 MPO → GSPO ~ 127.4 K 68.3 72.2 注:“→“表示先基于MPO (Half) 训练一轮, 再基于GSPO (Half)续训 一轮.
下载: 导出CSV
表 3 基于不同级联强化学习实例的训练结果
Table 3 Training results based on different CascadeRL instances
模型 训练响应数量 显卡小时数 得分 SFT → GRPO ~ 122.1 K 66.0 70.3 SFT → GSPO ~ 122.1 K 75.4 71.3 GRPO → DPO ~ 127.4 K 73.2 70.1 GRPO → MPO ~ 127.4 K 73.2 70.4 GSPO → DPO ~ 127.4 K 73.4 68.1 GSPO → MPO ~ 127.4 K 73.4 70.0 DPO → GRPO ~ 127.4 K 63.2 69.9 DPO → GSPO ~ 127.4 K 67.8 71.6 MPO → GRPO ~ 127.4 K 68.1 71.5 MPO → GSPO ~ 127.4 K 68.3 72.2 MPO (Full) → GRPO ~ 191.1 K 70.4 72.3 MPO (Full) → GSPO ~ 191.1 K 72.1 72.1
下载: 导出CSV
表 4 不同算法训练后的模型在域外评测集的性能
Table 4 Performance of models trained with different algorithms on out-of-domain evaluation sets
模型 M3CoT MMMU MathVerse InternVL3.5-4B-Instruct 57.5 62.3 42.4 SFT (Full) 67.0 59.6 39.8 MPO (Full) 70.8 60.7 44.2 GSPO (Full) 72.0 63.3 43.5 GSPO → MPO 70.0 60.4 43.5 MPO → GSPO 72.2 62.8 44.8
下载: 导出CSV
表 5 不同超参数设置对训练结果的影响
Table 5 Impact of different hyperparameter settings on training results
实验设置 MPO GSPO InternVL3.5-4B-Instruct 57.5 57.5 每个问题采样的响应数量 每个问题采样4条响应 63.4 60.1 每个问题采样8条响应 67.6 63.7 每个问题采样16条响应 70.8 72.0 每个问题采样32条响应 71.2 71.7 KL散度约束强度 coef = 0 — 71.7 coef = 0.001 — 71.7 coef = 0.100 — 71.8 coef = 1.000 — 71.1
下载: 导出CSV
表 6 不同比例离线和在线强化学习的训练结果
Table 6 Training results of different offline and online reinforcement learning ratios
实验设置 显卡小时数 GSPO InternVL3.5-4B-Instruct — 57.5 2.0 : 0 ~ 8.5 K 70.8 1.5 : 0.5 ~ 48.1 K 71.4 1.0 : 1.0 ~ 68.3 K 72.2 0.5 : 1.5 ~ 130.2 K 72.3 0 : 2.0 ~ 168.6 K 72.0
下载: 导出CSV
表 7 不同模型的多模态推理性能对比
Table 7 Comparison of multimodal reasoning performance across different models
模型 MMMU
(val)MathVista
(mini)MathVision MathVerse
(vision-only)DynaMath WeMath LogicVista 平均分 InternVL3-1B[60] 43.4 45.8 18.8 18.7 5.8 13.4 29.8 25.1 InternVL3.5-1B-CascadeRL 44.2 59.3 27.3 37.8 17.2 21.5 29.3 33.8 Ovis-2B[61] 45.6 64.1 17.7 29.4 10.0 9.9 34.7 30.2 Qwen2.5-VL-3B[62] 51.2 61.2 21.9 31.2 13.2 22.9 40.3 34.6 InternVL3-2B[60] 48.6 57.0 21.7 25.3 14.6 22.4 36.9 32.4 InternVL3.5-2B-CascadeRL 59.0 71.8 42.8 53.4 31.5 48.5 47.7 50.7 Ovis-4B[61] 49.0 69.6 21.5 38.5 18.0 16.9 35.3 35.5 MiniCPM-V-4-4B[63] 51.2 66.9 20.7 18.3 14.2 32.7 30.6 33.5 InternVL2.5-4B[64] 51.8 64.1 18.4 27.7 15.2 21.2 34.2 33.2 InternVL3.5-4B-CascadeRL 66.6 77.1 54.4 61.7 35.7 50.1 56.4 57.4 MiniCPM-o2.6[63] 50.9 73.3 21.7 35.0 10.4 25.2 36.0 36.1 Ovis-8B[61] 57.4 71.8 25.9 42.3 20.4 27.2 39.4 40.6 Qwen2.5-VL-8B[62] 55.0 67.8 25.4 41.1 21.0 35.2 44.1 41.4 MiMo-VL-RL-8B[44] 66.7 81.5 60.4 71.5 45.9 66.3 61.4 64.8 Keye-VL-8B[28] 71.4 80.7 50.8 54.8 37.3 60.7 54.8 58.6 GLM-4.1V-9B[27] 68.0 80.7 54.4 68.4 42.5 63.8 60.4 62.6 InternVL3-8B[60] 62.7 71.6 29.3 39.8 25.5 37.1 44.1 44.3 InternVL3.5-8B-CascadeRL 73.4 78.4 56.8 61.5 37.7 57.0 57.3 60.3 Gemma-3-12B[65] 55.2 56.1 30.3 21.1 20.8 33.6 41.2 36.9 Ovis2-16B[61] 60.7 73.7 30.1 45.8 26.3 45.0 47.4 47.0 InternVL3-14B[60] 67.1 75.1 37.2 44.4 31.3 43.0 51.2 49.9 InternVL3.5-14B-CascadeRL 73.3 80.5 59.9 62.8 38.7 58.7 60.2 62.0 Kimi-VL-A3B-2506[66] 64.0 80.1 54.4 54.6 28.1 42.0 51.4 53.5 InternVL3.5-30B-A3B-CascadeRL 75.6 80.9 55.7 60.4 36.5 48.4 55.7 59.0 Gemma-3-27B[65] 64.9 59.8 39.8 34.0 28.5 37.9 47.3 44.6 Ovis2-34B[61] 66.7 76.1 31.9 50.1 27.5 51.9 49.9 50.6 Qwen2.5-VL-32B[62] 70.2 74.8 38.1 57.6 35.1 46.5 52.6 53.6 Skywork-R1V3-38B[45] 76.0 77.1 52.6 59.6 35.1 56.5 59.7 59.5 InternVL3-38B[60] 70.1 75.1 34.2 48.2 35.3 48.6 58.4 52.8 InternVL3.5-38B-CascadeRL 76.9 81.9 63.7 67.6 41.7 64.8 65.3 66.0 GPT-5-nano-20250807[50] 72.6 73.1 59.7 66.6 47.9 59.4 57.5 62.4 GPT-5-20250807[50] 81.8 81.9 72.0 81.2 60.9 71.1 70.0 74.1 Claude-3.7-Sonnet[51] 75.0 66.8 41.9 46.7 39.7 49.3 58.2 53.9 Gemini-2.0-Pro[52] 69.9 71.3 48.1 67.3 43.3 56.5 53.2 58.5 Gemini-2.5-Pro[52] 74.7 80.9 69.1 76.9 56.3 78.0 73.8 72.8 Doubao-1.5-Pro[53] 73.8 78.6 51.5 64.7 44.9 65.7 64.2 63.3 GLM-4.5V[27] 75.4 84.6 65.6 72.1 53.9 68.8 62.4 69.0 QvQ-72B-Preview[46] 70.3 70.3 34.9 48.2 30.7 39.0 58.2 50.2 Qwen2.5-VL-72B[62] 68.2 74.2 39.3 47.3 35.9 49.1 55.7 52.8 Step3-321B-A38B[67] 74.2 79.2 64.8 62.7 50.1 59.8 60.2 64.4 InternVL3-78B[60] 72.2 79.0 43.1 51.0 35.1 46.1 55.9 54.6 InternVL3.5-241B-A28B-CascadeRL 77.7 82.7 63.9 68.5 46.5 62.3 66.7 66.9
下载: 导出CSV
表 8 模型在不同训练阶段结束后的多模态推理性能对比
Table 8 Comparison of multimodal reasoning performance after different training stages
模型 MMMU
(val)MathVista
(mini)MathVision MathVerse
(vision-only)DynaMath WeMath LogicVista 平均分 InternVL3.5-1B-Instruct 37.2 48.6 15.8 27.0 8.4 13.9 29.1 25.7 + MPO 40.3 50.5 22.0 32.1 9.0 16.8 32.7 29.1 + CascadeRL 44.2 59.3 27.3 37.8 17.2 21.5 29.3 33.8 InternVL3.5-2B-Instruct 53.0 60.8 27.0 39.6 19.8 28.1 41.2 38.5 + MPO 54.3 62.6 34.2 46.4 21.0 28.1 40.9 41.1 + CascadeRL 59.0 71.8 42.8 53.4 31.5 48.5 47.7 50.7 InternVL3.5-4B-Instruct 64.3 71.4 40.5 50.0 30.7 35.6 53.5 49.4 + MPO 65.4 71.7 48.0 54.9 30.7 39.8 55.9 52.3 + CascadeRL 66.6 77.1 54.4 61.7 35.7 50.1 56.4 57.4 InternVL3.5-8B-Instruct 68.1 74.2 46.4 55.8 30.7 46.0 53.9 53.6 + MPO 71.2 75.9 52.6 54.8 33.1 47.7 58.6 56.3 + CascadeRL 73.4 78.4 56.8 61.5 37.7 57.0 57.3 60.3 InternVL3.5-14B-Instruct 71.8 73.4 48.7 55.5 31.9 45.7 57.5 54.9 + MPO 73.3 74.0 53.0 57.5 32.3 45.2 60.9 56.6 + CascadeRL 73.3 80.5 59.9 62.8 38.7 58.7 60.2 62.0 InternVL3.5-30B-A3B-Instruct 72.3 73.3 45.1 50.4 31.9 39.7 56.4 52.7 + MPO 71.7 75.3 50.7 58.5 32.9 43.7 59.7 56.1 + CascadeRL 75.6 80.9 55.7 60.4 36.5 48.4 55.7 59.0 InternVL3.5-38B-Instruct 73.9 75.9 58.2 59.0 29.7 47.5 60.0 57.7 + MPO 76.9 80.5 56.3 59.4 36.9 55.6 64.2 61.4 + CascadeRL 76.9 81.9 63.7 67.6 41.7 64.8 65.3 66.0 InternVL3.5-241B-A28B-Instruct 76.2 80.1 55.6 61.7 36.5 49.7 63.3 60.4 + MPO 76.0 82.2 55.3 64.1 38.3 51.3 69.4 62.4 + CascadeRL 77.7 82.7 63.9 68.5 46.5 62.3 66.7 66.9
下载: 导出CSV
表 9 离线强化学习(MPO)、在线强化学习(GSPO)以及CascadeRL的训练效率及性能对比
Table 9 Comparison of training efficiency and performance among offline RL (MPO), online RL (GSPO), and CascadeRL
模型 显卡小时数 MMMU
(val)MathVista
(mini)MathVision MathVerse
(vision-only)DynaMath WeMath LogicVista 均分 InternVL3.5-8B-Instruct — 68.1 74.2 46.4 55.8 30.7 46.0 53.9 53.6 + MPO ~ 0.3 K 71.2 75.9 52.6 54.8 33.1 47.7 58.6 56.3 + GSPO (1 episode) ~ 5.5 K 73.8 77.9 51.6 58.8 35.1 48.9 54.8 57.3 + GSPO (2 episodes) ~ 11.0 K 72.0 78.1 51.6 58.5 35.7 54.1 57.0 58.2 + CascadeRL ~ 5.8 K 73.4 78.4 56.8 61.5 37.7 57.0 57.3 60.3
下载: 导出CSV
-
[1] Rafailov R, Sharma A, Mitchell E, Manning S, Ermon C D, Finn C. Direct preference optimization: Your language model is secretly a reward model. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2023. Article No. 2338 [2] Wang W Y, Chen Z, Wang W H, Cao Y, Liu Y Z, Gao Z W, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv: 2411.10442, 2024. [3] Shao Z H, Wang P Y, Zhu Q H, Xu R X, Song J X, Bi X, et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv: 2402.03300, 2024. [4] Zheng C J, Liu S X, Li M Z, Chen X H, Yu B W, Gao C, et al. Group sequence policy optimization. arXiv preprint arXiv: 2507.18071, 2025. [5] Chen Q G, Qin L B, Zhang J, Chen Z, Xu X, Che W X. M3CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok, Thailand: ACL, 2024. 8199−8221 [6] Wang W Y, Gao Z W, Gu L X, Pu H J, Cui L, Wei X G, et al. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv: 2508.18265, 2025. [7] Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Lacroix T, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv: 2302.13971, 2023. [8] Cai Z, Cao M S, Chen H J, Chen K, Chen K Y, Chen X, et al. InternLM2 technical report. arXiv preprint arXiv: 2403.17297, 2024. [9] Yang A, Li A F, Yang B S, Zhang B C, Hui B Y, Zheng B, et al. Qwen3 technical report. arXiv preprint arXiv: 2505.09388, 2025. [10] OpenAI. Introducing gpt-oss [Online], available: https://openai.com/index/introducing-gpt-oss/, August 5, 2025 [11] Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C L, Mishkin P, et al. Training language models to follow instructions with human feedback. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2022. Article No. 2011 [12] 徐正斐, 辛欣. 基于大语言模型的中文实体链接实证研究. 自动化学报, 2025, 51(2): 327−342Xu Zheng-Fei, Xin Xin. An empirical study of Chinese entity linking based on large language model. Acta Automatica Sinica, 2025, 51(2): 327−342 [13] 秦龙, 武万森, 刘丹, 胡越, 尹全军, 阳东升, 等. 基于大语言模型的复杂任务自主规划处理框架. 自动化学报, 2024, 50(4): 862−872 doi: 10.16383/j.aas.c240088Qin Long, Wu Wan-Sen, Liu Dan, Hu Yue, Yin Quan-Jun, Yang Dong-Sheng, et al. Autonomous planning and processing framework for complex tasks based on large language models. Acta Automatica Sinica, 2024, 50(4): 862−872 doi: 10.16383/j.aas.c240088 [14] 陈楚岩, 刘烨谞, 贾维宸, 何雨桐, 袁坤, 王立威. 一种基于单比特通信压缩的大模型训练方法研究. 自动化学报, DOI: 10.16383/j.aas.c250087Chen Chu-Yan, Liu Ye-Xu, Jia Wei-Chen, He Yu-Tong, Yuan Kun, Wang Li-Wei. An effective algorithm for LLMs training with one-bit communication compression. Acta Automatica Sinica, DOI: 10.16383/j.aas.c250087 [15] Yan H, Gao Y, Fei C Y, Yang X P, Qiu X P. Data and model architecture in base model training. In: Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum). Harbin, China: Chinese Information Processing Society of China, 2023. 1−15 [16] Li Z J, Zhang J W, Wang Y, Du M F, Liu Q W, Wang D Y, et al. From multi-modal pre-training to multi-modal large language models: An overview of architectures, training. In: Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum). Taiyuan, China: Chinese Information Processing Society of China, 2024. 1−33 [17] 郑逸宁, 余镇, 李不凡, 杨捷, 殷林琪, 印张悦, 等. 大语言模型的工具使用综述. 自动化学报, 2025, 51(11): 2371−2386 doi: 10.16383/j.aas.c240793Zheng Yi-Ning, Yu Zhen, Li Bu-Fan, Yang Jie, Yin Lin-Qi, Yin Zhang-Yue, et al. Survey of tool use in large language models. Acta Automatica Sinica, 2025, 51(11): 2371−2386 doi: 10.16383/j.aas.c240793 [18] Schuhmann C, Beaumont R, Vencu R, Gordon C, Wightman R, Cherti M, et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2022. Article No. 1833 [19] Li Q Y, Chen Z, Wang W Y, Wang W H, Ye S L, Jin Z J, et al. OmniCorpus: An unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv: 2406.08418, 2024. [20] Laurençon H, Saulnier L, Tronchon L, Bekman S, Singh A, Lozhkov A, et al. OBELICS: An open web-scale filtered dataset of interleaved image-text documents. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, USA: Curran Associates Inc., 2023. Article No. 3138 [21] Liu Y Z, Cao Y, Gao Z W, Wang W Y, Chen Z, Wang W H, et al. MMInstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity. arXiv preprint arXiv: 2407.15838, 2024. [22] Wang W Y, Shi M, Li Q Y, Wang W H, Huang Z H, Xing L J, et al. The all-seeing project: Towards panoptic visual recognition and understanding of the open world. In: Proceedings of the 12th International Conference on Learning Representations. Vienna, Austria: OpenReview.net, 2024. [23] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017. [24] Zheng R, Dou S H, Gao S Y, Hua Y, Shen W, Wang B H, et al. Secrets of RLHF in large language models part I: PPO. arXiv preprint arXiv: 2307.04964, 2023. [25] Bradley R A, Terry M E. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 1952, 39(3−4): 324−345 doi: 10.2307/2527550 [26] Jung S, Han G, Nam D W, On K W. Binary classifier optimization for large language model alignment. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. Vienna, Austria: ACL, 2025. 1858−1872 [27] Hong W Y, Yu W M, Gu X T, Wang G, Gan G B, Tang H M, et al. GLM-4.5V and GLM-4.1V-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv: 2507.01006, 2025. [28] Keye T K, Yang B, Wen B, Liu C Y, Chu C L, Song C R, et al. Kwai keye-VL technical report. arXiv preprint arXiv: 2507.01949, 2025. [29] Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning. Montreal, Canada: ACM, 2009. 41−48 [30] Wang X, Chen Y D, Zhu W W. A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 4555−4576 [31] Xi Z H, Chen W X, Hong B Y, Jin S J, Zheng R, He W, et al. Training large language models for reasoning through reverse curriculum reinforcement learning. In: Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria: JMLR.org, 2024. Article No. 2217 [32] Wu J X, Zhang D M, Zhong S L, Qiao H. Trajectory-based split hindsight reverse curriculum learning. In: Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Prague, Czech Republic: IEEE, 2021. 3971−3978 [33] Qu Y X, Zhang T J, Garg N M, Kumar A. RECURSIVE INTROSPECTION: Teaching language model agents how to self-improve. In: Proceedings of the 38th International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2024. Article No. 1754 [34] Wang Z T, Cui G F, Li Y J, Wan K, Zhao W T. DUMP: Automated distribution-level curriculum learning for RL-based LLM post-training. arXiv preprint arXiv: 2504.09710, 2025. [35] Wen L, Cai Y K, Xiao F R, He X, An Q, Duan Z Y, et al. Light-R1: Curriculum SFT, DPO and RL for long COT from scratch and beyond. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track). Vienna, Austria: ACL, 2025. 318−327 [36] Bi X, Chen D L, Chen G T, Chen S H, Dai D M, Deng C Q, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv: 2401.02954, 2024. [37] Sun C N, Jung G Y, Tran T X, Pompili D. Cascade reinforcement learning with state space factorization for O-RAN-based traffic steering. In: Proceedings of the 21st Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). Phoenix, USA: IEEE, 2024. 1−9 [38] Castillo G A, Weng B W, Zhang W, Hereid A. Reinforcement learning-based cascade motion policy design for robust 3D bipedal locomotion. IEEE Access, 2022, 10: 20135−20148 doi: 10.1109/ACCESS.2022.3151771 [39] Xu J, Qiao J B, Sun Q L, Shen K Y. A deep reinforcement learning framework for cascade reservoir operations under runoff uncertainty. Water, 2025, 17(15): Article No. 2324 doi: 10.3390/w17152324 [40] Pal A, Karkhanis D, Dooley S, Roberts M, Naidu S, White C. Smaug: Fixing failure modes of preference optimisation with DPO-positive. arXiv preprint arXiv: 2402.13228, 2024. [41] Bai J Z, Bai S, Yang S S, Wang S J, Tan S N, Wang P, et al. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv: 2308.12966, 2023. [42] Yue X, Ni Y S, Zhang K, Zheng T Y, Liu R Q, Zhang G, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv preprint arXiv: 2311.16502, 2023. [43] Zhang R R, Jiang D Z, Zhang Y C, Lin H K, Guo Z Y, Qiu P S, et al. MathVerse: Does your multi-modal LLM truly see the diagrams in visual math problems? arXiv preprint arXiv: 2403.14624, 2024. [44] Yue Z H, Lin Z R, Song Y F, Wang W K, Ren S H, Gu S H, et al. Mimo-VL technical report. arXiv preprint arXiv: 2506.03569, 2025. [45] Shen W, Pei J B, Peng Y, Song X C, Liu Y, Peng J, et al. Skywork-R1V3 technical report. arXiv preprint arXiv: 2507.06167, 2025. [46] Qwen Team. QVQ: To see the world with wisdom [Online], available: https://qwen.ai/blog?id=qvq-72b-preview, December 25, 2024 [47] Qwen Team. Qwen2.5: A party of foundation models! [Online], available: https://qwenlm.github.io/blog/qwen2.5/, September 19, 2024 [48] Mesnard T, Hardin C, Dadashi R, Bhupatiraju S, Pathak S, Sifre L, et al. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv: 2403.08295, 2024. [49] StepFun Research Team. Step-1v: A hundred billion parameter multimodal large model [Online], available: https://platform.stepfun.com, March 23, 2024 [50] OpenAI. Gpt-5 system card [Online], available: https://cdn.openai.com/pdf/8124a3ceab78-4f06-96eb-49ea29ffb52f/gpt5-system-card-aug7.pdf, August 7, 2025 [51] Anthropic. The Claude 3 model family: Opus, sonnet, haiku [Online], available: https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, March 4, 2024 [52] Pichai S, Hassabis D, Kavukcuoglu K. Introducing Gemini 2.0: Our new AI model for the agentic era [Online], https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/, December 11, 2024 [53] Guo D, Wu F M, Zhu F D, Leng F X, Shi G, Chen H B, et al. Seed1.5-VL technical report. arXiv preprint arXiv: 2505.07062, 2025. [54] Lu P, Bansal H, Xia T, Liu J C, Li C Y, Hajishirzi H, et al. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv: 2310.02255, 2023. [55] Wang K, Pan J T, Shi W K, Lu Z M, Zhan M J, Li H S. Measuring multimodal mathematical reasoning with MATH-vision dataset. arXiv preprint arXiv: 2402.14804, 2024. [56] Zou C K, Guo X G, Yang R, Zhang J Y, Hu B, Zhang H. DynaMath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv: 2411.00836, 2024. [57] Qiao R Q, Tan Q N, Dong G T, Wu M H, Sun C, Song X S, et al. We-Math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv: 2407.01284, 2024. [58] Xiao Y J, Sun E, Liu T Y, Wang W. LogicVista: Multimodal LLM logical reasoning benchmark in visual contexts. arXiv preprint arXiv: 2407.04973, 2024. [59] OpenCompass Contributors. OpenCompass: A universal evaluation platform for foundation models [Online], available: https://github.com/open-compass/opencompass, July 4, 2023 [60] Zhu J G, Wang W Y, Chen Z, Liu Z Y, Ye S L, Gu L X, et al. InternVL3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv: 2504.10479, 2025. [61] Lu S Y, Li Y, Chen Q G, Xu Z, Luo W H, Zhang K F, et al. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv: 2405.20797, 2024. [62] Bai S, Chen K Q, Liu X J, Wang J L, Ge W B, Song S B, et al. Qwen2.5-VL technical report. arXiv preprint arXiv: 2502.13923, 2025. [63] Yao Y, Yu T Y, Zhang A, Wang C Y, Cui J B, Zhu H J, et al. MiniCPM-V: A GPT-4V level MLLM on your phone. arXiv preprint arXiv: 2408.01800, 2024. [64] Chen Z, Wang W Y, Cao Y, Liu Y Z, Gao Z W, Cui E F, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv: 2412.05271, 2024. [65] Team G, Kamath A, Ferret J, Pathak S, Vieillard N, Merhej R, et al. Gemma 3 technical report. arXiv preprint arXiv: 2503.19786, 2025. [66] Team K, Du A G, Yin B H, Xing B W, Qu B W, Wang B W, et al. Kimi-Vl technical report. arXiv preprint arXiv: 2504.07491, 2025. [67] StepFun Team. Step3: Cost-effective multimodal intelligence [Online], available: https://stepfun.ai/research/en/step3, July 25, 2025 -
下载:
图(5) / 表(9)
计量
- 文章访问数: 623
- HTML全文浏览量: 532
- PDF下载量: 48
- 被引次数: 0
