A Review of Reinforcement Learning Applications in Tokamak Controlled Nuclear Fusion
-
摘要: 随着国际热核聚变实验堆等大型装置的推进, 托卡马克装置正迈向长脉冲和高性能运行阶段, 对等离子体控制提出更高要求. 强化学习作为一种数据驱动的序贯决策方法, 近年来在复杂系统控制中展现出重要潜力, 并逐渐应用于托卡马克等离子体控制研究. 围绕托卡马克放电全过程的控制需求, 对强化学习在可控核聚变中的应用进展进行系统综述. 首先介绍托卡马克装置原理、放电阶段及典型控制问题, 并概述强化学习的基本理论与算法体系. 随后按照托卡马克放电过程的不同阶段, 重点对强化学习在电流爬升、平顶阶段及电流下降阶段的代表性控制问题进行归纳与比较. 并进一步总结近年来基于世界模型的强化学习方法在托卡马克控制中的应用研究, 对比分析与传统基于等离子体动力学建模相比, 在提升样本效率与策略泛化能力方面的潜力. 最后从系统复杂性、部分可观测性、安全约束、仿真–现实差距以及实时部署等方面讨论强化学习在托卡马克控制中的挑战与未来发展方向. 本文旨在为强化学习方法在托卡马克中的研究与应用提供参考.Abstract: With the advancement of large-scale devices such as the international thermonuclear experimental reactor, tokamak devices are entering a stage of long-pulse and high-performance operation, which imposes increasingly stringent requirements on plasma control. Reinforcement learning, as a data-driven sequential decision-making paradigm, has recently demonstrated significant potential in the control of complex systems and has been increasingly applied in tokamak plasma control research. This paper presents a systematic review of advances in the applications of reinforcement learning for controlled nuclear fusion from the perspective of control requirements throughout the tokamak discharge process. First, the principles of tokamak devices, discharge phases, and typical control problems are introduced, followed by an overview of the fundamental theories and algorithmic frameworks of reinforcement learning. Subsequently, according to different stages of the tokamak discharge process, representative reinforcement learning control problems in the current ramp-up, flat-top, and ramp-down phases are summarized and compared. Recent studies on the application of world-model-based reinforcement learning methods to tokamak control are further reviewed, and their potential for improving sample efficiency and policy generalization, compared with traditional modeling based on plasma dynamics, is analyzed. Finally, challenges and future research directions of reinforcement learning in tokamak control are discussed from the perspectives of system complexity, partial observability, safety constraints, sim-to-real gaps, and real-time deployment. This review aims to provide a reference for future research and applications of reinforcement learning methods in tokamaks.1)
2 1https://gafusion.github.io/doc/cgyro.html 2https://transp.pppl.gov/ 3https://www.jorek.eu/ 4https://imas-data-dictionary.readthedocs.io/en/latest/ 5https://github.com/google-deepmind/torax 2)1 6https://github.com/MIT-PSFC/PopDownGym 7https://doi.org/10.5281/zenodo.16621120 -
图 3 托卡马克装置典型横截面结构示意图
Fig. 3 Schematic diagram of a typical tokamak device cross-sectional structure
图 10 “物理–数据”驱动动力学世界模型[37], 经许可转载自文献[37], ©Springer Nature, 2025((a)“物理–神经”混合世界模型的建模流程; (b)基于该模型的PPO约束策略训练流程)
Fig. 10 Physics-data-driven dynamics world model[37], reproduced with permission from reference [37], ©Springer Nature, 2025 ((a) Modeling process of the physics-neural hybrid world model; (b) PPO-based constrained policy training process using the model)
图 11 平顶阶段等离子体控制任务的问题特征与层级示意图
Fig. 11 Schematic diagram of the problem characteristics and hierarchy of plasma control tasks in the flat-top phase
图 13 磁平衡与形状控制方法的控制链条对比
Fig. 13 Comparison of control chains for magnetic equilibrium and shape control methods
图 14 强化学习磁控架构及TCV装置与等离子体截面示意图[13], 经许可转载自文献[13], ©Springer Nature, 2022((a)仿真训练循环; (b)仿真环境交互逻辑; (c)控制策略网络结构; (d)硬件部署流程; (e)强化学习单网络控制架构; (f)传统嵌套式控制架构; (g)TCV装置19组控制线圈布局; (h)等离子体截面关键参数标注)
Fig. 14 Overview of the reinforcement learning-based magnetic control architecture, the TCV tokamak device, and plasma cross-section[13], reproduced with permission from reference [13], ©Springer Nature, 2022 ((a) Simulation training loop; (b) Interaction logic of the simulation environment; (c) Structure of the control policy network; (d) Hardware deployment pipeline; (e) Single-network reinforcement learning control architecture; (f) Conventional nested control architecture; (g) Layout of the 19 control coil sets of the TCV device; (h) Key annotated parameters of the plasma cross-section)
图 15 不同强化学习训练策略(训练A、训练B、训练C)与基线传统控制方法(ITER VS)下的垂直稳定性控制响应曲线[40], 经许可转载自文献[40], ©Elsevier, 2023
Fig. 15 Vertical stability control response curves for different RL training policies (Training A, Training B, Training C) and a baseline traditional control method (ITER VS)[40], reproduced with permission from reference [40], ©Elsevier, 2023
图 18 传统物理模型与三种托卡马克世界模型的对比
Fig. 18 Comparison between traditional physical models and three types of tokamak world models
图 21 强化学习在托卡马克等离子体控制中面临的主要挑战概览
Fig. 21 Overview of major challenges in applying reinforcement learning to tokamak plasma control
表 1 人工智能/机器学习在可控核聚变领域综述的对比
Table 1 Comparison of reviews on AI/ML in the field of controlled nuclear fusion
工作 以聚变为主体 围绕托卡
马克展开RL主线 等离子体
世界建模显式区分阶段 文献[16] $ \checkmark $ $ \times $ $ \times $ $ \checkmark $ $ \times $ 文献[19] $ \checkmark $ $ \times $ $ \times $ $ \checkmark $ $ \times $ 文献[18] $ \checkmark $ $ \checkmark $ $ \times $ $ \times $ $ \times $ 文献[17] $ \times $ $ \times $ $ \times $ $ \checkmark $ $ \times $ 本文 $ \checkmark $ $ \checkmark $ $ \checkmark $ $ \checkmark $ $ \checkmark $ 
下载: 导出CSV
表 2 托卡马克关键物理变量及定义
Table 2 Definitions of key physical variables in tokamak
符号 名称 定义及物理意义 $ T $ 等离子体温度 离子与电子群体的温度, 用以克服库仑斥力实现有效聚变反应. $ P_{1\sim4} $ 边界点 等离子体边界沿$ z $轴的最高点$ P_1 $与最低点$ P_2 $、沿$ r $轴的最内点$ P_3 $与最外点$ P_4 $, 见图3右侧. $ R $ 大半径 托卡马克装置中心孔至等离子体中心的距离, 影响几何和磁约束特性. $ a $ 小半径 等离子体截面的径向半宽度, 定义为$ a=(r(P_4)-r(P_3))/2 $. $ b $ 垂直高度 等离子体截面的垂直尺度, 定义为$ b=z(P_1)-z(P_2) $. $ \kappa $ 拉长比 等离子体截面的形状参数, 定义为$ \kappa=b/(2a) $, 表征截面在垂直方向的拉伸程度. $ \delta $ 三角形变 等离子体横截面向外或向内偏斜程度, 是等离子体边界形状的无量纲度量, 影响MHD稳定性与输运行为. $ B_T $ 环向磁场强度 由外部线圈产生的主磁场分量, 为主要磁约束力提供支持. $ B_p $ 极向磁场强度 由等离子体电流及极向线圈产生的磁场分量, 与环向磁场叠加构成螺旋闭合磁力线. $ \psi $ 极向磁通 描述极向方向上的磁通函数, 是表征托卡马克磁面结构与磁约束拓扑的重要物理量, 常用于定义磁通面坐标. $ q $ 安全因子 描述磁力线在环向与极向上的缠绕比, 是衡量磁场拓扑与稳定性的关键参数, 该因子对磁流体不稳定性具有重要影响. $ q_{95} $ 边界安全因子 在归一化极向磁通$ \psi=0.95 $的磁通面处的安全因子值, 常用于表征等离子体边缘区域的磁场拓扑结构, 并与多种MHD稳定性密切相关. $ \beta $ 等离子体比压 等离子体压强与磁压强之比, 是衡量磁场约束效率的重要参数. $ \beta_N $ 归一化比压 对$ \beta $进行装置尺寸与等离子体电流归一化后的量, 用于不同托卡马克装置之间的性能比较, 是工程上常用的稳定性与性能指标. $ I_p $ 等离子体电流 电流规模决定极向磁场强度, 对等离子体稳定性与约束性能具有深刻影响. $ Q $ 聚变增益因子 聚变输出功率与输入加热功率的比值, $ Q>1 $表示输出功率大于输入功率, 是聚变装置性能的重要指标.
下载: 导出CSV
表 3 爬升与下降阶段代表性强化学习工作归纳
Table 3 Summary of representative reinforcement learning works in the ramp-up and ramp-down phases
下载: 导出CSV
表 4 平顶阶段代表性强化学习工作对比
Table 4 Comparison of representative reinforcement learning works in the flat-top phase
代表工作 控制任务 装置 观测与动作 奖励 应用特点 文献[13] 磁位形实时控制 TCV 磁诊断信息; PF线圈电压 位形误差; 约束惩罚 失稳终止; 实体实验验证; 端到端实时控制 文献[33] 增强型磁位形控制 TCV 磁诊断信息、积分误差信息; PF线圈电压 位形误差、X点误差; 稳态误差; 约束惩罚 实体实验验证; 提升复杂任务训练效率 文献[40] 垂直稳定控制 ITER VS回路电流、磁诊断信息; VS回路电压 位移误差、速度误差; 控制代价 考虑磁控回路耦合; 失稳终止; 高频子任务 文献[35] $ \beta_N $前馈场景设计 KSTAR $ \beta_N $; 场景控制旋钮 $ \beta_N $误差 实体实验验证; 前馈设计 文献[41] $ \beta_p $实时反馈控制 EAST $ \beta_p $误差、误差积分; 4.6 GHz LHW调节量 $ \beta_p $误差; 误差积分 噪声处理; 执行器限制; 实体实验验证; 单参数实时反馈 文献[15] $ \beta_p $、$ q_{95} $与$ l_i $多目标运行轨迹设计 KSTAR $ \beta_p $、$ q_{95} $、$ l_i $、目标值、上一时刻动作、NBI功率; 电流与边界形状轨迹 $ \beta_p $、$ q_{95} $、$ l_i $误差 目标范围设定; 实体实验验证; 自动生成运行点 文献[42] 离子温度梯度控制 JT-60U 梯度代理量、当前/历史NBI功率; NBI控制指令 温度梯度误差 多工况训练; 慢时间尺度剖面控制 文献[43] 安全因子剖面与$ \beta_N $协同控制 JT-60SA 输运特性/ITB强度估计; NB、EC功率设定 安全因子剖面误差; $ \beta_N $误差 分层动作设计; 剖面--全局量耦合控制 文献[44] 撕裂模前瞻规避 DIII-D 多模态诊断重构剖面状态; 总束功率、三角形度高层指令 高性能运行指标; 撕裂风险; 风险阈值引导 PCS协同; 实体实验验证; 分层实时控制 文献[45] 密度调节与破裂规避 — 密度状态、安全裕度; 加料控制指令 密度误差; 约束惩罚 安全强化学习
下载: 导出CSV
表 5 托卡马克控制中世界模型主要范式对比
Table 5 Comparison of major world-modeling paradigms for tokamak control
模型类别 代表工作 建模方式 主要优势 主要局限 典型适用场景 传统物理模型 文献[23, 73−75] 基于MHD、输运等物理方程; 主要依赖理论建模与参数整定 物理一致性强; 可解释性高; 新工况外推能力强; 适合作为离线验证基准 计算与交互开销高; 简化假设带来失配; 难直接支撑高频RL训练或实时闭环 高保真仿真; 平衡与稳定性分析 数据驱动世界模型 文献[44, 46−48] 基于神经网络; 利用历史放电、诊断与控制数据训练 训练/推理开销低; 易构建代理环境; 具备实时推理潜力 长时序误差累积; 物理约束弱; 泛化能力有限 风险预测; 多时间尺度代理建模 物理--数据混合世界模型 文献[37, 49−50] 物理方程骨架结合神经网络修正; 利用仿真与实验数据校准或训练 兼顾样本效率、物理一致性与建模灵活性; 工程可行性较强 仍依赖结构假设; 简化假设带来失配; 泛化能力有限; 当前仍以离线支撑为主 下降阶段建模; 鲁棒轨迹设计; 约束策略训练环境 基于元学习的模型融合 文献[51] 物理模型与数据模型融合; 利用实验数据训练融合器 新工况外推能力强; 可动态融合多模型 依赖候选模型质量; 融合器过复杂会退化; 泛化能力有限; 尚非实时闭环主流方案 剖面预测; 跨模型融合; 新工况外推
下载: 导出CSV
表 6 主流托卡马克物理仿真平台对比分析
Table 6 Comparative analysis of mainstream tokamak physics simulation platforms
平台名称 开发者 核心功能 主要特点 优势与局限性 CGYRO[76] 1 美国通用原子能公司(general atomics, GA) 等离子体湍流输运高精度模拟 基于陀螺动力学理论, 支持全几何位形 优点: 精度极高;
缺点: 计算开销大, 不适合实时
控制RAPTOR[77] 瑞士SPC /
荷兰DIFFER输运演化与控制优化 轻量化控制导向, 支持快速预测与优化 优点: 运行快, 适合RL训练;
缺点: 简化假设多TRANSP[78] 1 美国PPPL 实验数据解释与预测性输运模拟 集成多物理模块, 通用标准码 优点: 模型成熟;
缺点: 串行慢, 实时性差JOREK[79] 1 EUROfusion 非线性MHD稳定性分析 专注大尺度不稳定性 优点: 可处理壁相互作用;
缺点: 收敛慢ASTRA[80] 德国IPP 自动输运分析系统 模块化1.5D输运码 优点: 灵活;
缺点: 缺乏2D/3D细节ETS/IMAS[81] 1 ITER组织 综合建模分析平台 统一数据结构, 多求解器集成 优点: 标准化强;
缺点: 系统复杂COTSIM[82] 中科院等离子体所 控制系统工程仿真平台 面向EAST控制设计 优点: 适合快速仿真;
缺点: 非线性描述有限TORAX[83] 1 DeepMind / SPC 可微输运仿真器 基于JAX, 支持自动微分 优点: GPU加速;
缺点: 生态尚不完善
下载: 导出CSV
-
[1] Ongena J, Koch R, Wolf R, Zohm H. Magnetic-confinement fusion. Nature Physics, 2016, 12(5): 398−410 doi: 10.1038/nphys3745 [2] 中国共产党第二十届中央委员会. 中共中央关于制定国民经济和社会发展第十五个五年规划的建议[Online], available: https://www.gov.cn/gongbao/2025/issue_12386/202511/content_7047415.html, 2026-07-17Central Committee of the Communist Party of China. Recommendations of the Central Committee of the Communist Party of China for Formulating the 15th Five-Year Plan for National Economic and Social Development[Online], available: https://english.www.gov.cn/news/202510/28/content_WS6900adb9c6d00ca5f9a07216.html, July 17, 2026 [3] U.S. Department of Energy. Fusion Science & Technology Roadmap[Online], available: https://www.energy.gov/fusion-energy, July 17, 2026. [4] EUROfusion. EUROfusion Roadmap to Fusion Energy[Online], available: https://euro-fusion.org/eurofusion/roadmap/, July 17, 2026. [5] Wan B, Xu G. Steady-state burning plasma: a new stage in the development of magnetic confinement fusion energy. National Science Review, 2023, 10(12): Article No. nwad217 doi: 10.1093/nsr/nwad217 [6] Hurricane O A. Inertial confinement fusion: Status and challenges. Annual Review of Nuclear and Particle Science, 2025, 75(1): 153−172 doi: 10.1146/annurev-nucl-102622-022842 [7] 赵冬斌, 唐振韬, 邵坤, 朱圆恒. 深度强化学习进展: 从AlphaGo到AlphaGo Zero. 控制理论与应用, 2017, 34(12): 1529−1546 doi: 10.7641/CTA.2017.70808Zhao Dong-Bin, Tang Zhen-Tao, Shao Kun, Zhu Yuan-Heng. Recent progress of deep reinforcement learning: from AlphaGo to AlphaGo Zero. Control Theory & Applications, 2017, 34(12): 1529−1546 doi: 10.7641/CTA.2017.70808 [8] 唐振韬, 梁荣钦, 朱圆恒, 赵冬斌. 实时格斗游戏的智能决策方法. 控制理论与应用, 2022, 39(6): 969−985 doi: 10.7641/CTA.2022.10995Tang Zhen-Tao, Liang Rong-Qin, Zhu Yuan-Heng, Zhao Dong-Bin. Intelligent decision making approaches for real time fighting game. Control Theory & Applications, 2022, 39(6): 969−985 doi: 10.7641/CTA.2022.10995 [9] Chai J, Zhao Z, Zhu Y, Zhao D. A survey of cooperative multi-agent reinforcement learning for multi-task scenarios. Artificial Intelligence Science and Engineering, 2025, 1(2): 98−121 doi: 10.23919/AISE.2025.000008 [10] Silver D, Huang A, Maddison C J, Guez A, Sifre L, Van Den Driessche G, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 2016, 529(7587): 484−489 doi: 10.1038/nature16961 [11] Vinyals O, Babuschkin I, Czarnecki W M, Mathieu M, Dudzik A, Chung J, et al. Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature, 2019, 575(7782): 350−354 doi: 10.1038/s41586-019-1724-z [12] Kalyan K S. A survey of GPT-3 family large language models including ChatGPT and GPT-4. Natural Language Processing Journal, 2024, 6: Article No. 100048 doi: 10.1016/j.nlp.2023.100048 [13] Degrave J, Felici F, Buchli J, Neunert M, Tracey B, Carpanese F, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022, 602(7897): 414−419 doi: 10.1038/s41586-021-04301-9 [14] Kates-Harbeck J, Svyatkovskiy A, Tang W. Predicting disruptive instabilities in controlled fusion plasmas through deep learning. Nature, 2019, 568(7753): 526−531 doi: 10.1038/s41586-019-1116-4 [15] Seo J, Na Y S, Kim B, Lee C, Park M, Park S, et al. Development of an operation trajectory design algorithm for control of multiple 0D parameters using deep reinforcement learning in KSTAR. Nuclear Fusion, 2022, 62(8): Article No. 086049 doi: 10.1088/1741-4326/ac79be [16] Humphreys D, Kupresanin A, Boyer M D, Canik J, Chang C, Cyr E C, et al. Advancing fusion with machine learning research needs workshop report. Journal of Fusion Energy, 2020, 39(4): 123−155 doi: 10.1007/s10894-020-00258-1 [17] Arhouni F E, Abdo M A S, Ouakkas S, Bouhssa M L, Boukhair A. Artificial intelligence-driven advances in nuclear technology: Exploring innovations, applications, and future prospects. Annals of Nuclear Energy, 2025, 213: Article No. 111151 doi: 10.1016/j.anucene.2024.111151 [18] Priyanka M, Sangeetha J, Jayakumar C. A review of traditional and data-driven approaches for disruption prediction in different tokamaks. E3S Web of Conferences, 2024, 477: Article No. 00039 doi: 10.1051/e3sconf/202447700039 [19] Pavone A, Merlo A, Kwak S, Svensson J. Machine learning and Bayesian inference in nuclear fusion research: an overview. Plasma Physics and Controlled Fusion, 2023, 65(5): Article No. 053001 doi: 10.1088/1361-6587/acc60f [20] Lawson J D. Some criteria for a power producing thermonuclear reactor. Proceedings of the Physical Society. Section B, 1957, 70(1): 6−10 doi: 10.1088/0370-1301/70/1/303 [21] Chen F F. Introduction to Plasma Physics and Controlled Fusion: Vol. 1. New York, NY, USA: Plenum Press, 1984. [22] Wesson J, Campbell D J. Tokamaks: Vol. 149. Oxford, UK: Oxford University Press, 2011. [23] Freidberg J P. Ideal MHD. Cambridge, UK: Cambridge University Press, 2014. [24] Ongena J. Fusion: A true challenge for an enormous reward. EPJ Web of Conferences, 2015, 98: Article No. 05004 doi: 10.1051/epjconf/202226800011 [25] Zohm H. Edge localized modes (ELMs). Plasma Physics and Controlled Fusion, 1996, 38(2): 105−128 doi: 10.1088/0741-3335/38/2/001 [26] Horton W. Drift waves and transport. Reviews of Modern Physics, 1999, 71(3): 735−778 doi: 10.1103/RevModPhys.71.735 [27] Fisch N J. Theory of current drive in plasmas. Reviews of Modern Physics, 1987, 59(1): 175−234 doi: 10.1103/RevModPhys.59.175 [28] Singh M. Status of heating and current drive systems planned for ITER. IEEE Transactions on Plasma Science, 2016, 44(9): 1514−1524 doi: 10.1109/TPS.2016.2577709 [29] Ariola M, Pironti A. Magnetic Control of Tokamak Plasmas. London, UK: Springer, 2008. [30] Felici F, Sauter O, Coda S, Duval B, Goodman T, Moret J, et al. Real-time physics-model-based simulation of the current density profile in tokamak plasmas. Nuclear Fusion, 2011, 51(8): Article No. 083052 doi: 10.1088/0029-5515/51/8/083052 [31] Humphreys D, Ferron J, Johnson R, Leuer J, Penaflor B, Walker M, et al. Integrated plasma control for advanced tokamaks. In: Proceedings of 20th IEEE/NPSS Symposium on Fusion Engineering. San Diego, CA, USA: IEEE, 2003. 20-26. [32] Sutton R S, Barto A G. Reinforcement Learning: An Introduction. Second ed. Cambridge, MA, USA: The MIT Press, 2018. [33] Tracey B D, Michi A, Chervonyi Y, Davies I, Paduraru C, Lazic N, et al. Towards practical reinforcement learning for tokamak magnetic control. Fusion Engineering and Design, 2024, 200: Article No. 114161 doi: 10.1016/j.fusengdes.2024.114161 [34] Char I, Abbate J, Bardóczi L, Boyer M, Chung Y, Conlin R, et al. Offline model-based reinforcement learning for tokamak control. In: Proceedings of Learning for Dynamics and Control Conference. University of Pennsylvania, Philadelphia, PA, USA: PMLR, 2023. 1357-1372. [35] Seo J, Na Y S, Kim B, Lee C, Park M, Park S, et al. Feedforward beta control in the KSTAR tokamak by deep reinforcement learning. Nuclear Fusion, 2021, 61(10): Article No. 106010 doi: 10.1088/1741-4326/ac121b [36] Wakatsuki T, Suzuki T, Hayashi N, Oyama N, Ide S, Kamada Y. Safety factor profile control with reduced CS flux consumption during plasma current ramp-up phase using reinforcement learning technique. In: Proceedings of 27th IAEA Fusion Energy Conference Programme and Book of Abstracts. Gandhinagar, India: IAEA, 2018. 271-271. [37] Wang A M, Rea C, So O, Dawson C, Garnier D T, Fan C. Active ramp-down control and trajectory design for tokamaks with neural differential equations and reinforcement learning. Communications Physics, 2025, 8(1): Article No. 231 doi: 10.1038/s42005-025-02146-6 [38] Subbotin G, Sorokin D, Nurgaliev M, Granovskiy A, Kharitonov I, Adishchev E, et al. Reconstruction-free magnetic control of DIII-D plasma with deep reinforcement learning. arXiv: 2506.13267, 2025. [39] Kerboua-Benlarbi S, Nouailletas R, Faugeras B, Nardon E, Moreau P. Magnetic control of WEST plasmas through deep reinforcement learning. IEEE Transactions on Plasma Science, 2024, 52(9): 3698−3703 doi: 10.1109/TPS.2024.3377811 [40] Dubbioso S, De Tommasi G, Mele A, Tartaglione G, Ariola M, Pironti A. A deep reinforcement learning approach for vertical stabilization of tokamak plasmas. Fusion Engineering and Design, 2023, 194: Article No. 113725 doi: 10.1016/j.fusengdes.2023.113725 [41] Zhang Y, Wang S, Yuan Q, Xiao B, Huang Y. Real-time feedback control of βp based on deep reinforcement learning on EAST. Plasma Physics and Controlled Fusion, 2024, 66(5): Article No. 055014 [42] Wakatsuki T, Suzuki T, Oyama N, Hayashi N. Ion temperature gradient control using reinforcement learning technique. Nuclear Fusion, 2021, 61(4): Article No. 046036 doi: 10.1088/1741-4326/abe68d [43] Wakatsuki T, Yoshida M, Narita E, Suzuki T, Hayashi N. Simultaneous control of safety factor profile and normalized beta for JT-60SA using reinforcement learning. Nuclear Fusion, 2023, 63(7): Article No. 076017 doi: 10.1088/1741-4326/acd393 [44] Seo J, Kim S, Jalalvand A, Conlin R, Rothstein A, Abbate J, et al. Avoiding fusion plasma tearing instability with deep reinforcement learning. Nature, 2024, 626(8000): 746−751 doi: 10.1038/s41586-024-07024-9 [45] Paruchuri S T, Ward I, Rist N, Graber V, Al Khawaldeh H, Wang Z, et al. Density regulation with disruption avoidance in next-generation tokamaks using a safe reinforcement learning-based controller. Fusion Engineering and Design, 2025, 216: Article No. 115064 doi: 10.1016/j.fusengdes.2025.115064 [46] Kit A, Järvinen A, Poels Y, Wiesen S, Menkovski V, Fischer R, et al. On learning latent dynamics of the AUG plasma state. Physics of Plasmas, 2024, 31(3): Article No. 032504 doi: 10.1063/5.0174128 [47] Wu N, Yang Z, Li R, Wei N, Chen Y, Dong Q, et al. High-fidelity data-driven dynamics model for reinforcement learning-based control in HL-3 tokamak. Communications Physics, 2025, 8(1): Article No. 393 doi: 10.1038/s42005-025-02302-y [48] Sonker R, Capone A, Rothstein A, Kaga H J F, Kolemen E, Schneider J. Multi-timescale dynamics model bayesian optimization for plasma stabilization in tokamaks. arXiv: 2506.10287, 2025. [49] Liu Z, Stacey W M. Application of neural ordinary differential equations for tokamak plasma dynamics analysis. arXiv: 2403.01635, 2024. [50] Wang A M, Pau A, Rea C, So O, Dawson C, Sauter O, et al. Learning plasma dynamics and robust rampdown trajectories with predict-first experiments at TCV. Nature Communications, 2025, 16(1): Article No. 8877 doi: 10.1038/s41467-025-63917-x [51] Abbate J, Fable E, Tardini G, Fischer R, Kolemen E, Team A U, et al. Combining physics-based and data-driven models for quantitatively accurate plasma profile prediction that extrapolates well; with application to DIII-D, AUG, and ITER tokamaks. Nuclear Fusion, 2025, 65(5): Article No. 056014 doi: 10.1088/1741-4326/adc283 [52] Hausknecht M, Stone P. Deep reinforcement learning in parameterized action space. In: International Conference on Learning Representations. San Juan, Puerto Rico, 2016. [53] Watkins C J, Dayan P. Q-learning. Machine Learning, 1992, 8(3): 279−292 [54] Mnih V, Kavukcuoglu K, Silver D, Rusu A A, Veness J, Bellemare M G, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518(7540): 529−533 doi: 10.1038/nature14236 [55] Fujimoto S, Hoof H, Meger D. Addressing function approximation error in actor-critic methods. In: Proceedings of International Conference on Machine Learning. Stockholm, Sweden: PMLR, 2018. 1587-1596. [56] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O. Proximal policy optimization algorithms. arXiv: 1707.06347, 2017. [57] Abdolmaleki A, Springenberg J T, Tassa Y, Munos R, Heess N, Riedmiller M. Maximum a posteriori policy optimisation. In: International Conference on Learning Representations. Vancouver, Canada, 2018. [58] Lillicrap T P, Hunt J J, Pritzel A, Heess N, Erez T, Tassa Y, et al. Continuous control with deep reinforcement learning. In: International Conference on Learning Representations. San Juan, Puerto Rico, 2016. [59] Lange S, Gabel T, Riedmiller M. Batch reinforcement learning//Reinforcement Learning: State-of-the-art. Berlin, Heidelberg, Germany: Springer, 2012. 45-73. [60] Levine S, Kumar A, Tucker G, Fu J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv: 2005.01643, 2020. [61] Fujimoto S, Meger D, Precup D. Off-policy deep reinforcement learning without exploration. In: Proceedings of International Conference on Machine Learning. Long Beach, CA, USA: PMLR, 2019. 2052-2062. [62] Kumar A, Zhou A, Tucker G, Levine S. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 2020, 33: 1179−1191 doi: 10.52202/068431-0125 [63] Wu Y, Tucker G, Nachum O. Behavior regularized offline reinforcement learning. arXiv: 1911.11361, 2019. [64] Jaques N, Ghandeharioun A, Shen J H, Ferguson C, Lapedriza A, Jones N, et al. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv: 1907.00456, 2019. [65] Hafner D, Lillicrap T, Ba J, Norouzi M. Dream to control: Learning behaviors by latent imagination. In: International Conference on Learning Representations. Virtual Event, 2020. [66] Hafner D, Lillicrap T P, Norouzi M, Ba J. Mastering Atari with discrete world models. In: International Conference on Learning Representations. Virtual Event, 2021. [67] Altman E. Constrained Markov Decision Processes. New York, NY, USA: Routledge, 2021. [68] Achiam J, Held D, Tamar A, Abbeel P. Constrained policy optimization. In: Proceedings of International Conference on Machine Learning. Sydney, Australia: PMLR, 2017. 22-31. [69] Tessler C, Mankowitz D J, Mannor S. Reward constrained policy optimization. In: International Conference on Learning Representations. New Orleans, Louisiana, USA, 2019. [70] Urano H, Fujita T, Ide S, Miyata Y, Matsunaga G, Matsukawa M. Development of operation scenarios for plasma breakdown and current ramp-up phases in JT-60SA tokamak. Fusion Engineering and Design, 2015, 100: 345−356 doi: 10.1016/j.fusengdes.2015.06.138 [71] Martin Y, Takizuka T. Power requirement for accessing the H-mode in ITER. Journal of Physics: Conference Series, 2008, 123(1): Article No. 012033 doi: 10.1088/1742-6596/123/1/012033 [72] Lazarus E, Lister J, Neilson G. Control of the vertical instability in tokamaks. Nuclear Fusion, 1990, 30(1): 111−125 doi: 10.1088/0029-5515/30/1/010 [73] Braginskii S. Transport processes in a plasma. Reviews of Plasma Physics, 1965, 1: 205−311 [74] Rosenbluth M N, MacDonald W M, Judd D L. Fokker-Planck equation for an inverse-square force. Physical Review, 1957, 107(1): 1−6 doi: 10.1103/physrev.107.1 [75] Hager R, Yoon E, Ku S, D'Azevedo E F, Worley P H, Chang C S. A fully non-linear multispecies Fokker–Planck–Landau collision operator for simulation of fusion plasma. Journal of Computational Physics, 2016, 315: 644−660 doi: 10.1016/j.jcp.2016.03.064 [76] Candy J, Belli E A, Bravenec R. A high-accuracy eulerian gyrokinetic solver for collisional plasmas. Journal of Computational Physics, 2016, 324: 73−93 doi: 10.1016/j.jcp.2016.07.039 [77] Felici F, Sauter O. Non-linear model-based optimization of actuator trajectories for tokamak plasma profile control. Plasma Physics and Controlled Fusion, 2012, 54(2): Article No. 025002 doi: 10.1088/0741-3335/54/2/025002 [78] Pankin A Y, Breslau J, Gorelenkova M, Andre R, Grierson B, Sachdev J, et al. TRANSP integrated modeling code for interpretive and predictive analysis of tokamak plasmas. Computer Physics Communications, 2025, 312: Article No. 109611 doi: 10.1016/j.cpc.2025.109611 [79] Huysmans G, Czarny O. MHD stability in x-point geometry: simulation of ELMs. Nuclear Fusion, 2007, 47(7): 659−666 doi: 10.1088/0029-5515/47/7/016 [80] Pereverzev G, Yushmanov P N. ASTRA automated system for transport analysis in a tokamak, Technical Report IPP–5-98, Max-Planck-Institut für Plasmaphysik, Garching, Germany, 2002. [81] Imbeaux F, Pinches S, Lister J, Buravand Y, Casper T, Duval B, et al. Design and first applications of the ITER integrated modelling & analysis suite. Nuclear Fusion, 2015, 55(12): Article No. 123006 doi: 10.1088/0029-5515/55/12/123006 [82] Wang Z, Schuster E, Rafiq T, Huang Y, Luo Z, Yuan Q, et al. Enabling model-based scenario control in EAST by fast surrogate modeling within COTSIM. Fusion Engineering and Design, 2025, 215: Article No. 114969 doi: 10.1016/j.fusengdes.2025.114969 [83] Citrin J, Goodfellow I, Raju A, Chen J, Degrave J, Donner C, et al. TORAX: A fast and differentiable tokamak transport simulator in JAX. arXiv: 2406.06718, 2024. [84] Eldon D, Kolemen E, Barton J L, Briesemeister A R, Humphreys D A, Leonard A W, et al. Controlling marginally detached divertor plasmas. Nuclear Fusion, 2017, 57(6): Article No. 066039 [85] Kerboua-Benlarbi S, Nouailletas R, Faugeras B, Moreau P. Curriculum reinforcement learning for tokamak control. In: Proceedings of International Joint Conference on Artificial Intelligence. Jeju, South Korea: Springer Nature Singapore, 2024. 39-56. [86] Farea A, Yli-Harja O, Emmert-Streib F. Understanding physics-informed neural networks: Techniques, applications, trends, and challenges. AI, 2024, 5(3): 1534−1557 doi: 10.3390/ai5030074 [87] Rodriguez A, Parr R, Koller D. Reinforcement learning using approximate belief states. In: Proceedings of Advances in Neural Information Processing Systems. Denver, CO, USA: Curran Associates, Inc., 1999. 1036-1042. [88] Zhang L, Qi Z, Shi Y. Multi-objective reinforcement learning–concept, approaches and applications. Procedia Computer Science, 2023, 221: 526−532 [89] Yan Z, Xu F, Tan J, Liu H, Liang B. Reinforcement learning-based integrated active fault diagnosis and tracking control. ISA Transactions, 2023, 132: 364−376 doi: 10.1016/j.isatra.2022.06.020 [90] Garcıa J, Fernández F. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 2015, 16(1): 1437−1480 [91] Matsuo Y, LeCun Y, Sahani M, Precup D, Silver D, Sugiyama M, et al. Deep learning, reinforcement learning, and world models. Neural Networks, 2022, 152: 267−275 doi: 10.1016/j.neunet.2022.03.037 [92] Xie T, Zhao S, Wu C H, Liu Y, Luo Q, Zhong V, et al. Text2Reward: Automated dense reward function generation for reinforcement learning. In: International Conference on Learning Representations. Vienna, Austria, 2024. [93] Ma R, Luijkx J, Ajanović Z, Kober J. ExploRLLM: Guiding exploration in reinforcement learning with large language models. In: Proceedings of 2025 IEEE International Conference on Robotics and Automation (ICRA). Atlanta, GA, USA: IEEE, 2025. 9011-9017. -
下载:
计量
- 文章访问数: 6
- HTML全文浏览量: 3
- 被引次数: 0
