From Rule-driven to Collective Intelligence Emergence: A Review of Research on Multi-robot Air-ground Collaboration
-
摘要: 多机器人空地协同系统作为一种在搜索救援、自主探索等领域具有广泛应用前景的异构机器人协作系统, 近年来受到研究者的高度关注. 针对限制空地协同系统自治性能的低智能性、弱自主性挑战, 如何增强个体智能、提高群体协同自主性是加快空地系统应用落地亟需解决的关键问题. 近年来, 随着以深度学习、群体智能为代表的人工智能(Artificial intelligence, AI)算法在感知、决策等领域的不断发展, 将其应用于空地协同系统成为了当前的研究热点. 基于空地协同的自主化程度, 总结从规则驱动到群智涌现不同协作水平下的空地协同工作, 强调通过增强个体智能涌现群体智慧. 同时, 构建并拓宽空地协同群智系统的概念及要素, 阐述其自组织、自适应、自学习与持续演化的群智特性. 最后, 通过列举空地协同代表性应用场景, 总结空地协同所面临的挑战, 并展望未来方向.Abstract: The multi-robot air-ground collaboration system, which is crucial for search and rescue, exploration, and other fields, has garnered significant attention from researchers in recent years. Overcoming challenges related to limited intelligence and weak autonomy in such systems is essential to enhance individual intelligence and strengthen collective collaboration autonomy, thereby accelerating their practical applications. In recent years, with the continuous advancement of artificial intelligence (AI) algorithms in perception and decision-making, such as deep learning and collective intelligence, their applications to air-ground collaborative systems have become a research hotspot. Based on the level of autonomy in air-ground collaboration, this paper summarizes air-ground collaboration efforts at different collaboration levels, ranging from rule-driven approaches to collective intelligence emergence, emphasizing the enhancement of individual intelligence to achieve collective intelligence. Furthermore, this paper constructs the concepts and expands the features of the air-ground collaboration collective intelligence system, and outlines its self-organizing, self-adaptation, self-learning, and continuously evolving qualities. Finally, by listing representative application scenarios, this paper encapsulates the challenges and explores future directions in air-ground collaboration.
-
我国化纤产业作为国民经济发展的关键支柱产业之一, 近年来逐渐明确了高质量发展的需求, 即推动化纤制造向中高端迈进, 实现成为化纤强国的目标[1]. 其中, 化纤丝线作为一种重要的合成纤维产品, 在纺织品制作等领域因其耐磨、高强度和高弹性等优点, 得到广泛应用[2]. 在化纤丝线的质量检测过程中, 网络度作为质量评估的关键指标之一, 因其具有直观且易于观察与统计的特点, 在许多企业中广泛应用. 网络度是指单位长度的化纤丝线中, 具有一定牢度的未散开的网络结数量. 以锦纶6型全拉伸丝为例, 丝线在水面上散开后呈现的形态如图1所示. 对于正常的化纤丝线, 其网络结数需要在规定的范围内. 当网络结数偏多时, 使丝线在松弛加工过程中无法充分解捻, 同时在染色时上染率不同, 导致面料表面出现斑点. 相反, 当网络结数偏少时, 使丝线在织造过程中, 网络内部存在张力, 导致丝线网络结点松散, 进而产生毛羽. 总之, 化纤丝线的网络度作为重要的质量指标, 对于化纤产品的制造工艺和织物的最终品质都有着不可忽视的影响, 同时对网络度的准确评估具有重要意义.
在工业领域, 常见的网络度检测方法包括手工移针法、仪器移针法、手工重锤法和水浴法等[3]. 目前常用的网络度检测设备主要基于仪器移针法研发, 但是使用基于仪器移针法的检测设备时, 每次只能检测单根丝线, 难以适用于大规模检测. 因此, 许多化纤生产企业选择水浴法作为网络度检测的方法, 将丝线放入盛有纯净水的水槽中, 待丝线接触水面散开成网络状后, 统计丝线网络结的数量. 然而, 此方法也存在一些局限性. 首先, 人眼目测依赖于主观评价, 易受光照等外界因素的影响, 使网络度检测结果不稳定; 其次, 在产量较大的情况下, 人工检测的能力有限, 难以同时对多根丝线进行检测. 因此需要设计更为准确与高效的化纤丝线网络度检测方法.
与传统工业技术相比, 基于计算机视觉与深度学习的检测技术具有准确率高和抗干扰能力强等诸多优势. 通过研发深度学习算法与自动化检测设备, 实现对化纤丝线网络度的准确检测和量化分析, 从而减少人为主观判断带来的不确定性. 此外, 深度学习算法能够自动学习并提取复杂特征, 高效处理大规模数据, 进而提升质量分析的效率和精确性. 这为化纤行业提供了一种先进的技术手段, 提升了化纤产品的质量控制效果和后道质检结果的准确性. 目前, 基于目标检测的深度学习方法已广泛应用于丝线断裂检测[4]、毛羽检测[5-6]和丝线直径均匀度[7]等方面. 例如, 基于注意力机制改进的CenterNet 模型[8]在丝线毛羽检测方面表现出良好的性能, 通过全局阈值化方法处理样本图像, 可有效分割出丝线主干位置. 然而, 化纤丝线网络结点尺寸较小, 使用上述基于目标检测的方法容易导致丝线结点位置信息关注不足, 从而影响检测结果的准确性.
为提高化纤丝线网络度检测结果的准确性, 本文将网络度检测问题视为像素级的二分类问题, 即将丝线的网络结点位置视为感兴趣区域, 并将样本图像中其他像素区域归属于背景区域, 最后按照像素进行分类. 目前针对此类问题, 主要采用编码器−解码器结构进行特征提取. 其中, 编码器用于提取图像的抽象特征, 解码器用于恢复特征至原始图像尺寸并细化特征. 双线性插值[9-10]和跳跃连接[11]有助于在编码和解码过程中保留细节和边缘信息, 多层空洞卷积[12-14]有助于在解码过程中增加图像的感受野. 然而, 对于丝线网络结点这种小目标而言, 使用大量的卷积层和较大的空洞卷积率容易导致语义信息丢失. 为增强对语义信息的提取能力及检测性能, 注意力机制[15-16]和Transformer[17-19]等方法相继提出. 其中, 通过多尺度融合策略[20]和残差特征金字塔[21]等方式使得模型专注于特定的感兴趣区域, 金字塔特征融合[22-24]和损失注意力机制[25]等方法在特征图融合和损失函数优化方面取得了显著的成果, 提高了对语义信息的提取能力. 然而, 考虑其对计算资源的较高要求, 对小样本数据集的适用性有限, 同时对局部特征信息处理不足, 训练难度较大, 故难以适用于本文的网络度检测问题.
基于以上分析, 本文提出一种基于语义信息增强的化纤丝线网络度检测方法, 提高从化纤丝线图像中提取语义信息的能力, 实现化纤丝线网络度的批量计算, 同时解决人工检测误检率高的问题. 本文的主要贡献如下:
1) 针对化纤丝线网络度结点目标较小, 导致检测准确性低的问题, 提出语义信息增强模块, 使得模型更加关注丝线网络结点区域的信息. 同时, 设计多级特征扩张模块, 使用不同大小的扩张卷积核提取深层特征的语义信息, 能够增强高质量的空间细节信息的提取能力, 提高检测结果的准确性.
2) 针对丝线样本数量大且检测耗时长的问题, 提出网络度并行检测方法, 将化纤丝线图像中的丝线主干提取与网络结点提取两部分并行处理, 实现网络度的批量计算.
3) 针对人工检测误检率高的问题, 研制网络度检测设备, 结合网络度检测算法与掩膜提取方法, 实现网络度自动化视觉检测. 通过使用所研制的设备进行丝线样本采集并建立数据集, 对所提检测模型进行实验验证.
1. 基于语义信息增强的网络度检测算法
本文提出一种基于语义信息增强的“编码−解码”两阶段语义分割检测算法, 模型结构如图2所示. 在编码阶段, 首先基于MobileNetV2[26]进行改进, 用于提取输入图像的特征信息, 并获得包含局部细节信息的浅层特征图和全局语义信息的深层特征图. 然后, 使用交叉注意力模块和全局注意力模块增强浅层特征图中语义信息的提取能力. 同时使用扩张卷积方法提取深层特征图中的多级语义信息. 最后, 将得到的浅层特征信息与深层特征信息通过解码阶段的阶段性特征融合模块进行加权融合, 并将特征图像尺寸恢复至输入图像大小, 得到检测结果.
1.1 主干网络
为保证所提网络度检测算法的可靠性, 本文基于MobileNetV2模型设计主干网络结构. 在这一模型中, 使用深度可分离卷积[27]的方法进行卷积操作. 对于传统卷积操作而言, 这种方法有效地减少了模型的参数数量和计算量, 从而在保持高质量特征提取的同时, 显著地降低了模型的复杂度. 深度可分离卷积将普通的卷积过程分解为深度卷积和逐点卷积两部分, 其中深度卷积用于获得输入特征图中包含的空间信息, 逐点卷积用于将提取的特征进行整合与变换. 使用这种卷积方式能够减少模型的运算量, 提高模型的运算效率. 经下采样操作后, 减小了特征图的尺寸, 同时增加了特征图的通道数. 这一操作通过扩大网络的感受野, 使得模型能够提取更为抽象的特征. 在本文中, 下采样的层数位置设定为2层、4层、8层和10层, 其网络架构如表1所示.
表 1 主干网络架构Table 1 Architecture of the backbone network特征尺寸(像素) 扩展因子 循环次数 输出通道数 步长 512 × 512 × 3 — 1 32 2 256 × 256 × 32 1 2 32 1 128 × 128 × 32 6 4 64 2 64 × 64 × 64 6 2 96 2 32 × 32 × 96 — — — — 1.2 基于语义信息增强模块
在主干网络中, 浅层特征包含原始输入图像中丰富的局部细节和位置信息. 为更加有效地从浅层特征中提取到原始输入图像的语义信息, 即提取像素点及其周围像素之间的关系信息, 本文提出一种基于语义信息增强模块, 对主干网络中的浅层特征图$ {x_{1}} $和$ {x_{2}} $进行处理, 模型结构如图3所示.
对于特征图$ {x_{1}} $, 由于其像素相对较大, 若多次进行特征提取, 将导致模型计算量增加, 影响模型的处理速度. 为确保输出特征图的维度与输入特征图一致, 同时避免增加模型的运算量. 本文首先对特征图$ {x_{1}} $使用$ 1\times1 $卷积核进行卷积操作, 将维度降至原始特征图的一半, 得到特征图$ x_{1}^{\prime} $, 然后采用全局注意力机制的方法进行特征提取, 计算方法为
$$ \begin{array}{*{20}{l}} X_{1}=\sigma(f_{\rm con}(f_{\rm avg}(X_{1}^{\prime}),\;f_{\rm glo}(X_{1}^{\prime}))) \times X_{1}^{\prime} \end{array} $$ (1) $$ \begin{array}{*{20}{l}} X_{1}^{\prime}=\sigma(f_{\rm avg}(x_{1}^{\prime})+f_{\rm glo}(x_{1}^{\prime}))\times x_{1}^{\prime} \end{array} $$ (2) 其中, $ \sigma(\cdot) $表示sigmoid 归一化函数, $ f_{\rm con}(\cdot) $表示特征图融合卷积操作, $ f_{\rm avg}(\cdot) $表示全局平均池化操作, $ f_{\rm glo}(\cdot) $表示全局最大池化操作, $ X_{1}^{\prime} $表示经通道注意力机制处理后的特征图. 对于特征图$ {x_{2}} $, 首先使用$ 1\times1 $卷积核进行卷积操作, 将维度降至原始特征图的一半, 得到特征图$ x_{2}^{\prime} $. 然后使用全局最大池化方法提取空间注意力信息, 将所得特征图与原始特征图$ {x_{2}} $进行相似性判断, 通过归一化的方式获取像素注意力权重. 最后将权重加权赋值到原始图像上, 从像素和通道两种角度增强特征的语义信息, 得到增强后的特征图$ {X_{2}} $. 为获得两个特征图中共同关注的感兴趣区域, 本文通过对比两个特征图中对应像素位置的相似度, 使模型更好地理解不同尺度下的语义信息. 此外, 通过对不同尺度的特征图进行加权融合, 有助于提高语义分割的精度和鲁棒性. 在像素相似性判断中, 常用方法包括点积、余弦相似度和欧氏距离等[28-29]. 综合考虑计算复杂度与数据量, 本文使用自注意力机制中的像素点积方式进行处理, 得到特征图$ {X_{2}} $, 并将其进行上采样至特征图$ {X_{1}} $的尺寸后进行特征融合, 得到特征图H. 计算方法为
$$ \begin{array}{*{20}{l}} X_{2}=x_{2} \times \sigma((f_{\rm glo}(x_{2}^{\prime}))^{\rm T} \times x_{2}^{\prime}) \end{array} $$ (3) $$ \begin{array}{*{20}{l}} H=f_{\rm con}(X_{1},\;f_{\rm up}(X_{2})) \end{array} $$ (4) 其中, $ f_{\rm up}(\cdot) $表示上采样操作, $ X_{1} $表示特征图$ x_{1} $经全局注意力机制模块处理后得到的特征图, $ X_{2} $表示特征图$ x_{2} $经本文提出的自注意力机制模块处理后得到的特征图.
1.3 多级特征扩张模块
在主干网络中, 深层特征中包含更为抽象和丰富的语义信息, 对于图像分析和理解至关重要, 但是使用过于频繁的特征提取可能导致特征信息的丢失.
在DeepLab系列的网络中, 通过采用空洞卷积模块提取深层语义信息, 能够得到多尺度信息. 然而使用单一的空洞卷积结构进行特征提取时, 可能导致不同特征之间的空间相关性不足, 使提取到的有效信息减少, 从而增加模型的参数量等[30]. 因此, 本文提出一种多级特征扩张模块, 旨在增强深层特征图中的语义信息提取能力, 如图4所示.
对于从主干网络中提取的特征图$ {x_{4}} $与上一级特征图$ {x_{3}} $, 按照上述分析进行扩张卷积操作, 即对卷积核的相邻两个权值之间进行扩张, 计算方法为
$$ \begin{split} d_{k}(m,\;n)=\;&\sum\limits_{p=1}^{M} \sum\limits_{q=1}^{N} f(m+k \times p,\; \\ &n+k \times q)\cdot w(p,\;q) \end{split} $$ (5) 其中, $ d_{k}(m,\;n) $表示像素点$ (m,\;n) $经扩张率为k的卷积核扩张后的特征图, $ w(p,\;q) $表示卷积核, $ M $和$ N $分别表示卷积核的长度与宽度.
采用式(5)所示的方法, 使用扩张率为4和6的卷积核分别对特征图$ {x_{3}} $进行上述操作, 经特征融合后得到特征图$ {h_{1}} $. 同时, 使用扩张率为1和3的卷积核分别对特征图$ {x_{4}} $进行卷积, 经特征融合后得到特征图$ {h_{2}} $. 由于使用不同级别的特征图进行扩张卷积后, 生成的特征图的尺寸不同, 无法直接进行融合, 因此需要对特征图$ {h_{2}} $进行上采样操作. 目前传统的上采样方法主要包括最近邻插值、双线性插值和上池化等, 但是这些方法只是简单地复制或插值原始像素值, 忽略了对图像语义信息的保留以及对图像语义信息一致性的考虑, 容易导致特征信息的丢失. 为了克服这一问题, 本文使用卷积注意力机制模块对特征图$ {h_{2}} $进行处理. 首先对特征图每个通道进行全局平均池化和全局最大池化操作, 得到缩放后的全局特征向量, 然后采用反转置卷积的方式将特征图放大至与特征图$ {x_{3}} $相同的尺寸, 最后将上采样后的特征图与缩放后的全局特征向量逐元素相乘, 得到加权后的特征图L, 方法如下:
$$ \begin{array}{*{20}{l}} L=f_{\rm con}(C^{\rm T} \times l,\;h_{1}) \end{array} $$ (6) $$ \begin{array}{*{20}{l}} l=f_{\rm con}(f_{\rm max}(h_{2}),\;f_{\rm avg}(h_{2})) \end{array} $$ (7) 其中, $ f_{\rm max}(\cdot) $表示全局最大池化操作, $ C^{\rm T} $表示转置卷积核矩阵, $ {h_{1}} $和$ {h_{2}} $表示特征图$ {x_{3}} $和$ {x_{4}} $经扩张卷积模块处理并融合后得到的特征图. 由于全局特征向量涵盖整体信息, 因此使用所提方法有助于保持全局特征信息的一致性. 通过与上采样后的特征图进行相似性判断, 使得模型更加专注于提取全局特征信息, 从而增强了模型的特征信息提取能力.
1.4 阶段性特征融合模块
通过使用本文提出的基于语义信息增强模块和多级特征扩张卷积模块分别对主干网络的浅层特征图和深层特征图进行处理, 然后将得到的特征图进行融合, 构成模型的解码器部分. 然而, 直接将语义特征信息通过上采样至输入图像的分辨率大小时, 容易丢失原特征图的语义信息. 因此, 本文设计了阶段性特征融合模块, 如图5所示. 该模块的输入分为两大部分, 包括使用多级特征扩张模块后得到的特征图$ L $和使用基于语义信息增强模块处理后得到的特征图$ H $. 首先, 采用两个不同尺度的分支提取特征$ L $的通道注意力权重, 旨在从多个尺度上提取通道注意力信息. 其中, 第1个分支采用全局平均池化的方法提取全局特征的注意力信息; 第2个分支则采用逐点卷积的方式提取局部特征的通道注意力信息. 随后, 将这两个分支提取到的特征进行融合, 经过归一化操作后, 得到最终的特征权重值. 这一方法旨在有效地结合多尺度的语义特征信息, 以实现更精细的特征融合和更准确的解码结果. 为调节浅层特征图与深层特征图融合的权重, 使用像素级注意力掩膜进行权重分配. 经过相应的注意掩膜相乘后, 将这两种不同尺度的语义特征图对应的像素进行融合, 得到输出结果. 计算过程为
$$ \begin{array}{*{20}{l}} {H}'=\alpha \times H+f_{{\mathrm{up}}}((1-\alpha)\times L) \end{array} $$ (8) 其中, $ {H}' $表示输出特征图, $ \alpha $表示特征权重. 将两特征图融合以后, 采用上采样方法将特征图恢复到原始图像的大小, 再经过$ 1\times1 $卷积核进行卷积处理, 最后对特征图按照类别数量进行分类, 得到输出结果.
2. 网络度并行检测方法
使用基于语义信息增强的网络度检测算法检测单根丝线的网络度时, 能够有效提取丝线网络结点的位置区域. 然而, 如果图像中丝线样本数量较大, 则难以区分视野内每根丝线的网络度是否符合标准. 因此, 针对多根化纤丝线的网络度的批量计算, 以及独立提取每根丝线的网络度检测结果的需求, 本文提出一种网络度并行检测方法. 此方法在原有检测算法基础上, 增加一条并行检测支路. 其中, 主支路用于检测视野内所有丝线网络结点的位置, 而并行检测支路用于提取每根丝线的位置. 此并行检测支路主要通过提取样本图像中丝线主干的边缘轮廓, 获得丝线的掩膜图像, 最后得到每根丝线主干的位置信息. 当图像同时经过两条支路处理后, 将所得结果进行融合, 从而得到每根丝线上的网络结点图像.
2.1 丝线掩膜提取算法
通过提取丝线掩膜, 定位化纤丝线图像中每一条丝线主干区域, 能够从含有多根丝线的图像中分离出每根丝线所在的位置, 检测过程如图6所示. 具体方法如下:
1) 获取丝线样本图像. 从化纤丝线生产车间获取待检测的丝线样本后, 取一段单位长度的丝线放置于盛有纯净水的水槽中, 使丝线在水槽中自然散开并形成较好的形态, 然后使用工业相机拍摄包含多根并行的待测丝线样本的图像. 为配合检测算法模型中对输入图像的分辨率需求, 设置图像分辨率为2 048 × 512像素.
2) 提取丝线主干. 使用工业相机拍摄图像的分辨率较高, 导致图像处理时间较长. 为减小模型处理时间, 同时确保获取到完整的化纤丝线轮廓区域, 在图像处理前将样本图像缩小至原尺寸的一半. 在实际检测过程中, 容易有小颗粒灰尘落入水槽中, 导致采样图像出现毛刺和杂点. 因此, 本文使用均值滤波算法处理图像. 使用最大类间方差算法对图像进行阈值分割, 通过遍历所有可能的阈值寻找最佳的分割结果. 最后对丝线所在区域与图像背景区域进行处理, 得到包含多根丝线掩膜区域的丝线主干图像.
3) 提取单根丝线掩膜. 由于丝线主干区域与背景区域的对比度较为显著, 因此本文选择Canny边缘检测算子对上一步处理得到的丝线主干图像进行处理, 从而得到视野中每根丝线的边缘拟合图像. 然而, 由于边缘检测方法对图像中的噪音和复杂纹理较为敏感, 若目标边缘不清晰可能导致丝线主干边缘不完整, 进而影响丝线区域的完整性. 因此, 本文借助边缘提取方法, 以图像中每根化纤丝线轮廓区域的中心点为基准, 以丝线轮廓长度值作为掩膜图像的长度, 以轮廓最大宽度为掩膜的图像宽度, 得到轮廓尺寸信息. 然后根据所得尺寸信息, 将丝线拟合轮廓绘制在与输入图像相同尺寸的空白图像上, 得到最终的丝线掩膜图像.
4) 提取单根丝线图像. 由于丝线掩膜图像为二值图像, 即每个像素点由0或1组成. 因此, 将单根丝掩膜与丝线样本图像的每个像素进行乘积处理, 得到单根丝线的图像.
2.2 网络度并行检测过程
在实际生产环境中, 使用工业相机采集到的含多根丝线图像进行网络度检测, 方法如图7所示. 首先使用基于语义信息增强的网络度检测算法检测经预处理后的丝线样本图像的结点区域, 同时使用丝线掩膜提取算法检测单根丝线的位置区域. 其次, 将得到的两个特征图进行对应像素乘积, 得到多张含单根丝线且已标注网络结点位置的图像. 然后根据前述轮廓提取算法得到的数据, 绘制结点特征的外接矩形, 并根据矩形尺寸与标准值比较, 将长度小于2 mm的结点视为可忽略的结点, 筛选之后的结点即为该丝线所有的网络结点. 最后统计丝线上所有网络结点数, 并转换为网络度指标, 计算方法为
图 7 多根丝线网络度并行检测方法Fig. 7 Parallel detection method for interlacing degree of the multiple filament yarns$$ k_{i}=\frac{m}{d} $$ (9) 其中, $ k_{i} $表示单根丝线在单位长度内的网络度指标,单位为(个/m); $i $表示图像中丝线的位置序号($i= $1, 2, $ \cdots $); $m $表示丝线中含有的网络结点数; $d $表示视野范围内待测丝线对应的长度, 单位为m.
3. 实验结果与分析
3.1 网络度检测设备
为实现化纤丝线网络度自动视觉检测, 解决人工检测导致结果误检率高的问题, 本文研制了网络度检测设备. 该设备主要包括CME标准型闭环步进电机、M3ST502M-H型工业相机、3030型铝型材支撑架、亚克力水槽、丝线夹持装置、LED灯源和滑轨, 如图8所示. 使用网络度检测设备检测化纤丝线网络度的流程如图9所示. 在检测开始之前, 需要对设备进行初始化操作, 包括控制参数设置, 以及使用上位机控制两个丝线夹紧装置移动至同一侧. 在丝线网络度检测过程中, 首先将丝线挂载至夹取装置的固定孔中. 待丝线夹紧后, 由上位机控制滑轨转动, 拖动水平滑轨上的夹爪至水槽的另一侧. 然后控制垂直滑轨, 将丝线浸入水中. 最后使用工业相机采集丝线在水中的图像, 建立丝线样本图像数据集.
图 8 多根丝线并行网络度检测设备Fig. 8 Parallel detection equipment for interlacing degree of multiple filament yarns与人工检测方法相比, 使用本文提出的检测设备能够实现精准统计视野范围内每根丝线的网络度, 降低因人工主观因素导致的检测结果偏差. 同时, 人工检测最多只能同时抓取4根丝线进行检测, 而使用本文所提装置进行检测时, 通过合理设计丝线夹持装置的固定孔数量及滑轨长度, 可实现多达10根以上丝线的同时检测, 从而提高丝线的检测效率.
3.2 数据集构建与实验配置
本文采用化纤丝线生产工厂提供的锦纶6型丝线进行样本采集, 并制作数据集. 经图像增强和裁剪等预处理操作后, 得到1 200张512 × 512像素大小的样本图像. 训练集与测试集的比例设置为9 : 1, 然后进行模型训练与验证. 实验中使用的配置信息如表2和表3所示.
表 2 模型训练环境配置Table 2 Configuration of model training environment项目 版本参数 操作系统 Ubuntu 18.04.6 LTS CUDA cuda 11.3 GPU NVIDIA RTX 3 090 训练框架 PyTorch 1.10.2 内存 128 GB 编程语言 Python 3.8 表 3 模型训练超参数配置Table 3 Configuration of model training hyperparameter参数 配置信息 输入图像尺寸 512 × 512 像素 下采样倍数 16 初始学习率 $5 \times 10^{-3}$ 最小学习率 $5 \times 10^{-5}$ 优化器 Adam 权值衰减 $5 \times 10^{-4}$ 批量大小 12 3.3 评价指标
本文采用平均交并比、$ F_{1} $分数和每秒传输帧数三个指标对训练模型进行评价, 具体介绍如下:
1) 平均交并比: 用于计算真实值和预测值两个集合的交集和并集之比, 计算方法为
$$ {MIoU}=\frac{1}{k+1}\sum\limits_{i=0}^{k} \sum\limits_{j=0}^{k} \frac{p_{ii}}{p_{ij}-p_{ii}} $$ (10) 其中, MIoU表示平均交并比值, $ { k} $表示类别数, $ { i} $表示真实值, $ { j} $表示预测值, $ {p_{ij}} $表示将$ { i} $预测为$ { j} $的像素点数量.
2) $ F_{1} $分数: 用于平衡模型的精确率和召回率, 衡量模型在正负样本上的分割结果, 计算方法为
$$ F_{1}=2\times \frac{pre \times rec}{pre + rec} $$ (11) 其中, $ pre $表示模型的精确率, $ rec $表示模型的召回率.
3) 每秒传输帧数: 用于评估模型在每秒内可以处理的图片数量或者处理一张图片所需时间, 进而评估模型的检测速度. 计算方法为
$$ {FPS}=\frac{1}{t} $$ (12) 其中, FPS表示每秒传输帧数, $ { t} $表示模型处理一张图片时消耗的时间.
3.4 损失函数
本文使用图像语义分割损失函数中评估样本相似性的度量函数Dice损失和交叉熵损失函数作为损失函数, 具体介绍如下:
1) Dice损失: 用于评估真实值$ {T} $与预测值$ { P} $的相似性, 计算方法为
$$ {L_{\rm dice}}=1-\frac{2\left | T \cap P \right|}{\left | T \right | + \left | P \right |} $$ (13) 其中, $ {L_{\rm dice}} $表示样本相似值, $ {\left | T \cap P \right|} $表示真实标签T和预测标签P之间的交集元素个数, $ \left | T \right | $和$ \left | P \right | $分别表示真实标签和预测标签中元素的个数.
2) 交叉熵损失函数: 用于衡量每个像素点的预测类别与真实类别之间的差异, 计算方法为
$$ {L_{\rm ce}}=-\frac{1}{N}\sum\limits_{i=1}^{s}\left[t_{i}\lg_{}{p_{i}}+(1-t_{i})\lg(1-p_{i})\right] $$ (14) 其中, $ {L_{\rm ce}} $表示交叉熵损失值, N表示一个批次中含有的特征图数量, $ { s} $表示图像中像素点的数量, $ { t_{i}} $表示第$ { i} $个像素点的真实标签, $ { p_{i}} $表示第$ { i} $个像素点的预测概率.
3.5 性能比较
为验证所提方法在化纤丝线网络度检测方面的有效性, 本文使用语义分割领域常用模型进行对比实验, 包括PSPNet[9], UNet[10], DeepLabV3+[11], BiSeNet[31], CGNet[32], HRNet[33]和SegFormer[34], 并计算其平均交并比、$ F_{1} $分数和每秒传输帧数三个评定指标, 并比较训练模型权重的参数, 比较结果如表4所示. 由表4可以看出, 本文提出的方法在平均交并比、$ F_{1} $分数和每秒传输帧数三个指标均优于其他几种方法, 特别是在模型的每秒传输帧数方面, 本文方法的提升更加显著. 此外, 在模型参数量方面, 所提方法相对于对比实验中的大部分模型具有优势. 其中, BiSeNet和CGNet使用相对简单的轻量级结构, 模型运算时的处理速度较高. DeepLabV3+、PSPNet和SegFormer中使用到运算较为复杂的多尺度特征融合模块与特征金字塔模块, 使得模型权重参数量较大. 本文提出的网络度检测算法使用语义信息增强的方式对特征图进行处理, 同时使用交叉注意力的方法对特征图进行加权赋值, 增强了对特征信息的提取能力. 为直观展示本文所提模型的识别效果, 根据丝线在水中呈现的样态频率, 选择了四种不同类型的化纤丝线的样本图像, 如图10所示. 使用不同的模型检测这四种化纤丝线图像, 实验结果如图11 ~ 14所示.
表 4 不同方法的评价指标比较Table 4 Comparison of evaluation indicators for different methods方法 平均交并比
(%)$F_{1}$分数
(%)每秒传输
帧数(帧/s)参数量
(MB)BiSeNet 78.95 86.76 63.84 48.93 CGNet 79.00 86.79 33.17 2.08 DeepLabV3+ 79.50 87.35 43.11 209.70 HRNet 78.74 86.69 12.43 37.53 PSPNet 73.58 82.46 49.80 178.51 SegFormer 79.04 86.84 40.87 14.34 UNet 79.83 87.63 22.54 94.07 本文方法 81.52 88.12 76.16 7.98 注: 加粗字体表示各列最优结果. 
图 11 丝线散开大于20 mm时的图像检测结果Fig. 11 Detection results of images with filament yarn spreading greater than 20 mm图11所示为丝线散开大于20 mm时的图像检测结果, 此类图像中丝线与背景区域具有明显的对比度, 使得网络结点的位置容易区分. 由实验结果可以看出, 多数模型能够识别出网络结点的位置. 然而, 经过BiSeNet、CGNet、HRNet和SegFormer四个模型检测后得到的结果图中, 矩形框标记位置处出现了像素预测区域不连续的情况. 此外, 在PSPNet模型的检测结果图中, 圆形框标记位置处出现了漏检.
图12所示为丝线散开小于10 mm时的图像检测结果, 其中误检结点区域均已在图中标注. 在此类图像中, 网络结点区域和丝线主干区域的对比度较低, 而且存在许多易混淆的结点, 容易导致漏检. SegFormer模型中使用到了自注意力机制, 从结果可以看出识别效果相对较好. PSPNet模型中使用金字塔池化提取语义信息, 然而在检测结果中, 容易将网络结点附近的背景区域误判为感兴趣区域.
图 12 丝线散开小于10 mm时的图像检测结果Fig. 12 Detection results of images with filament yarn spreading less than 10 mm图13所示为含倾斜丝线的图像检测结果. 实验结果表明, 当图像存在一定倾斜角度时, 能够准确识别明显的网络结点像素区域. 针对图中标注矩形框的位置区域, 在BiSeNet和CGNet的识别结果中, 部分结点像素未完全识别. 当识别图中框选出来的不明显区域时, 采用本文提出的语义信息增强方法能够很好地识别出结点区域图像. 而HRNet, SegFormer和DeepLabV3+ 模型虽能识别出结点信息, 但是识别区域不完整, 且有断续现象.
图14所示为图像中含不明显结点的图像检测结果, 如图中矩形框标注位置所示. 实验结果表明, 视野中存在不同尺寸大小的网络结点时, 容易在模型识别上存在混淆, 但网络结点位置的整体识别情况较好. 针对图中第三根丝线, 使用BiSeNet, CGNet, DeepLabV3+, HRNet和SegFormer模型进行检测时均忽略了小网络结点, 使用PSPNet识别像素不完整, 体现出本文模型在细节信息提取上效果较好.
图 14 含不明显网络结点的图像检测结果Fig. 14 Detection results of images with indistinct interlacing nodes综上所述, 当图像中存在不同网络散开程度的丝线, 以及丝线倾斜与不明显结点等异常情况时, 与常用的语义分割模型相比, 使用本文提出的模型能够较好地获得网络结点区域, 且准确率较高.
3.6 消融实验
3.6.1 模块有效性验证
本实验针对提出的模块进行有效性验证, 设计了8种实验方案, 分别是: 1) 仅使用主干网络; 2) 使用特征图$ x_{4} $和所提语义信息增强模块直接融合; 3) 使用特征图$ x_{2} $和所提多级特征扩张模块直接融合; 4) 使用特征图$ x_{2} $, $ x_{4} $和所提阶段性特征融合模块; 5) 使用特征图$ x_{2} $、所提多级特征扩张和阶段性特征融合模块; 6) 使用特征图$ x_{4} $、所提语义信息增强和阶段性特征融合模块; 7) 使用所提语义信息增强和多级特征扩张模块直接融合; 8) 使用本文所提方法. 实验结果如表5所示.
表 5 模块有效性验证实验结果Table 5 Results of module validity verification experimental方案序号 语义信息增强模块 多级特征扩张模块 阶段性特征融合模块 MIoU (%) FPS (帧/s) 1 $\times$ $\times$ $\times$ 77.18 72.75 2 $\surd$ $\times$ $\times$ 79.91 72.15 3 $\times$ $\surd$ $\times$ 79.85 66.32 4 $\times$ $\times$ $\surd$ 79.33 68.31 5 $\times$ $\surd$ $\surd$ 80.71 55.30 6 $\surd$ $\times$ $\surd$ 81.15 61.48 7 $\surd$ $\surd$ $\times$ 80.25 78.16 8 $\surd$ $\surd$ $\surd$ 81.52 76.16 注: $\surd$指使用此模块, $\times$指不使用此模块. 由表5第5行和第8行可知, 当主干网络的浅层特征缺少语义信息增强模块时, MIoU值下降了0.81%, FPS值下降了20.86帧/s. 当主干网络的深层特征缺少语义信息增强模块时, MIoU值下降了0.37%, FPS值下降了14.68帧/s. 由此可见, 本文提出的全局注意力机制与扩张卷积方法能够增强模型对语义信息的提取能力, 同时改善了模型的运算速度. 由表5的第7行和第8行可知, 使用本文提出的阶段性特征融合模块使得MIoU值增加1.27%, 说明本文提出的阶段性特征提取方式能够提高检测结果的准确性.
3.6.2 主干网络提取效率实验比较
本文采用6种主干网络与本文所提主干网络进行对比实验, 包括FCN, MobileNetV2, Xception, VGGNet, ResNet18和ResNet50. 实验结果如表6所示. 结果表明, 使用本文提出的主干网络在检测结果准确性方面均有提升.
表 6 不同主干网络提取效率比较Table 6 Comparison of extraction efficiency of different backbone networks方案序号 主干网络 MIoU (%) FPS (帧/s) 1 FCN 79.65 33.45 2 MobileNetV2 80.25 43.11 3 Xception 79.61 27.45 4 VGGNet 77.45 30.12 5 ResNet18 77.52 45.21 6 ResNet50 78.01 47.06 7 本文方法 81.52 76.16 3.6.3 语义信息提取方法实验
提取主干网络浅层特征图的语义信息时, 本实验设计4种实验方案与所提方法进行对比, 分别是: 1) 只使用空间注意力机制SA (Spatial attention); 2) 使用注意力机制SE (Squeeze-and-excitation); 3) 使用空间注意力机制SA与通道注意力机制CA (Channel attention)串行的注意力机制CBAM (Convolutional block attention module); 4) 使用通道注意力机制ECA (Efficient channel attention). 实验结果如表7所示. 结果表明, 只使用一种通道注意力机制的方法, 提取特征的效果相对较弱. 使用通道注意力机制ECA进行特征提取后,MIoU值低于原通道注意力机制处理后的结果, 说明在特征提取时只关注局部注意力容易减少网络结点特征的提取效果. 将空间注意力机制SA与通道注意力机制CA串行处理后, 检测结果得到提升, 证明了多注意力机制的有效性.
表 7 不同语义信息提取方法结果比较Table 7 Comparison results of extraction method for different context information方案序号 注意力选择 MIoU (%) 1 SA 80.36 2 SE 80.44 3 CBAM 80.83 4 ECA 79.89 5 本文方法 81.52 3.6.4 扩张卷积模块实验比较
提取主干网络深层特征中的语义信息时, 针对主干网络处理后输出的最后两级特征图$ {x_{3}} $和$ {x_{4}} $, 设计4种扩张卷积提取方案, 分别是: 1) 不使用扩张卷积模块; 2) 只使用特征图$ x_{3} $进行扩张卷积; 3) 只使用特征图$ x_{4} $进行扩张卷积; 4) 使用本文所提方法. 实验结果如表8所示. 结果表明, 使用扩张卷积后, MIoU指标提升0.96%, 使用特征图$ {x_{3}} $进行辅助特征提取能够有效增强扩张卷积的效果.
表 8 不同扩张卷积提取方式结果比较Table 8 Comparison results of different dilated convolution extraction methods方案序号 $x_{3}$ $x_{4}$ MIoU (%) 1 $\times$ $\times$ 80.56 2 $\surd$ $\times$ 81.02 3 $\times$ $\surd$ 81.13 4 $\surd$ $\surd$ 81.52 注: $\surd$指使用此模块, $\times$指不使用此模块. 3.6.5 阶段性特征融合方法实验
对语义信息增强模块与多级语义信息扩张卷积模块使用通道注意力权重融合时, 设计4种方案, 分别是: 1) 只使用全局平均池化方法; 2) 只使用逐点卷积方法; 3) 使用全局平均池化方法和逐点卷积方法串行处理; 4) 使用全局平均池化方法和逐点卷积方法并行处理. 实验结果如表9所示. 实验结果表明, 将全局平均池化与逐点卷积方法进行串行连接时, 导致图像原有特征信息丢失严重. 而使用并行连接时, 能够更有效地保留图像的特征信息.
表 9 阶段性特征融合方法实验比较Table 9 Comparison results of staged feature fusion module方案序号 全局平均池化 逐点卷积 组合方法 MIoU (%) 1 $\surd$ $\times$ 无 80.91 2 $\times$ $\surd$ 无 80.65 3 $\surd$ $\surd$ 串行 78.91 4 $\surd$ $\surd$ 并行 81.52 注: $\surd$指使用此模块, $\times$指不使用此模块. 4. 结束语
针对人工检测化纤丝线网络度过程中存在误检率高的问题, 本文提出一种基于语义信息增强的化纤丝线网络度检测方法, 同时研制网络度检测设备, 实现网络度的自动化视觉精准检测. 其中, 为解决丝线网络结点这一小目标检测问题, 设计语义信息增强模块和多级特征扩张模块, 增强结点信息的提取能力, 并设计阶段性特征融合模块减少特征融合时造成的语义信息丢失. 为实现多根丝线网络度的批量快速检测, 基于掩膜提取与检测算法并行处理机制设计了一种网络度并行检测方法. 利用自主研发的网络度检测装备对所提方法进行了实验验证. 结果表明, 本文所提方法能够有效地增强化纤丝线网络结点的提取能力, 同时提高目标识别的准确性. 下一步工作主要针对化纤工业现场环境进行实验验证, 从检测效率方面进一步对本文提出的检测方法和设备进行迭代更新.
-
图 5 空地机器人无线通信((a) 集中式; (b) 分布式; (c) 移动自组织网络)
Fig. 5 Wireless communications of air-ground robots ((a) Centralized; (b) Distributed; (c) Mobile ad-hoc network)
图 7 生物群智迁移映射空地协同群智系统
Fig. 7 Biological collective intelligence transfer and mapping air-ground collaboration collective intelligence system
表 1 通信方式分析
Table 1 Analysis of communication methods
通信方式 设备 常见任务 常见环境 通信范围 特点 研究代表 有线通信 光缆 搜索救援 范围有限的室外区域 取决于光缆长度 准确性高, 不易出错,
但会影响空地主体机动性[9, 48] 集中式 ZigBee WiFi节点 数据收集 室内, 范围有限的
室外区域0 ~ 100 m 依赖于中心节点, 准确性高,
但通信效率低[49−51] 分布式 WiFi节点 大规模建图 室外 与空地机器人数目成正比 不依赖中心节点, 鲁棒性强,
但通信范围有限[15, 53] 移动自组织网络 IEEE 802.11中继器
LoRA节点搜索救援 隧道等通信设施
缺乏的场景可随节点个数增加
不断扩大不依赖中心节点, 抗干扰能力强,
通信范围广, 灵活部署[9, 30, 42, 54] 
下载: 导出CSV
表 2 空地协同中的语义分割算法应用总结
Table 2 Application summary of semantic segmentation algorithm in air-ground collaboration
下载: 导出CSV
表 5 空地协同等级分类方法总结
Table 5 Summary of classification methods for air-ground collaboration level
等级分类 代表研究 任务 方法 决策拓扑 实验环境 非自主等级协同 [9] 探索地下隧道 基于图的路径规划器, 地图融合算法 分布式 Sim2Real [80] 协作攀爬 未知环境可穿越性地形判断 集中式 Real 弱自主等级协同 [59] 探索地下隧道 BPMN表示法, 有限状态机 集中式 Real [112] 区域搜索 神经进化算法 分布式 Sim [108] 海上平台作业 多角色目标分配 集中式 Sim [110] 野外建图 规约语言, 确定性有限状态机 集中式 Sim [111] 目标跟踪 多智能体强化学习 分布式 Sim 强自主等级协同 [113] 提供通信计算服务 目标层次分解 分布式 Sim [114] 协同作战 PDDL模型, 基于图的任务分解 集中式 Sim [115] 提供通信计算服务 Lyapunov优化法 集中式 Sim
下载: 导出CSV
表 6 空地模拟器总结
Table 6 Summary of air-ground simulator
仿真环境 物理引擎 是否开源 特点 不足 Gazebo 支持ODE、Bullet、Simbody和DART 是 ROS集成使用, 支持多种插件 视觉渲染效果差 MORSE BGE 是 分布计算, 自由度可控 同步性差, 无法精确动力学建模 Pybullet Bullet 是 跨平台, 操作简单 运行效率慢 CoppeliaSim 支持ODE、Bullet和Vortex 是 分布式, 支持ROS接口,
支持多语言编程视觉渲染效果差, 运行效率慢 AirSim UE4 是 跨平台, 视觉逼真 动力学仿真效果差, 物理接口不足 Collaborative robots Sim ODE 否 基于Gazebo强物理交互 视觉渲染效果差, 运行速率慢 Gibson Env 神经网络 是 融入真实数据, 逼真的渲染效果 需要采集大量真实数据
下载: 导出CSV
-
[1] Xue S Y, Na J M, Wang L B, Wang S J, Xu X X. The outlook of green building development in China during the “fourteenth Five-Year Plan” period. International Journal of Environmental Research and Public Health, 2023, 20(6): Article No. 5122 doi: 10.3390/ijerph20065122 [2] Aguilar-Hernandez G A, Rodrigues J F D, Tukker A. Macroeconomic, social and environmental impacts of a circular economy up to 2050: A meta-analysis of prospective studies. Journal of Cleaner Production, 2021, 278: Article No. 123421 doi: 10.1016/j.jclepro.2020.123421 [3] Zhang K T, Chermprayong P, Xiao F, Tzoumanikas D, Dams B, Kay S, et al. Aerial additive manufacturing with multiple autonomous robots. Nature, 2022, 609(7928): 709−717 doi: 10.1038/s41586-022-04988-4 [4] Li S G, Batra R, Brown D, Chang H D, Ranganathan N, Hoberman C, et al. Particle robotics based on statistical mechanics of loosely coupled components. Nature, 2019, 567(7748): 361−365 doi: 10.1038/s41586-019-1022-9 [5] 蒲志强, 易建强, 刘振, 丘腾海, 孙金林, 李非墨. 知识和数据协同驱动的群体智能决策方法研究综述. 自动化学报, 2022, 48(3): 627−643Pu Zhi-Qiang, Yi Jian-Qiang, Liu Zhen, Qiu Teng-Hai, Sun Jin-Lin, Li Fei-Mo. Knowledge-based and data-driven integrating methodologies for collective intelligence decision making: A survey. Acta Automatica Sinica, 2022, 48(3): 627−643 [6] Ye Y X, Liu C H, Dai Z P, Zhao J X, Yuan Y, Wang G R, et al. Exploring both individuality and cooperation for air-ground spatial crowdsourcing by multi-agent deep reinforcement learning. In: Proceedings of the IEEE 39th International Conference on Data Engineering (ICDE). Anaheim, USA: IEEE, 2023. 205−217 [7] Zhao N, Lu W D, Sheng M, Chen Y F, Tang J, Yu F R, et al. UAV-assisted emergency networks in disasters. IEEE Wireless Communications, 2019, 26(1): 45−51 doi: 10.1109/MWC.2018.1800160 [8] Gao Y, Chen J F, Chen X, Wang C Y, Hu J J, Deng F Q, et al. Asymmetric self-play-enabled intelligent heterogeneous multirobot catching system using deep multiagent reinforcement learning. IEEE Transactions on Robotics, 2023, 39(4): 2603−2622 doi: 10.1109/TRO.2023.3257541 [9] Tranzatto M, Miki T, Dharmadhikari M, Bernreiter L, Kulkarni M, Mascarich F, et al. CERBERUS in the DARPA subterranean challenge. Science Robotics, 2022, 7(66): Article No. eabp9742 doi: 10.1126/scirobotics.abp9742 [10] He J H, Zhou Y M, Huang L X, Kong Y, Cheng H. Ground and aerial collaborative mapping in urban environments. IEEE Robotics and Automation Letters, 2021, 6(1): 95−102 doi: 10.1109/LRA.2020.3032054 [11] Girma A, Bahadori N, Sarkar M, Tadewos T G, Behnia M R, Mahmoud M N, et al. IoT-enabled autonomous system collaboration for disaster-area management. IEEE/CAA Journal of Automatica Sinica, 2020, 7(5): 1249−1262 doi: 10.1109/JAS.2020.1003291 [12] Schuster M J, Müller M G, Brunner S G, Lehner H, Lehner P, Sakagami R, et al. The ARCHES space-analogue demonstration mission: Towards heterogeneous teams of autonomous robots for collaborative scientific sampling in planetary exploration. IEEE Robotics and Automation Letters, 2020, 5(4): 5315−5322 doi: 10.1109/LRA.2020.3007468 [13] Zhang J H, Liu R Y, Yin K J, Wang Z Y, Gui M P, Chen S Y. Intelligent collaborative localization among air-ground robots for industrial environment perception. IEEE Transactions on Industrial Electronics, 2019, 66(12): 9673−9681 doi: 10.1109/TIE.2018.2880727 [14] Gawel A, Lin Y K, Koutros T, Siegwart R, Cadena C. Aerial-ground collaborative sensing: Third-person view for teleoperation. In: Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Philadelphia, USA: IEEE, 2018. 1−7 [15] Miller I D, Cladera F, Smith T, Taylor C J, Kumar V. Stronger together: Air-ground robotic collaboration using semantics. IEEE Robotics and Automation Letters, 2022, 7(4): 9643−9650 doi: 10.1109/LRA.2022.3191165 [16] Huo J W, Zenkevich S L, Nazarova A V, Zhai M X. Path planning based on map matching in UAV/UGV collaboration system. International Journal of Intelligent Unmanned Systems, 2021, 9(2): 81−95 doi: 10.1108/IJIUS-03-2019-0020 [17] Yang R H, Zhang M H, Hansen N, Xu H Z, Wang X L. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal Transformers. arXiv: 2107.03996, 2022. [18] Silver D, Huang A, Maddison C J, Guez A, Sifre L, van den Driessche G, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016, 529(7587): 484−489 doi: 10.1038/nature16961 [19] Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 2022, 7(62): Article No. eabk2822 doi: 10.1126/scirobotics.abk2822 [20] Zhou X, Wen X Y, Wang Z P, Gao Y M, Li H J, Wang Q H, et al. Swarm of micro flying robots in the wild. Science Robotics, 2022, 7(66): Article No. eabm5954 doi: 10.1126/scirobotics.abm5954 [21] Hu D F, Gan V J L, Wang T, Ma L. Multi-agent robotic system (MARS) for UAV-UGV path planning and automatic sensory data collection in cluttered environments. Building and Environment, 2022, 221: Article No. 109349 doi: 10.1016/j.buildenv.2022.109349 [22] Ropero F, Muñoz P, R-Moreno M D. TERRA: A path planning algorithm for cooperative UGV–UAV exploration. Engineering Applications of Artificial Intelligence, 2019, 78: 260−272 doi: 10.1016/j.engappai.2018.11.008 [23] Duan H B, Liu S Q. Unmanned air/ground vehicles heterogeneous cooperative techniques: Current status and prospects. Science China Technological Sciences, 2010, 53(5): 1349−1355 doi: 10.1007/s11431-010-0122-4 [24] Chen J, Zhang X, Xin B, Fang H. Coordination between unmanned aerial and ground vehicles: A taxonomy and optimization perspective. IEEE Transactions on Cybernetics, 2016, 46(4): 959−972 doi: 10.1109/TCYB.2015.2418337 [25] Cajo R, Mac T T, Plaza D, Copot C, Keyser R D, Ionescu C. A survey on fractional order control techniques for unmanned aerial and ground vehicles. IEEE Access, 2019, 7: 66864−66878 doi: 10.1109/ACCESS.2019.2918578 [26] Ding Y L, Xin B, Chen J. A review of recent advances in coordination between unmanned aerial and ground vehicles. Unmanned Systems, 2021, 9(2): 97−117 doi: 10.1142/S2301385021500084 [27] Liu C, Zhao J, Sun N Y. A review of collaborative air-ground robots research. Journal of Intelligent & Robotic Systems, 2022, 106(3): Article No. 60 [28] 郭斌, 刘思聪, 於志文. 人机物融合群智计算. 北京: 机械工业出版社, 2022. 23−60Guo Bin, Liu Si-Cong, Yu Zhi-Wen. Crowd Intelligence With the Deep Fusion of Human, Machine, and Things. Beijing: Machinery Industry Press, 2022. 23−60 [29] Saboia M, Clark L, Thangavelu V, Edlund J A, Otsu K, Correa G J, et al. ACHORD: Communication-aware multi-robot coordination with intermittent connectivity. IEEE Robotics and Automation Letters, 2022, 7(4): 10184−10191 doi: 10.1109/LRA.2022.3193240 [30] Kulkarni M, Dharmadhikari M, Tranzatto M, Zimmermann S, Reijgwart V, Petris P D, et al. Autonomous teamed exploration of subterranean environments using legged and aerial robots. In: Proceedings of the International Conference on Robotics and Automation (ICRA). Philadelphia, USA: IEEE, 2022. 3306−3313 [31] Parker L E. Distributed intelligence: Overview of the field and its application in multi-robot systems. Journal of Physical Agents, 2008, 2 (1): 5−14 [32] Luckcuck M, Farrell M, Dennis L A, Dixon C, Fisher M. Formal specification and verification of autonomous robotic systems: A survey. ACM Computing Surveys (CSUR), 2020, 52(5): Article No. 100 [33] Lin P, Abney K, Bekey G. Robot ethics: Mapping the issues for a mechanized world. Artificial Intelligence, 2011, 175(5−6): 942−949 doi: 10.1016/j.artint.2010.11.026 [34] Russel S J, Norvig P. Artificial Intelligence: A Modern Approach. Upper Saddle River: Prentice Hall, 2010. 69−83 [35] Plocher G H. The German Air Force Versus Russia, 1942. New Zealand: Eschenburg Press, 2017. 3−5 [36] Cheng N, Xu W C, Shi W S, Zhou Y, Lu N, Zhou H B, et al. Air-ground integrated mobile edge networks: Architecture, challenges, and opportunities. IEEE Communications Magazine, 2018, 56(8): 26−32 doi: 10.1109/MCOM.2018.1701092 [37] Qin Z, He S S, Wang H, Qu Y B, Dai H P, Xiong F, et al. Air-ground collaborative mobile edge computing: Architecture, challenges, and opportunities. China Communications, 2024, 21(5): 1−16 [38] Michael N, Shen S J, Mohta K, Mulgaonkar Y, Kumar V, Nagatani K, et al. Collaborative mapping of an earthquake damaged building via ground and aerial robots. Journal of Field Robotics, 2012, 29(5): 832−841 doi: 10.1002/rob.21436 [39] Christie G, Shoemaker A, Kochersberger K, Tokekar P, Mclean L, Leonessa A. Radiation search operations using scene understanding with autonomous UAV and UGV. Journal of Field Robotics, 2017, 34(8): 1450−1468 doi: 10.1002/rob.21723 [40] Krizmancic M, Arbanas B, Petrovic T, Petric F, Bogdan S. Cooperative aerial-ground multi-robot system for automated construction tasks. IEEE Robotics and Automation Letters, 2020, 5(2): 798−805 doi: 10.1109/LRA.2020.2965855 [41] Sycara K P. Multiagent systems. AI Magazine, 1998, 19(2): 79−92 [42] Ginting M F, Otsu K, Edlund J A, Gao J, Agha-Mohammadi A A. CHORD: Distributed data-sharing via hybrid ROS 1 and 2 for multi-robot exploration of large-scale complex environments. IEEE Robotics and Automation Letters, 2021, 6(3): 5064−5071 doi: 10.1109/LRA.2021.3061393 [43] Rizk Y, Awad M, Tunstel E W. Cooperative heterogeneous multi-robot systems: A survey. ACM Computing Surveys (CSUR), 2020, 52(2): Article No. 29 [44] Liu Y, Shi J M, Liu Z, Huang J C, Zhou T R. Two-layer routing for high-voltage powerline inspection by cooperated ground vehicle and drone. Energies, 2019, 12(7): Article No. 1385 doi: 10.3390/en12071385 [45] Kiener J, Stryk O V. Towards cooperation of heterogeneous, autonomous robots: A case study of humanoid and wheeled robots. Robotics and Autonomous Systems, 2010, 58(7): 921−929 doi: 10.1016/j.robot.2010.03.013 [46] Sarfati R, Hayes J C, Peleg O. Self-organization in natural swarms of Photinus carolinus synchronous fireflies. Science Advances, 2021, 7(28): Article No. eabg9259 doi: 10.1126/sciadv.abg9259 [47] Yan Z, Jouandeau N, Cherif A A. A survey and analysis of multi-robot coordination. International Journal of Advanced Robotic Systems, 2013, 10(12): Article No. 399 doi: 10.5772/57313 [48] Tranzatto M, Mascarich F, Bernreiter L, Godinho C, Camurri M, Khattak S, et al. CERBERUS: Autonomous legged and aerial robotic exploration in the tunnel and urban circuits of the DARPA subterranean challenge. Field Robotics, 2022, 2(1): 274−324 doi: 10.55417/fr.2022011 [49] Harik E H C, Guinand F, Pelvillain H, Guérin F, Brethé J F. A decentralized interactive architecture for aerial and ground mobile robots cooperation. In: Proceedings of the International Conference on Control, Automation and Robotics. Singapore: IEEE, 2015. 37−43 [50] Chen J, Chen Z Z, Fang M, Li J Q, Ming Z, Wang S L. A GAN-based active terrain mapping for collaborative air-ground robotic system. In: Proceedings of the IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM). Toyonaka, Japan: IEEE, 2019. 622−627 [51] Asadi K, Suresh A K, Ender A, Gotad S, Maniyar S, Anand S, et al. An integrated UGV-UAV system for construction site data collection. Automation in Construction, 2020, 112: Article No. 103068 doi: 10.1016/j.autcon.2019.103068 [52] Bupe P, Haddad R, Rios-Gutierrez F. Relief and emergency communication network based on an autonomous decentralized UAV clustering network. In: Proceedings of the SoutheastCon. Fort Lauderdale, USA: IEEE, 2015. 1−8 [53] Yue Y F, Zhao C Y, Wu Z Y, Yang C L, Wang Y Z, Wang D W. Collaborative semantic understanding and mapping framework for autonomous systems. IEEE/ASME Transactions on Mechatronics, 2021, 26(2): 978−989 doi: 10.1109/TMECH.2020.3015054 [54] Huang Y W, Lu C L, Chen K L, Ser P S, Huang J T, Shen Y C, et al. Duckiefloat: A collision-tolerant resource-constrained blimp for long-term autonomy in subterranean environments. arXiv: 1910.14275, 2019. [55] Nguyen H T, Garratt M, Bui L T, Abbass H. Supervised deep actor network for imitation learning in a ground-air UAV-UGVs coordination task. In: Proceedings of the IEEE Symposium Series on Computational Intelligence. Honolulu, USA: IEEE, 2017. 1−8 [56] Nouri N, Abouei J, Sepasian A R, Jaseemuddin M, Anpalagan A, Plataniotis K N. Three-dimensional multi-UAV placement and resource allocation for energy-efficient IoT communication. IEEE Internet of Things Journal, 2022, 9(3): 2134−2152 doi: 10.1109/JIOT.2021.3091166 [57] Mox D, Calvo-Fullana M, Gerasimenko M, Fink J, Kumar V, Ribeiro A. Mobile wireless network infrastructure on demand. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Paris, France: IEEE, 2020. 7726−7732 [58] Clark L, Edlund J A, Net M S, Vaquero T S, Agha-mohammadi A A. PropEM-L: Radio propagation environment modeling and learning for communication-aware multi-robot exploration. arXiv: 2205.01267, 2022. [59] Otsu K, Tepsuporn S, Thakker R, Vaquero T S, Edlund J A, Walsh W, et al. Supervised autonomy for communication-degraded subterranean exploration by a robot team. In: Proceedings of the IEEE Aerospace Conference. Big Sky, USA: IEEE, 2020. 1−9 [60] Guo Y L, Wang H Y, Hu Q Y, Liu H, Liu L, Bennamoun M. Deep learning for 3D point clouds: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4338−4364 doi: 10.1109/TPAMI.2020.3005434 [61] Peterson J, Chaudhry H, Abdelatty K, Bird J, Kochersberger K. Online aerial terrain mapping for ground robot navigation. Sensors, 2018, 18(2): Article No. 630 doi: 10.3390/s18020630 [62] Kruijff G J M, Kruijff-Korbayová I, Keshavdas S, Larochelle B, Janíček M, Colas F, et al. Designing, developing, and deploying systems to support human–robot teams in disaster response. Advanced Robotics, 2014, 28(23): 1547−1570 doi: 10.1080/01691864.2014.985335 [63] Delmerico J, Mueggler E, Nitsch J, Scaramuzza D. Active autonomous aerial exploration for ground robot path planning. IEEE Robotics and Automation Letters, 2017, 2(2): 664−671 doi: 10.1109/LRA.2017.2651163 [64] Chen L C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv: 1706.05587, 2017. [65] Hu W, Tan T N, Wang L, Maybank S. A survey on visual surveillance of object motion and behaviors. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2004, 34(3): 334−352 doi: 10.1109/TSMCC.2004.829274 [66] Mueggler E, Faessler M, Fontana F, Scaramuzza D. Aerial-guided navigation of a ground robot among movable obstacles. In: Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics. Hokkaido, Japan: IEEE, 2014. 1−8 [67] Minaeian S, Liu J, Son Y J. Vision-based target detection and localization via a team of cooperative UAV and UGVs. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2016, 46(7): 1005−1016 doi: 10.1109/TSMC.2015.2491878 [68] Zhao Z Q, Zheng P, Xu S T, Wu X D. Object detection with deep learning: A review. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(11): 3212−3232 doi: 10.1109/TNNLS.2018.2876865 [69] Howard A G, Zhu M L, Chen B, Kalenichenko D, Wang W J, Weyand T, et al. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1704.04861, 2017. [70] Sandler M, Howard A, Zhu M L, Zhmoginov A, Chen L C. MobileNetV2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 4510−4520 [71] Redmon J, Farhadi A. YOLOv3: An incremental improvement. arXiv: 1804.02767, 2018. [72] Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014. 580−587 [73] Girshick R. Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015. 1440−1448 [74] Yang T, Ren Q, Zhang F B, Xie B L, Ren H L, Li J, et al. Hybrid camera array-based UAV auto-landing on moving UGV in GPS-denied environment. Remote Sensing, 2018, 10(11): Article No. 1829 doi: 10.3390/rs10111829 [75] Krátký V, Petráček P, Báča T, Saska M. An autonomous unmanned aerial vehicle system for fast exploration of large complex indoor environments. Journal of Field Robotics, 2021, 38(8): 1036−1058 doi: 10.1002/rob.22021 [76] Hudson N, Talbot F, Cox M, Williams J, Hines T, Pitt A, et al. Heterogeneous ground and air platforms, homogeneous sensing: Team CSIRO Data61's approach to the DARPA subterranean challenge. Field Robotics, 2022, 2(1): 595−636 doi: 10.55417/fr.2022021 [77] Tychsen-Smith L, Petersson L. DeNet: Scalable real-time object detection with directed sparse sampling. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017. 428−436 [78] Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE, 2016. 779−788 [79] Tung F, Mori G. Similarity-preserving knowledge distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, South Korea: IEEE, 2019. 1365−1374 [80] Miki T, Khrapchenkov P, Hori K. UAV/UGV autonomous cooperation: UAV assists UGV to climb a cliff by attaching a tether. In: Proceedings of the International Conference on Robotics and Automation (ICRA). Montreal, Canada: IEEE, 2019. 8041−8047 [81] Fedorenko R, Gabdullin A, Fedorenko A. Global UGV path planning on point cloud maps created by UAV. In: Proceedings of the 3rd IEEE International Conference on Intelligent Transportation Engineering. Singapore: IEEE, 2018. 253−258 [82] Ebadi K, Chang Y, Palieri M, Stephens A, Hatteland A, Heiden E, et al. LAMP: Large-scale autonomous mapping and positioning for exploration of perceptually-degraded subterranean environments. In: Proceedings of the IEEE International Conference on Robotics and Automation. Paris, France: IEEE, 2020. 80−86 [83] Yu C, Liu Z X, Liu X J, Xie F G, Yang Y, Wei Q, et al. DS-SLAM: A semantic visual SLAM towards dynamic environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Madrid, Spain: IEEE, 2018. 1168−1174 [84] Wu C C, Agarwal S, Curless B, Seitz S M. Multicore bundle adjustment. In: Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE, 2011. 3057−3064 [85] Furukawa Y, Ponce J. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8): 1362−1376 doi: 10.1109/TPAMI.2009.161 [86] Zhang L L, Gao F, Deng F, Xi L L, Chen J. Distributed estimation of a layered architecture for collaborative air-ground target geolocation in outdoor environments. IEEE Transactions on Industrial Electronics, 2023, 70(3): 2822−2832 doi: 10.1109/TIE.2022.3165245 [87] Forster C, Pizzoli M, Scaramuzza D. Air-ground localization and map augmentation using monocular dense reconstruction. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo, Japan: IEEE, 2013. 3971−3978 [88] Wu J A, Yang S W, Zhang Y J. Segment-based cross-domain localization between aerial and ground robots. In: Proceedings of the Chinese Automation Congress (CAC). Shanghai, China: IEEE, 2020. 5374−5381 [89] Majdik A L, Verda D, Albers-Schoenberg Y, Scaramuzza D. Air-ground matching: Appearance-based GPS-denied urban localization of micro aerial vehicles. Journal of Field Robotics, 2015, 32(7): 1015−1039 doi: 10.1002/rob.21585 [90] Petris P D, Khattak S, Dharmadhikari M, Waibel G, Nguyen H, Montenegr M, et al. Marsupial walking-and-flying robotic deployment for collaborative exploration of unknown environments. In: Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Sevilla, Spain: IEEE, 2022. 188−194 [91] Qin H L, Meng Z H, Meng W, Chen X D, Sun H, Lin F, et al. Autonomous exploration and mapping system using heterogeneous UAVs and UGVs in GPS-denied environments. IEEE Transactions on Vehicular Technology, 2019, 68(2): 1339−1350 doi: 10.1109/TVT.2018.2890416 [92] Queralta J P, Li Q Q, Schiano F, Westerlund T. VIO-UWB-based collaborative localization and dense scene reconstruction within heterogeneous multi-robot systems. In: Proceedings of the International Conference on Advanced Robotics and Mechatronics (ICARM). Guilin, China: IEEE, 2022. 87−94 [93] Patle B K, Babu L G, Pandey A, Parhi D R K, Jagadeesh A. A review: On path planning strategies for navigation of mobile robot. Defence Technology, 2019, 15(4): 582−606 doi: 10.1016/j.dt.2019.04.011 [94] Shen C S, Zhang Y Z, Li Z M, Gao F, Shen S J. Collaborative air-ground target searching in complex environments. In: Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Shanghai, China: IEEE, 2017. 230−237 [95] Stocco D M. A StAR search: Implications in controlling steroidogenesis. Biology of Reproduction, 1997, 56(2): 328−336 doi: 10.1095/biolreprod56.2.328 [96] Likhachev M, Ferguson D, Gordon G, Stentz A, Thrun S. Anytime dynamic A*: An anytime, replanning algorithm. In: Proceedings of the 15th International Conference on International Conference on Automated Planning and Scheduling. Monterey, USA: AAAI, 2005. 262−271 [97] Deng D, Jing W, Fu Y H, Huang Z Y, Liu J H, Shimada K. Constrained heterogeneous vehicle path planning for large-area coverage. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Macao, China: IEEE, 2019. 4113−4120 [98] Dong L, He Z C, Song C W, Sun C Y. A review of mobile robot motion planning methods: From classical motion planning workflows to reinforcement learning-based architectures. Journal of Systems Engineering and Electronics, 2023, 34 (2): 439−459 [99] Yang Y, Li J T, Peng L L. Multi-robot path planning based on a deep reinforcement learning DQN algorithm. CAAI Transactions on Intelligence Technology, 2020, 5(3): 177−183 doi: 10.1049/trit.2020.0024 [100] Hussein A, Gaber M M, Elyan E, Jayne C. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 2018, 50(2): Article No. 21 [101] Xiao X S, Liu B, Stone P. Agile robot navigation through hallucinated learning and sober deployment. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Xi'an, China: IEEE, 2021. 7316−7322 [102] Wang Z H, Xiao X S, Nettekoven A J, Umasankar K, Singh A, Bommaka S, et al. From agile ground to aerial navigation: Learning from learned hallucination. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Prague, Czech Republic: IEEE, 2021. 148−153 [103] Duong T, Atanasov N. Hamiltonian-based neural ODE networks on the SE(3) manifold for dynamics learning and control. arXiv: 2106.12782, 2021. [104] Rodriguez-Ramos A, Sampedro C, Bavle H, Puente P D L, Campoy P. A deep reinforcement learning strategy for UAV autonomous landing on a moving platform. Journal of Intelligent & Robotic Systems, 2019, 93: 351−366 [105] Long P X, Fan T X, Liao X Y, Liu W X, Zhang H, Pan J. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Brisbane, Australia: IEEE, 2018. 6252−6259 [106] Skaltsis G M, Shin H S, Tsourdos A. A survey of task allocation techniques in MAS. In: Proceedings of the International Conference on Unmanned Aircraft Systems (ICUAS). Athens, Greece: IEEE, 2021. 488−497 [107] Kim M H, Baik H, Lee S. Resource welfare based task allocation for UAV team with resource constraints. Journal of Intelligent & Robotic Systems, 2015, 77: 611−627 [108] Carreno Y, Pairet È, Petillot Y, Petrick R P A. Task allocation strategy for heterogeneous robot teams in offshore missions. In: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. Auckland, New Zealand: International Foundation for Autonomous Agents and Multiagent Systems, 2020. 222−230 [109] Phan C, Liu H H T. A cooperative UAV/UGV platform for wildfire detection and fighting. In: Proceedings of the Asia Simulation Conference——7th International Conference on System Simulation and Scientific Computing. Beijing, China: IEEE, 2008. 494−498 [110] Ju C, Son H I. Modeling and control of heterogeneous field robots under partial observation. Information Sciences, 2021, 580: 419−435 doi: 10.1016/j.ins.2021.08.071 [111] Yu Q F, Shen Z X, Pang Y J, Liu R. Proficiency constrained multi-agent reinforcement learning for environment-adaptive multi UAV-UGV teaming. In: Proceedings of the 17th IEEE International Conference on Automation Science and Engineering (CASE). Lyon, France: IEEE, 2021. 2114−2118 [112] Behjat A, Manjunatha H, Kumar P K, Jani A, Collins L, Ghassemi P, et al. Learning robot swarm tactics over complex adversarial environments. In: Proceedings of the International Symposium on Multi-robot and Multi-agent Systems (MRS). Cambridge, UK: IEEE, 2021. 83−91 [113] Deng M, Liu B J, Li S M, Du R H, Wu G H, Li H F, et al. A two-phase coordinated planning approach for heterogeneous earth-observation resources to monitor area targets. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2021, 51(10): 6388−6403 doi: 10.1109/TSMC.2019.2962205 [114] Ding Y H, Wang N, Li X. An air-ground cooperative task decomposition method based on graph processing and PDDL. In: Proceedings of the International Conference on Autonomous Unmanned Systems (ICAUS 2021). Changsha, China: Springer, 2021. 239−247 [115] Liao H J, Zhou Z Y, Zhao X W, Wang Y. Learning-based queue-aware task offloading and resource allocation for space–air–ground-integrated power IoT. IEEE Internet of Things Journal, 2021, 8(7): 5250−5263 doi: 10.1109/JIOT.2021.3058236 [116] Han X N, Liu H P, Sun F C, Zhang X Y. Active object detection with multistep action prediction using deep Q-network. IEEE Transactions on Industrial Informatics, 2019, 15(6): 3723−3731 doi: 10.1109/TII.2019.2890849 [117] Shi Y F, Wang N, Zheng J M, Zhang Y, Yi S, Luo W H, et al. Adaptive informative sampling with environment partitioning for heterogeneous multi-robot systems. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas, USA: IEEE, 2020. 11718−11723 [118] Lesser V R. Evolution of the GPGP/TAEMS domain-independent coordination framework. In: Proceedings of the 1st International Joint Conference on Autonomous Agents and MultiAgent Systems: Part 1. Bologna, Italy: Springer, 2002. 1−2 [119] Decker K S, Lesser V R. Designing a family of coordination algorithms. In: Proceedings of the 1st International Conference on MultiAgent Systems. San Francisco, USA: ICMAS, 1995. 73−80 [120] Haslum P, Lipovetzky N, Magazzeni D, Muise C. An Introduction to the Planning Domain Definition Language. Cham: Springer, 2019. 9−18 [121] Pulvermüller F, Fadiga L. Active perception: Sensorimotor circuits as a cortical basis for language. Nature Reviews Neuroscience, 2010, 11(5): 351−360 doi: 10.1038/nrn2811 [122] Chang D, Johnson-Roberson M, Sun J. An active perception framework for autonomous underwater vehicle navigation under sensor constraints. IEEE Transactions on Control Systems Technology, 2022, 30(6): 2301−2316 doi: 10.1109/TCST.2021.3139307 [123] Liu X, Zhang F J, Hou Z Y, Mian L, Wang Z Y, Zhang J, et al. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(1): 857−876 [124] Li Y P, Zhang D B, Yin F, Zhang Y. Cleaning robot operation decision based on causal reasoning and attribute learning. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas, USA: IEEE, 2020. 6878−6885 [125] Zion M Y B, Fersula J, Bredeche N, Dauchot O. Morphological computation and decentralized learning in a swarm of sterically interacting robots. Science Robotics, 2023, 8(75): Article No. eabo6140 doi: 10.1126/scirobotics.abo6140 [126] Wu L, Huang X D, Cui J G, Liu C, Xiao W S. Modified adaptive ant colony optimization algorithm and its application for solving path planning of mobile robot. Expert Systems With Applications, 2023, 215: Article No. 119410 doi: 10.1016/j.eswa.2022.119410 [127] Hart D, Sussman R W. The influence of predation on primate and early human evolution: Impetus for cooperation. Origins of Altruism and Cooperation, 2011 : 19−40 [128] Egerton F N. Changing concepts of the balance of nature. The Quarterly Review of Biology, 1973, 48(2): 322−350 doi: 10.1086/407594 [129] Stahler D, Heinrich B, Smith D. Common ravens, Corvus corax, preferentially associate with grey wolves, Canis lupus, as a foraging strategy in winter. Animal Behaviour, 2002, 64(2): 283−290 doi: 10.1006/anbe.2002.3047 [130] Nandakumar S C, Harper S, Mitchell D, Blanche J, Lim T, Yamamoto I, et al. Bio-inspired multi-robot autonomy. arXiv: 2203.07718, 2022. [131] Ni J J, Wang X T, Tang M, Cao W D, Shi P F, Yang S X. An improved real-time path planning method based on dragonfly algorithm for heterogeneous multi-robot system. IEEE Access, 2020, 8: 140558−140568 doi: 10.1109/ACCESS.2020.3012886 [132] Nguyen D H. A nature-inspired distributed robust control design for ground-aerial vehicle cooperation. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(4): 4454−4463 doi: 10.1109/TITS.2022.3229336 [133] Zhang J, Yu Z T, Mao S W, Periaswamy S C G, Patton J, Xia X. IADRL: Imitation augmented deep reinforcement learning enabled UGV-UAV coalition for tasking in complex environments. IEEE Access, 2020, 8: 102335−102347 doi: 10.1109/ACCESS.2020.2997304 [134] Manjunatha H, Distefano J P, Jani A, Ghassemi P, Chowdhury S, Dantu K, et al. Using physiological measurements to analyze the tactical decisions in human swarm teams. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). Toronto, Canda: IEEE, 2020. 256−261 [135] Kudithipudi D, Aguilar-Simon M, Babb J, Bazhenov M, Blackiston D, Bongard J, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 2022, 4(3): 196−210 doi: 10.1038/s42256-022-00452-0 [136] Mathews N, Christensen A L, Stranieri A, Scheidler A, Dorigo M. Supervised morphogenesis: Exploiting morphological flexibility of self-assembling multirobot systems through cooperation with aerial robots. Robotics and Autonomous Systems, 2019, 112: 154−167 doi: 10.1016/j.robot.2018.11.007 [137] Sorokin M, Tan J, Liu C K, Ha S. Learning to navigate sidewalks in outdoor environments. IEEE Robotics and Automation Letters, 2022, 7(2): 3906−3913 doi: 10.1109/LRA.2022.3145947 [138] Dong Z Y, Wei P X, Lin L. Adversarially-aware robust object detector. In: Proceedings of the 17th European Conference on Computer Vision. Tel Aviv, Israel: Springer, 2022. 297−313 [139] Luo F M, Jiang S Y, Yu Y, Zhang Z Z, Zhang Y F. Adapt to environment sudden changes by learning a context sensitive policy. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. Virtual Event: AAAI, 2022. 7637−7646 [140] Liu S C, Lin Y Y, Zhou Z M, Nan K M, Liu H, Du J Z. On-demand deep model compression for mobile devices: A usage-driven model selection framework. In: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. Munich, Germany: ACM, 2018. 389−400 [141] Liu S C, Guo B, Ma K, Yu Z W, Du J Z. AdaSpring: Context-adaptive and runtime-evolutionary deep model compression for mobile applications. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021, 5(1): Article No. 24 [142] Nygaard T F, Martin C P, Torresen J, Glette K, Howard D. Real-world embodied AI through a morphologically adaptive quadruped robot. Nature Machine Intelligence, 2021, 3(5): 410−419 doi: 10.1038/s42256-021-00320-3 [143] Li L, Wang S Q, Zhang Y Y, Song S Y, Wang C Q, Tan S C, et al. Aerial-aquatic robots capable of crossing the air-water boundary and hitchhiking on surfaces. Science Robotics, 2022, 7(66): Article No. eabm669 [144] Ye Z H, Wang K, Chen Y N, Jiang X H, Song G H. Multi-UAV navigation for partially observable communication coverage by graph reinforcement learning. IEEE Transactions on Mobile Computing, 2023, 22(7): 4056−4069 doi: 10.1109/TMC.2022.3146881 [145] Song X Y, Yang Y X, Choromanski K, Caluwaerts K, Gao W B, Finn C, et al. Rapidly adaptable legged robots via evolutionary meta-learning. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Las Vegas, USA: IEEE, 2020. 3769−3776 [146] Parisi G I, Kemker R, Part J L, Kanan C, Wermter S. Continual lifelong learning with neural networks: A review. Neural Networks, 2019, 113: 54−71 doi: 10.1016/j.neunet.2019.01.012 [147] Park J S, O'Brien J, Cai C J, Morris M R, Liang P, Bernstein M S. Generative agents: Interactive simulacra of human behavior. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. San Francisco, USA: ACM, 2023. Article No. 2 [148] Johnson M T J, Stinchcombe J R. An emerging synthesis between community ecology and evolutionary biology. Trends in Ecology & Evolution, 2007, 22(5): 250−257 [149] Tokekar P, Vander Hook J, Mulla D, Isler V. Sensor planning for a symbiotic UAV and UGV system for precision agriculture. IEEE Transactions on Robotics, 2016, 32(6): 1498−1511 doi: 10.1109/TRO.2016.2603528 [150] Niroui F, Zhang K C, Kashino Z, Nejat G. Deep reinforcement learning robot for search and rescue applications: Exploration in unknown cluttered environments. IEEE Robotics and Automation Letters, 2019, 4(2): 610−617 doi: 10.1109/LRA.2019.2891991 [151] Kruijff-Korbayová I, Colas F, Gianni M, Pirri F, Greeff J D, Hindriks K, et al. TRADR project: Long-term human-robot teaming for robot assisted disaster response. KI——Künstliche Intelligenz, 2015, 29(2): 193−201 [152] Yamauchi B. A frontier-based approach for autonomous exploration. In: Proceedings of the IEEE International Symposium on Computational Intelligence in Robotics and Automation. Monterey, USA: IEEE, 1997. 146−151 [153] Agha A, Otsu K, Morrell B, Fan D D, Thakker R, Santamaria-Navarro A, et al. NeBula: Quest for robotic autonomy in challenging environments; TEAM CoSTAR at the DARPA subterranean challenge. arXiv: 2103.11470, 2021. [154] Yu H L, Meier K, Argyle M, Beard R W. Cooperative path planning for target tracking in urban environments using unmanned air and ground vehicles. IEEE/ASME Transactions on Mechatronics, 2015, 20(2): 541−552 doi: 10.1109/TMECH.2014.2301459 [155] Chen J Y C. UAV-guided navigation for ground robot tele-operation in a military reconnaissance environment. Ergonomics, 2010, 53(8): 940−950 doi: 10.1080/00140139.2010.500404 [156] Cantelli L, Mangiameli M, Melita C D, Muscato G. UAV/UGV cooperation for surveying operations in humanitarian demining. In: Proceedings of the IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR). Linköping, Sweden: IEEE, 2013. 1−6 [157] Potena C, Khanna R, Nieto J, Siegwart R, Nardi D, Pretto A. AgriColMap: Aerial-ground collaborative 3D mapping for precision farming. IEEE Robotics and Automation Letters, 2019, 4(2): 1085−1092 doi: 10.1109/LRA.2019.2894468 [158] Pan Y, Li S N, Chen Q W, Zhang N, Chent T, Li Z G, et al. Efficient schedule of energy-constrained UAV using crowdsourced buses in last-mile parcel delivery. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021, 5(1): Article No. 28 [159] Kirschstein T. Comparison of energy demands of drone-based and ground-based parcel delivery services. Transportation Research Part D: Transport and Environment, 2020, 78: Article No. 102209 doi: 10.1016/j.trd.2019.102209 [160] Peng K, Du J X, Lu F, Sun Q G, Dong Y, Zhou P, et al. A hybrid genetic algorithm on routing and scheduling for vehicle-assisted multi-drone parcel delivery. IEEE Access, 2019, 7: 49191−49200 doi: 10.1109/ACCESS.2019.2910134 [161] Chen Y, Chen M Q, Chen Z H, Cheng L, Yang Y H, Li H. Delivery path planning of heterogeneous robot system under road network constraints. Computers & Electrical Engineering, 2021, 92: Article No. 107197 [162] Couceiro M S, Portugal D, Ferreira J F, Rocha R P. SEMFIRE: Towards a new generation of forestry maintenance multi-robot systems. In: Proceedings of the IEEE/SICE International Symposium on System Integration. Paris, France: IEEE, 2019. 270−276 [163] Mathew N, Smith S L, Waslander S L. Multirobot rendezvous planning for recharging in persistent tasks. IEEE Transactions on Robotics, 2015, 31(1): 128−142 doi: 10.1109/TRO.2014.2380593 [164] Dorigo M, Theraulaz G, Trianni V. Reflections on the future of swarm robotics. Science Robotics, 2020, 5(49): Article No. eabe4385 doi: 10.1126/scirobotics.abe4385 [165] Reynolds C W. Flocks, herds and schools: A distributed behavioral model. In: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. Anaheim, USA: ACM, 1987. 25−34 [166] Mirzaeinia A, Heppner F, Hassanalian M. An analytical study on leader and follower switching in V-shaped Canada Goose flocks for energy management purposes. Swarm Intelligence, 2020, 14(2): 117−141 doi: 10.1007/s11721-020-00179-x [167] Fierro R, Das A, Spletzer J, Esposito J, Kumar V, Ostrowski J P, et al. A framework and architecture for multi-robot coordination. The International Journal of Robotics Research, 2002, 21(10−11): 977−995 doi: 10.1177/0278364902021010981 [168] Wang W, Wang L, Wu J F, Tao X P, Wu H J. Oracle-guided deep reinforcement learning for large-scale multi-UAVs flocking and navigation. IEEE Transactions on Vehicular Technology, 2022, 71(10): 10280−10292 doi: 10.1109/TVT.2022.3184043 [169] Kannan S S, Venkatesh V L N, Min B C. SMART-LLM: Smart multi-agent robot task planning using large language models. arXiv: 2309.10062, 2024. [170] Wang F Y, Yang J, Wang X X, Li J J, Han Q L. Chat with ChatGPT on Industry 5.0: Learning and decision-making for intelligent industries. IEEE/CAA Journal of Automatica Sinica, 2023, 10(4): 831−834 doi: 10.1109/JAS.2023.123552 [171] Vemprala S H, Bonatti R, Bucker A, Kapoor A. ChatGPT for robotics: Design principles and model abilities. IEEE Access, 2024, 12: 55682−55696 doi: 10.1109/ACCESS.2024.3387941 [172] Yuan L Y, Gao X F, Zheng Z L, Edmonds M, Wu Y N, Rossano F, et al. In situ bidirectional human-robot value alignment. Science Robotics, 2022, 7(68): Article No. eabm4183 doi: 10.1126/scirobotics.abm4183 [173] Liu S J, Chang P X, Huang Z, Chakraborty N, Hong K W, Liang W H, et al. Intention aware robot crowd navigation with attention-based interaction graph. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). London, United Kingdom: IEEE, 2023. 12015−12021 [174] Schranz M, Caro G A D, Schmickl T, Elmenreich W, Arvin F, Şekercioğlu A, et al. Swarm intelligence and cyber-physical systems: Concepts, challenges and future trends. Swarm and Evolutionary Computation, 2021, 60: Article No. 100762 doi: 10.1016/j.swevo.2020.100762 [175] Koenig N, Howard A. Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Sendai, Japan: IEEE, 2004. 2149−2154 [176] Petrillo M D, Beard J, Gu Y, Gross J N. Search planning of a UAV/UGV team with localization uncertainty in a subterranean environment. IEEE Aerospace and Electronic Systems Magazine, 2021, 36(6): 6−16 doi: 10.1109/MAES.2021.3065041 [177] Greff K, Belletti F, Beyer L, Doersch C, Du Y L, Duckworth D, et al. Kubric: A scalable dataset generator. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE, 2022. 3739−3751 [178] Shah S, Dey D, Lovett C, Kapoor A. AirSim: High-fidelity visual and physical simulation for autonomous vehicles. In: Proceedings of the 11th International Conference on Field and Service Robotics. Zurich, Switzerland: Springer, 2018. 621−635 [179] Liu H Y, Guo P, Jin X Y, Zhang H J, Deng H C, Xu K, et al. Collaborative robots Sim: A simulation environment of air-ground robots with strong physical interactivity. In: Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO). Sanya, China: IEEE, 2021. 1841−1847 [180] Echeverria G, Lassabe N, Degroote A, Lemaignan S. Modular open robots simulation engine: MORSE. In: Proceedings of the IEEE International Conference on Robotics and Automation. Shanghai, China: IEEE, 2011. 46−51 [181] Pitonakova L, Giuliani M, Pipe A, Winfield A. Feature and performance comparison of the V-REP, Gazebo and ARGoS robot simulators. In: Proceedings of the 19th Annual Conference on Towards Autonomous Robotic Systems. Bristol, UK: IEEE, 2018. 357−368 [182] Xia F, Zamir A R, He Z Y, Sax A, Malik J, Savarese S. Gibson Env: Real-world perception for embodied agents. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 9068−9079 [183] Yang Z Q, Zhang Q, Jiang Z L, Chen Z Q. Flocking of multi-agents with time delay. International Journal of Systems Science, 2012, 43(11): 2125−2134 doi: 10.1080/00207721.2011.564675 [184] Mekdad Y, Aris A, Babun L, Fergougui A E, Conti M, Lazzeretti R, et al. A survey on security and privacy issues of UAVs. Computer Networks, 2023, 224: Article No. 109626 doi: 10.1016/j.comnet.2023.109626 [185] 何道敬, 杜晓, 乔银荣, 朱耀康, 樊强, 罗旺. 无人机信息安全研究综述. 计算机学报, 2019, 42(5): 1076−1094 doi: 10.11897/SP.J.1016.2019.01076He Dao-Jing, Du Xiao, Qiao Yin-Rong, Zhu Yao-Kang, Fan Qiang, Luo Wang. A survey on cyber security of unmanned aerial vehicles. Chinese Journal of Computers, 2019, 42(5): 1076−1094 doi: 10.11897/SP.J.1016.2019.01076 [186] Sun G, Li J H, Wang A M, Wu Q Q, Sun Z M, Liu Y H. Secure and energy-efficient UAV relay communications exploiting collaborative beamforming. IEEE Transactions on Communications, 2022, 70(8): 5401−5416 doi: 10.1109/TCOMM.2022.3184160 [187] Zhao W S, Queralta J P, Westerlund T. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In: Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI). Canberra, Australia: IEEE, 2020. 737−744 [188] Tan J, Zhang T N, Coumans E, Iscen A, Bai Y F, Hafner D, et al. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv: 1804.10332, 2018. [189] Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P. Domain randomization for transferring deep neural networks from simulation to the real world. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Vancouver, Canada: IEEE, 2017. 23−30 [190] Bousmalis K, Silberman N, Dohan D, Erhan D, Krishnan D. Unsupervised pixel-level domain adaptation with generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017. 95−104 [191] Chebotar Y, Handa A, Makoviychuk V, Macklin M, Issac J, Ratliff N, et al. Closing the Sim-to-real loop: Adapting simulation randomization with real world experience. In: Proceedings of the International Conference on Robotics and Automation (ICRA). Montreal, Canada: IEEE, 2019. 8973−8979 [192] Bogue R. Cloud robotics: A review of technologies, developments and applications. Industrial Robot: An International Journal, 2017, 44(1): 1−5 doi: 10.1108/IR-10-2016-0265 [193] Zhang P Y, Zhong Y X, Li X Q. SlimYOLOv3: Narrower, faster and better for real-time UAV applications. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). Seoul, South Korea: IEEE, 2019. 37−45 [194] 刘亚波. 异构多机器人系统协同技术研究 [博士学位论文], 浙江大学, 中国, 2011.Liu Ya-Bo. The Coordination Approach for Heterogeneous Multi-robot Teams [Ph.D. dissertation], Zhejiang University, China, 2011. -
下载:
计量
- 文章访问数: 4271
- HTML全文浏览量: 403
- PDF下载量: 592
- 被引次数: 0
