2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于单应性扩散约束的二步网格优化视差图像对齐

陈殷齐 郑慧诚 严志伟 林峻宇

陈殷齐, 郑慧诚, 严志伟, 林峻宇. 基于单应性扩散约束的二步网格优化视差图像对齐. 自动化学报, 2022, 48(x): 1−14 doi: 10.16383/j.aas.c210966
引用本文: 陈殷齐, 郑慧诚, 严志伟, 林峻宇. 基于单应性扩散约束的二步网格优化视差图像对齐. 自动化学报, 2022, 48(x): 1−14 doi: 10.16383/j.aas.c210966
Chen Yin-Qi, Zheng Hui-Cheng, Yan Zhi-Wei, Lin Jun-Yu. Parallax image alignment with two-stage mesh optimization based on homography diffusion constraints. Acta Automatica Sinica, 2022, 48(x): 1−14 doi: 10.16383/j.aas.c210966
Citation: Chen Yin-Qi, Zheng Hui-Cheng, Yan Zhi-Wei, Lin Jun-Yu. Parallax image alignment with two-stage mesh optimization based on homography diffusion constraints. Acta Automatica Sinica, 2022, 48(x): 1−14 doi: 10.16383/j.aas.c210966

基于单应性扩散约束的二步网格优化视差图像对齐

doi: 10.16383/j.aas.c210966
基金项目: 国家自然科学基金 (61976231), 广东省基础与应用基础研究基金 (2019A1515011869), 广州市科技计划项目 (201803030029) 资助
详细信息
    作者简介:

    陈殷齐:中山大学计算机学院硕士研究生. 主要研究方向为图像对齐与拼接. E-mail: chenyq277@mail2.sysu.edu.cn

    郑慧诚:中山大学计算机学院副教授. 2004年获得法国里尔一大博士学位. 主要研究方向为计算机视觉, 神经网络和机器学习. 本文通信作者. E-mail: zhenghch@mail.sysu.edu.cn

    严志伟:中山大学计算机学院硕士研究生. 主要研究方向为深度学习, 目标检测. E-mail: yanzhw5@mail2.sysu.edu.cn

    林峻宇:中山大学电子与信息工程学院本科生. 主要研究方向为自动驾驶. E-mail: linjy95@mail2.sysu.edu.cn

Parallax Image Alignment with Two-stage Mesh Optimization Based on Homography Diffusion Constraints

Funds: Supported by National Natural Science Foundation of China (61976231), Guangdong Basic and Applied Basic Research Foundation (2019A1515011869), and Science and Technology Program of Guangzhou (201803030029)
More Information
    Author Bio:

    Chen Yin-Qi Master student at the School of Computer Science and Engineering, Sun Yat-sen University. His main research interest is image alignment and stitching

    Zheng Hui-Cheng Associate professor at the School of Computer Science and Engineering, Sun Yat-sen University. He received his Ph.D. degree from University of Lille 1, France, in 2004. His research interest covers computer vision, neural networks, and machine learning. Corresponding author of this paper

    Yan Zhi-Wei Master student at the School of Computer Science and Engineering, Sun Yat-sen University. His research interest covers deep learning, object detection

    Lin Jun-Yu Undergraduate student at the School of Electronics and Information Technology, Sun Yat-sen University. His main research interest is autonomous vehicles

  • 摘要: 目前, 在带有视差场景的图像对齐中, 主要难点在某些无法找到足够匹配特征的区域, 这些区域称为匹配特征缺失区域. 现有算法往往忽略匹配特征缺失区域的对齐建模, 而只将有足够匹配特征区域中的部分单应变换系数(如相似性变换系数)传递给匹配特征缺失区域, 或者采用将匹配特征缺失区域转化为有足够匹配特征区域的间接方式, 因此对齐效果仍不理想. 在客观事实上, 位于相同平面的区域应该拥有相同的完整单应变换而非部分变换参数. 由此出发, 利用单应变换系数扩散的思想设计了一个二步网格优化的图像对齐算法. 该方法在第一步网格优化时获得有足够匹配特征区域的单应变换, 再基于提出的单应性扩散约束将这些单应变换系数扩散到邻域网格, 进行第二步网格优化, 在保证优化任务简洁高效的前提下实现单应变换系数的传播与图像对齐. 相较于现有的针对视差场景图像对齐算法, 所提方法在各项指标上都获得了更好的效果.
  • 图  1  HDW优势实例

    Fig.  1  Examples of HDW advantages

    图  2  HDW对齐示意图.

    Fig.  2  The alignment process of HDW.

    图  3  平面分割特性的分析

    Fig.  3  Analysis of plane segmentation characteristics

    图  4  各图像对上量化指标的直观对比

    Fig.  4  Intuitive comparison of the quantitative indicators on all the image pairs

    图  5  Plant实例上的对齐结果对比

    Fig.  5  Comparison of the alignment results on the Plant case

    图  6  Carpark实例上的对齐结果对比

    Fig.  6  Comparison of the alignment results on the Carpark case

    图  7  Stationery实例上的对齐结果对比

    Fig.  7  Comparison of the alignment results on the Stationery case

    图  8  Temple与Railtrack实例上的对齐结果对比

    Fig.  8  Comparison of the alignment results on the Temple and Railtrack cases

    图  9  HDW仍然有提升空间的实例

    Fig.  9  Examples where HDW still has room for improvement

    图  10  HDW的更多对齐结果实例

    Fig.  10  Alignment results of HDW on more examples

    表  1  HDW相对其他算法在Err、PSNR和SSIM上的平均改进(%)

    Table  1  The average improvement of HDW compared with other algorithms on Err, PSNR, and SSIM(%)

    SMH[17]PCPS[18]APAP[6]CPW[8]LLPC[37]ACW[41]
    Err−63.80−66.07−67.15−57.50−71.78−54.49
    PSNR+4.59+9.01+8.02+7.48+13.46+7.12
    SSIM+6.24+8.48+8.65+10.33+36.45+5.67
    下载: 导出CSV

    表  2  图像对的对齐效果量化指标对比

    Table  2  Quantitative comparison of alignment performance on image pairs

    PlantCarparkStationery
    SMH[17]Err1.23381.26591.0580
    PSNR15.749612.375822.9365
    SSIM0.65160.571860.9316
    PCPS[18]Err0.93961.19230.6695
    PSNR15.331411.555023.5924
    SSIM0.68580.56870.9355
    APAP[6]Err4.18541.13371.0154
    PSNR13.299511.936123.4257
    SSIM0.62450.63540.9236
    CPW[8]Err6.37181.64350.9038
    PSNR12.739711.203423.4680
    SSIM0.48180.58620.9258
    ACW[41]Err3.43020.79180.8070
    PSNR13.215912.073823.6319
    SSIM0.65890.65050.9278
    HDWErr0.27410.27870.5134
    PSNR18.196813.501924.2972
    SSIM0.82210.71430.9400
    下载: 导出CSV

    表  3  其他算法相对HDW在耗时上的对比

    Table  3  Temporal costs of other algorithms compared with that of HDW

    方法耗时占比(%)实际耗时(ms)
    HDW100106ms
    SMH[17]12561331
    PCPS[18]5423575
    ARAP[6]19351220511
    CPW[8]4548
    LLPC[37]298316
    ACW[41]20887422140
    下载: 导出CSV
  • [1] Hartley R, Zisserman A. Multiple View Geometry in Computer Vision. Britain: Cambridge University Press, 2003.
    [2] Matsushita Y, Ofek E, Ge W, Tang X, Shum H Y. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(7): 1150-1163 doi: 10.1109/TPAMI.2006.141
    [3] 韩敏, 闫阔, 秦国帅. 基于改进KAZE的无人机航拍图像拼接算法. 自动化学报, 2019, 45(2): 305-314 doi: 10.16383/j.aas.2018.c170521

    Han Min, Yan Kuo, Qin Guo-Shuai. A mosaic algorithm for UAV aerial image with improved KAZE. Acta Automatica Sinica, 2019, 45(2): 305-314 doi: 10.16383/j.aas.2018.c170521
    [4] Li X, Hui N, Shen H, Fu Y, Zhang L. A robust mosaicking procedure for high spatial resolution remote sensing images. ISPRS Journal of Photogrammetry and Remote Sensing, 2015, 109: 108-125 doi: 10.1016/j.isprsjprs.2015.09.009
    [5] Xiang T Z, Xia G S, Bai X, Zhang L. Image stitching by line-guided local warping with global similarity constraint. Pattern Recognition, 2018, 83: 481-497 doi: 10.1016/j.patcog.2018.06.013
    [6] Zaragoza J, Chin T J, Tran Q H, Brown M S, Suter D. As-projective-as-possible image stitching with moving DLT. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1285-1298 doi: 10.1109/TPAMI.2013.247
    [7] Gao J, Kim S J, Brown M S. Constructing image panoramas using dual-homography warping. In: Proceedings of the 2011 Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE, 2011. 49−56
    [8] Zhang F, Liu F. Parallax-tolerant image stitching. In: Proceedings of the 2014 Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014. 3262−3269
    [9] Lin K, Jiang N, Liu S, Cheong L F, Do M, Lu J. Direct photometric alignment by mesh deformation. In: Proceedings of the 2017 Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017. 2405−2413
    [10] Chang C H, Sato Y, Chuang Y Y. Shape-preserving half-projective warps for image stitching. In: Proceedings of the 2014 Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014. 3254−3261
    [11] Lin C C, Pankanti S U, Ramamurthy K N, Aravkin A Y. Adaptive as-natural-as-possible image stitching. In: Proceedings of the 2015 Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015. 1155−1163
    [12] Xue W, Xie W, Zhang Y, Chen S. Stable linear structures and seam measurements for parallax image stitching. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(1): 253-261 doi: 10.1109/TCSVT.2021.3058655
    [13] Capel D. Image Mosaicing and Super-resolution. Berlin: Springer, 2004. 47−79
    [14] Szeliski R. Image alignment and stitching: A tutorial. Foundations & Trends in Computer Graphics & Vision, 2006, 2(1): 1-104
    [15] Mann S, Picard R W. Video orbits of the projective group: A simple approach to featureless estimation of parameters. IEEE Transactions on Image Processing, 1997, 6(9): 1281-1295 doi: 10.1109/83.623191
    [16] Chang C, Chen C, Chuang Y. Spatially-varying image warps for scene alignment. In: Proceedings of the 2014 International Conference on Pattern Recognition. Stockholm, Sverige: IEEE, 2014. 64−69
    [17] Kim S, Uh Y J, Byun H. Generating panorama image by synthesizing multiple homography. In: Proceedings of the 2012 International Conference on Image Processing. Florida, USA: IEEE, 2012. 2981−2984
    [18] Zheng J, Wang Y, Wang H, Li B, Hu H M. A novel projective-consistent plane based image stitching method. IEEE Transactions on Multimedia, 2019, 21(10): 2561-2575 doi: 10.1109/TMM.2019.2905692
    [19] Lin W Y, Liu S, Matsushita Y, Ng T T, Cheong L F. Smoothly varying affine stitching. In: Proceedings of the 2011 Conference on Computer Vision and Pattern Recognition. Colorado Springs, USA: IEEE, 2011. 345−352
    [20] Igarashi T, Moscovich T, Hughes J F. As-rigid-as-possible shape manipulation. ACM Transactions on Graphics, 2005, 24(3): 1134-1141 doi: 10.1145/1073204.1073323
    [21] Guo Y, Liu F, Shi J, Zhou Z H, Gleicher M. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia, 2009, 11(5): 856-867 doi: 10.1109/TMM.2009.2021781
    [22] Hu W, Luo Z, Fan X. Image retargeting via adaptive scaling with geometry preservation. IEEE Journal of Emerging and Selected Topics in Power Electronics, 2014, 4(1): 70-81
    [23] Chang C H, Chuang Y Y. A line-structure-preserving approach to image resizing. In: Proceedings of the 2012 Conference on Computer Vision and Pattern Recognition. Providence, USA: IEEE, 2012. 1075−1082
    [24] Wang Y S, Tai C L, Sorkine O, Lee T Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics, 2008, 27(5): Article No. 118
    [25] Li S, Yuan L, Sun J, Quan L. Dual-feature warping-based motion model estimation. In: Proceedings of the 2015 International Conference on Computer Vision. Santiago, Chile: IEEE, 2015. 428−4291
    [26] 何川, 周军. 具有直线结构保护的网格化图像拼接. 中国图像图形学报, 2018, 23(7): 973-983

    He Chuan, Zhou Jun. Mesh-based image stitching algorithm with linear structure protection. Journal of Image and Graphics, 2018, 23(7): 973-983
    [27] Li J, Wang Z, Lai S. Parallax-tolerant image stitching based on robust elastic warping. IEEE Transactions on Multimedia, 2018, 20(7): 1672-1687 doi: 10.1109/TMM.2017.2777461
    [28] Chen K, Tu J, Xiang B, Li L, Yao J. Multiple combined constraints for image stitching. In: Proceedings of the 2018 International Conference on Image Processing. Athens, The Hellenic Republic: IEEE, 2018. 1253−1257
    [29] 李加亮, 蒋品群. 结合变形函数和幂函数权重的图像拼接. 计算机应用, 2019, 39(10): 3060-4064

    Li Jia Liang, Jiang Pin Qun. Image stitching by combining deformation function and power function weight. Journal of Computer Applications, 2019, 39(10): 3060-4064
    [30] 颜雪军, 赵春霞, 袁夏. 2DPCA-SIFT: 一种有效的局部特征描述方法. 自动化学报, 2014, 40(4): 675-682

    Yan Xue Jun, Zhao Chun Xia, Yuan Xia. 2DPCA-SIFT: An efficient local feature descriptor. Acta Automatica Sinica, 2014, 40(4): 675-682
    [31] 闫自庚, 蒋建国, 郭丹. 基于SURF特征和Delaunay三角网格的图像匹配. 自动化学报, 2014, 40(6): 1216-1222

    Yan Zi Geng, Jiang Jian Guo, Guo Dan. Image matching based on SURF feature and Delaunay triangular meshes. Acta Automatica Sinica, 2014, 40(6): 1216-1222
    [32] Liu W X, Chin T J. Correspondence insertion for as-projective-as-possible image stitching. arXiv: 1608.07997, 2016.
    [33] Fan B, Wu F, Hu Z. Line matching leveraged by point correspondences. In: Proceedings of the 2010 Conference on Computer Vision and Pattern Recognition. San Francisco, USA: IEEE, 2010. 390−397
    [34] Joo K, Kim N, Oh T H, Kweon I S. Line meets as-projective-as-possible image stitching with moving DLT. In: Proceedings of the 2015 International Conference on Image Processing. Quebec, Canada: IEEE, 2015. 1175−1179
    [35] Xiang T, Xia G, Zhang L, Huang N. Locally warping-based image stitching by imposing line constraints. In: Proceedings of the 2016 International Conference on Pattern Recognition. Cancun, Mexico: IEEE, 2016. 4178−4183
    [36] Luo X, Li Y, Yan J, Guan X. Image stitching with positional relationship constraints of feature points and lines. Pattern Recognition Letters, 2020, 135: 431-440 doi: 10.1016/j.patrec.2020.05.003
    [37] Jia Q, Li Z, Fan X, Zhao H, Teng S, Ye X, Latecki L J. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In: Proceedings of the 2021 Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE, 2021. 12186−12195
    [38] Herrmann C, Wang C, Bowen R S, Keyder E, Krainin M, Liu C, et al. Robust image stitching with multiple registrations. In: Proceedings of the 2018 European Conference on Computer Vision. Munich, Germany: 2018. 53−69
    [39] Lou Z, Gevers T. Image alignment by piecewise planar region matching. IEEE Transactions on Multimedia, 2014, 16(7): 2052-2061 doi: 10.1109/TMM.2014.2346476
    [40] Lee K Y, Sim J Y. Warping residual based image stitching for large parallax. In: Proceedings of the 2020 Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020. 8198−8206
    [41] Chen Y, Zheng H, Ma Y, Yan Z. Image stitching based on angle-consistent warping. Pattern Recognition, 2021, 117: Article No. 107993 doi: 10.1016/j.patcog.2021.107993
    [42] Song D Y, Um G M, Lee H K, Cho D. End-to-end image stitching network via multi-homography estimation. IEEE Signal Processing Letters, 2021, 99: 1-5
    [43] Le H, Liu F, Zhang S, and Agarwala A. Deep homography estimation for dynamic scenes. In: Proceedings of the 2020 Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020. 7652−7661
    [44] Zhao Q, Ma Y, Zhu C, Yao C, Dai F. Image stitching via deep homography estimation. Neurocomputing, 2021, 450: 219-229 doi: 10.1016/j.neucom.2021.03.099
    [45] Dai Q, Fang F, Li J, Zhang G, Zhou A. Edge-guided composition network for image stitching. Pattern Recognition, 2021, 118: Article No. 108019 doi: 10.1016/j.patcog.2021.108019
    [46] Cao S, Hu J, Shen Z, Shen H. Iterative deep homography estimation. In: Proceedings of the 2022 Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022.
    [47] Zhao Y, Huang X, Zhang Z. Deep Lucas-Kanade homography for multimodal image alignment. In: Proceedings of the 2021 Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE, 2021, 15950−15959
    [48] Mo Y, Kang X, Duan P, Li S. A robust UAV hyperspectral image stitching method based on deep feature matching. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14
    [49] Nie L, Lin C, Liao K, Liu S, Zhao Y. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Transactions on Image Processing, 2021, 30: 6184-6197 doi: 10.1109/TIP.2021.3092828
    [50] Nie L, Lin C, Liao K, Liu S, Zhao Y. Depth-aware multi-grid deep homography estimation with contextual correlation. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(7): 4460-4472 doi: 10.1109/TCSVT.2021.3125736
    [51] Liu F, Gleicher M, Jin H, Agarwala A. Content-preserving warps for 3d video stabilization. ACM Transactions on Graphics, 2009, 28(3): 1-9
    [52] Fischler M A, Bolles R C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communication of ACM, 1981, 24(6): 381-395 doi: 10.1145/358669.358692
    [53] Zhang Z. Parameter estimation techniques: A tutorial with application to conic fitting. Image and Vision Computing, 1997, 15(1): 59-76 doi: 10.1016/S0262-8856(96)01112-2
    [54] Heckbert P S. Fundamentals of texture mapping and image warping[Ph. D. dissertation], EECS Department, University of California, Berkeley, 1989
    [55] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612 doi: 10.1109/TIP.2003.819861
    [56] Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91-110 doi: 10.1023/B:VISI.0000029664.99615.94
    [57] Brown M, Lowe D G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 2007, 74(1): 59-73 doi: 10.1007/s11263-006-0002-3
    [58] Ma J, Jiang X, Fan A, Jiang J, Yan J. Image matching from handcrafted to deep features: A survey. International Journal of Computer Vision, 2021, 129: 23-79 doi: 10.1007/s11263-020-01359-2
  • 加载中
计量
  • 文章访问数:  221
  • HTML全文浏览量:  114
  • 被引次数: 0
出版历程
  • 收稿日期:  2021-10-13
  • 录用日期:  2022-05-17
  • 网络出版日期:  2022-09-05

目录

    /

    返回文章
    返回