快速鲁棒的城市场景分段平面重建

王伟; 高伟; 朱海; 胡占义

doi:10.16383/j.aas.2017.c160261

快速鲁棒的城市场景分段平面重建

doi: 10.16383/j.aas.2017.c160261

王伟^1,,
高伟^2,3, ,,
朱海^1,,
胡占义^2,3,

1.
周口师范学院网络工程学院周口 466000
2.
中国科学院自动化研究所模式识别国家重点实验室北京 100190
3.
中国科学院大学北京 100049

基金项目:

河南省高校重点科研项目 17A520018

国家自然科学基金 61472419

河南省科技攻关项目 162102310589

河南省高校重点科研项目 15A520116

河南省高校重点科研项目 16B520034

河南省自然科学基金 162300410347

河南省科技攻关项目 162102210396

河南省高校重点科研项目 17A520019

国家高技术研究发展计划（863计划） 2015AA124102

国家自然科学基金 61273280

周口师范学院高层次人才科研启动基金 zknuc2015103

国家自然科学基金 61333015

详细信息

作者简介:
王伟周口师范学院网络工程学院副教授.2014年获得中国科学院自动化研究所博士学位.主要研究方向为计算机视觉和模式识别.E-mail:wangwei@zknu.cn

朱海周口师范学院网络工程学院副教授.2010年获得西安电子科技大学博士学位.主要研究方向为计算机视觉和云计算.E-mail:zhusea@163.com

胡占义中国科学院自动化研究所研究员.1993年获得比利时列日大学博士学位.主要研究方向为计算机视觉和机器学习.E-mail:huzy@nlpr.ia.ac.cn

通讯作者:
高伟中国科学院自动化研究所副研究员.2008年获得中国科学院自动化研究所博士学位.主要研究方向为计算机视觉和三维重建.E-mail:wgao@nlpr.ia.ac.cn

计量
- 文章访问数: 2625
- HTML全文浏览量: 389
- PDF下载量: 676
- 被引次数: 0
出版历程
- 收稿日期: 2016-03-11
- 录用日期: 2016-07-11
- 刊出日期: 2017-04-20

Rapid and Robust Piecewise Planar Reconstruction of Urban Scenes

WANG Wei^1
,,
GAO Wei^{2,3
, ,},
ZHU Hai^1
,,
HU Zhan-Yi^{2,3
,}

1.
School of Network Engineering, Zhoukou Normal University, Zhoukou 466000
2.
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190
3.
University of Chinese Academy of Sciences, Beijing 100049

Funds:

College Key Research Project of Henan 17A520018

National Natural Science Foundation of China 61472419

Key Scientiflc and Technological Project of Henan 162102310589

College Key Research Project of Henan 15A520116

College Key Research Project of Henan 16B520034

Natural Science Foundation of of Henan Province 162300410347

Key Scientiflc and Technological Project of Henan 162102210396

College Key Research Project of Henan 17A520019

National High Technology Research and Development Program of China (863 Program) 2015AA124102

National Natural Science Foundation of China 61273280

Scientiflc Research Starting Foundation for Advanced Talents of Zhoukou Normal University zknuc2015103

National Natural Science Foundation of China 61333015

More Information

Author Bio:
Associate professor at the School of Network Engineering, Zhoukou Normal University. He received his Ph. D. degree from the Institute of Automation, Chinese Academy of Sciences in 2014. His research interest covers computer vision and pattern recognition

Associate professor at the School of Network Engineering, Zhoukou Normal University. He received his Ph. D. degree from Xidian University in 2010. His research interest covers computer vision and cloud computing

Professor at the Institute of Automation, Chinese Academy of Sciences. He received his Ph. D. degree from the University of Liege, Belgium in 1993. His research interest covers computer vision and machine learning

Corresponding author: GAO Wei Associate professor at the Institute of Automation, Chinese Academy of Sciences. He received his Ph. D. degree from the Institute of Automation, Chinese Academy of Sciences in 2008. His research interest covers computer vision and 3D reconstruction. Corresponding author of this paper

摘要

摘要: 对于基于图像的城市场景重建，由于光照变化、透视畸变、弱纹理区域等因素的影响，传统像素级与区域级的重建算法通常难以获得可靠的重建结果.为了解决此问题，本文提出一种快速、鲁棒的分段平面重建算法.根据城市场景结构特征与分段平面假设，本文算法首先利用基于连通域检测的空间平面拟合方法从初始空间点中抽取充分且可靠的候选空间平面，然后在MRF（Markov random field）能量最小化框架下将场景的完整结构推断问题转化为平面标记问题进行求解.由于候选平面集与融合灰度一致性度量、空间几何与可见性约束的能量模型的高可靠性，场景的完整结构因此可被有效地重建.实验结果表明，本文算法能较好地克服传统算法可靠性差、重建场景不完整等缺点，同时具有较高的计算效率.
- 能量最小化 /
- 平面拟合 /
- 三维重建 /
- 深度图
Abstract: For image-based urban scene reconstruction, traditional pixel-level or region-level methods often fail to achieve satisfactory results because of various negative factors such as illumination variations, perspective distortion, poorly textured regions, etc. To address this problem, a rapid and robust piecewise planar stereo method is proposed in this paper. Under the scene piecewise planar assumption and by taking into account the structural characteristics of urban scenes, the proposed method at first extracts sufficient and reliable candidate planes from initial spatial points based on the connected-region detection; then the problem of scene reconstruction is converted into a plane labeling problem under the Markov random field (MRF) framework. By virtue of the high reliability of our extracted candidate planes and the energy model properly designed by incorporating geometric constraints, spatial visibility and photo-consistency measures, a complete scene structure is effectively reconstructed. Experiment results show that the proposed method can satisfactorily handle the low-reliability and incomplete-reconstruction problems in traditional methods with computational efficiency.
- Energy minimization /
- plane fitting /
- 3D reconstruction /
- depth map
注释:

1) 本文责任编委贾云得

HTML全文

图 1 摄像机与图像示例

Fig. 1 Cameras and sample images

下载: 全尺寸图片幻灯片

图 2 城市场景重建问题

Fig. 2 Problems in reconstructing urban scenes

下载: 全尺寸图片幻灯片

图 3 本文算法基本流程图

Fig. 3 Flowchart of our proposed method

下载: 全尺寸图片幻灯片

图 4 候选平面的抽取

Fig. 4 Extraction of candidate planes

下载: 全尺寸图片幻灯片

图 5 场景结构的推断

Fig. 5 Inference of scene structures

下载: 全尺寸图片幻灯片

图 6 数据集例图 (上:标准数据; 下:真实数据)

Fig. 6 Sample images (Up: standard data; Down: real-world data)

下载: 全尺寸图片幻灯片

图 7 标准数据集实验结果

Fig. 7 Results for standard data sets

下载: 全尺寸图片幻灯片

图 8 真实数据集实验结果 (图像从上而下为: #1, #2, #3和#4)

Fig. 8 Results for real-world data sets (Images from top to bottom are #1, #2, #3 and #4.)

下载: 全尺寸图片幻灯片

图 9 多幅图像对应的初始空间点与本文算法重建结果(左:场景#1;右:场景#2)

Fig. 9 Initial 3D points and reconstruction results of multiple images (Left: scene #1; Right: scene #2)

下载: 全尺寸图片幻灯片

表 1 在标准数据集上的算法性能比较

Table 1 Performance comparisons on standard data sets

图像	初始空间点数量	本文算法						文献[12]
		超像素数量	平面数量	重建精度	时间 (s)			超像素数量	平面数量	重建精度	时间 (s)
		超像素数量	平面数量	重建精度	抽取	推断	合计	超像素数量	平面数量	重建精度	抽取	推断	合计
Herz-Jesu	29 541	49 326	12	0.8043	73.15	138.81	211.96	3 096	7	0.3296	136.72	124.50	258.22
Castle	12 337	26 825	9	0.7452	57.96	218.06	276.01	4 127	5	0.2701	107.40	196.80	304.21

下载: 导出CSV

表 2 在真实数据集上的算法性能比较

Table 2 Performance comparisons on real-world data sets

图像序号	初始空间点	本文算法				文献[12]
图像序号	初始空间点	超像素	平面	精度	时间 (s)	超像素	平面	精度	时间 (s)
#1	16 520	13 907	6	0.7554	58.22	1 996	4	0.3187	78.13
#2	20 468	9 562	7	0.7852	49.36	1 768	6	0.4165	60.98
#3	13 710	23 955	6	0.6486	70.01	2 341	6	0.3547	76.39
#4	10 694	18 933	7	0.5713	64.52	3 011	5	0.2310	72.67

下载: 导出CSV

参考文献(19)

[1]	Hong L, Chen G. Segment-based stereo matching using graph cuts. In:Proceeding of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC, USA:IEEE, 2004. I-74-I-81
[2]	Wang Z F, Zheng Z G. A region based stereo matching algorithm using cooperative optimization. In:Proceeding of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA:IEEE, 2008. 1-8
[3]	Gorelick L, Boykov Y, Veksler O, Ayed I B, Delong A. Submodularization for binary pairwise energies. In:Proceeding of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA:IEEE, 2014. 1154-1161
[4]	Çiğla C, Zabulis X, Aydin A A. Segment-based stereo-matching via plane and angle sweeping. In:Proceedings of the 2007 3DTV Conference. Kos Island, Greece:IEEE, 2007. 1-4
[5]	Furukawa Y, Curless B, Seitz S M, Szeliski R. Manhattan-world stereo. In:Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, USA:IEEE, 2009. 1422-1429
[6]	Furukawa Y, Ponce J. Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(8):1362-1376 doi: 10.1109/TPAMI.2009.161
[7]	Gallup D, Frahm J M, Mordohai P, Yang Q X, Pollefeys M. Real-time plane-sweeping stereo with multiple sweeping directions. In:Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, USA:IEEE, 2007. 1-8
[8]	Mičušík B, Košecká J. Multi-view superpixel stereo in urban environments. International Journal of Computer Vision, 2010, 89(1):106-119 doi: 10.1007/s11263-010-0327-9
[9]	Sinha S N, Steedly D, Szeliski R. Piecewise planar stereo for image-based rendering. In:Proceedings of the 12th IEEE International Conference on Computer Vision. Kyoto, Japan:IEEE, 2009. 1881-1888
[10]	Chauve A L, Labatut P, Pons J P. Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data. In:Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, USA:IEEE, 2010. 1261-1268
[11]	Kowdle A, Sinha S N, Szeliski R. Multiple view object cosegmentation using appearance and stereo cues. In:Proceedings of the 12th European Conference on Computer Vision. Florence, Italy:Springer-Verlag, 2012. 789-803
[12]	Bódis-Szomorú A, Riemenschneider H, van Gool L. Fast, approximate piecewise-planar modeling based on sparse structure-from-motion and superpixels. In:Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA:IEEE, 2014. 469-476
[13]	Comaniciu D, Meer P. Mean shift:a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(5):603-619 doi: 10.1109/34.1000236
[14]	Kumar M P, Turki H, Preston D, Koller D. Parameter estimation and energy minimization for region-based semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(7):1373-1386 doi: 10.1109/TPAMI.2014.2372766
[15]	Toldo R, Fusiello A. Robust multiple structures estimation with J-linkage. In:Proceedings of the 10th European Conference on Computer Vision. Marseille, France:Springer, 2008. 537-547
[16]	Isack H, Boykov Y. Energy-based geometric multi-model fitting. International Journal of Computer Vision, 2012, 97(2):123-147 doi: 10.1007/s11263-011-0474-7
[17]	Strecha C, von Hansen W, van Gool L, Fua P, Thoennessen U. On benchmarking camera calibration and multi-view stereo for high resolution imagery. In:Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, USA:IEEE, 2008. 1-8
[18]	Shen S H. Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes. IEEE Transactions on Image Processing, 2013, 22(5):1901-1914 doi: 10.1109/TIP.2013.2237921
[19]	王伟, 高伟, 胡占义.基于语义约束与Graph Cuts的稠密三维场景重建.中国科学:信息科学, 2014, 44(6):774-792 http://www.cnki.com.cn/Article/CJFDTOTAL-PZKX201406009.htm Wang Wei, Gao Wei, Hu Zhan-Yi. Dense 3D scene reconstruction based on semantic constraint and graph cuts. Science China Information Sciences, 2014, 44(6):774-792 http://www.cnki.com.cn/Article/CJFDTOTAL-PZKX201406009.htm