【论文整理】3D 重建论文集 3D reconstruction paper list Contents • Tutorials • Papers – SLAM – SFM Incremental SfM Global SfM Hierarchical SfM Multi-Stage SfM • Non Rigid SfM
Viewing graph optimization Unordered feature tracking • Large scale image matching for SfM – Localization • Real time localization in SfM reconstructions – MVS Point cloud computation Surface computation & refinements Machine Learning MVS • Multiple View Mesh Texturing – UAV Trajectory Optimization for model completeness • OpenSource software resources SfM Multiple View Geometry Library Solvers MVS (Multiple View Stereovision) SLAM – Large Scale Image Retrieval
Minimization Nearest Neighbor Search – Mesh storage processing • Feature detection description • Datasets with ground truth - Reproducible research • License • Contributing
Tutorials SLAM Tutorial & survey Micro Flying Robots: from Active Vision to Event-based Vision D. Scaramuzza. ICRA 2016 Aerial Robotics - (Visual odometry) D. Scaramuzza Simultaneous Localization And Mapping: Present, Future, and the Robust-Perception Age. C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. D. Reid, J. J. Leonard. • “The paper summarizes the outcome of the workshop “The Problem of Mobile Sensors: Setting future goals and indicators of progress for SLAM” held during the Robotics: Science and System (RSS) conference (Rome, July 2015).” Visual Odometry: Part I - The First 30 Years and Fundamentals, D. Scaramuzza and F. Fraundorfer, IEEE Robotics and Automation Magazine, Volume 18, issue 4, 2011 Visual Odometry: Part II - Matching, robustness, optimization, and applications, F. Fraundorfer and D. Scaramuzza, IEEE Robotics and Automation Magazine, Volume 19, issue 2, 2012 Large-scale, real-time visual-inertial localization revisited S. Lynen, B. Zeisl, D. Aiger, M. Bosse, J. Hesch, M. Pollefeys, R. Siegwart and T. Sattler. Arxiv 2019. SfM tutorial Open Source Structure-from-Motion. M. Leotta, S. Agarwal, F. Dellaert, P. Moulon, V. Rabaud. CVPR 2015 Tutorial. Large-scale 3D Reconstruction from Images. T. Shen, J. Wang, T.Fang, L. Quan. ACCV 2016 Tutorial.
MVS tutorial Multi-View Stereo: A Tutorial. Y. Furukawa, C. Hernández. Foundations and Trends® in Computer Graphics and Vision, 2015. State of the Art 3D Reconstruction Techniques N. Snavely, Y. Furukawa, CVPR 2014 tutorial slides. Introduction MVS with priors - Large scale MVS RGB-D mapping 3D indoor scene modeling from RGB-D data: a survey K. Chen, YK. Lai and SM. Hu. Computational Visual Media 2015. State of the Art on 3D Reconstruction with RGB-D Cameras K. Hildebrandt and C. Theobalt EUROGRAPHICS 2018. All in one tutorial Introduction of Visual SLAM, Structure from Motion and Multiple View Stereo. Yu Huang 2014. Computer vision books Computer Vision: Algorithms and Applications. R. Szeliski. 2010.
Papers
SLAM/VO Visual odometry (image based only) Real-time simultaneous localisation and mapping with a single camera. A. J. Davison. ICCV 2003. Visual odometry. D. Nister, O. Naroditsky, and J. Bergen. CVPR 2004. Real time localization and 3d reconstruction. E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd. CVPR 2006. Parallel Tracking and Mapping for Small AR Workspaces. G. Klein, D. Murray. ISMAR 2007. Real-Time 6-DOF Monocular Visual SLAM in a Large-scale Environments. H. Lim, J. Lim, H. Jin Kim. ICRA 2014.
Direct Sparse Odometry, J. Engel, V. Koltun, D. Cremers, arXiv:1607.02565, 2016. Visual SLAM algorithms: a survey from 2010 to 2016, T. Taketomi, H. Uchiyama, S. Ikeda, IPSJ T Comput Vis Appl 2017. ∇SLAM: Dense SLAM meets Automatic Differentiation. K. M. Jatavallabhula, G. Iyer, L. Paull. arXiv:1910.10672, 2019. Direct Sparse Mapping J. Zubizarreta, I. Aguinaga and J. M. M. Montiel. arXiv:1904.06577, 2019. OpenVSLAM: A Versatile Visual SLAM Framework Sumikura, Shinya and Shibuya, Mikiya and Sakurada, Ken. In Proceedings of the 27th ACM International Conference on Multimedia 2019
SfM papers
Incremental SfM Photo Tourism: Exploring Photo Collections in 3D. N. Snavely, S. M. Seitz, and R. Szeliski. SIGGRAPH 2006. Towards linear-time incremental structure from motion. C. Wu. 3DV 2013. Structure-from-Motion Revisited. Schöenberger, Frahm. CVPR 2016.
Global SfM Combining two-view constraints for motion estimation V. M. Govindu. CVPR, 2001. Lie-algebraic averaging for globally consistent motion estimation. V. M. Govindu. CVPR, 2004. Robust rotation and translation estimation in multiview reconstruction. D. Martinec and T. Pajdla. CVPR, 2007. Non-sequential structure from motion. O. Enqvist, F. Kahl, and C. Olsson. ICCV OMNIVIS Workshops 2011. Global motion estimation from point matches. M. Arie-Nachimson, S. Z. Kovalsky, I. KemelmacherShlizerman, A. Singer, and R. Basri. 3DIMPVT 2012. Global Fusion of Relative Motions for Robust, Accurate and Scalable Structure from Motion. P. Moulon, P. Monasse and R. Marlet. ICCV 2013.
A Global Linear Method for Camera Pose Registration. N. Jiang, Z. Cui, P. Tan. ICCV 2013. Global Structure-from-Motion by Similarity Averaging. Z. Cui, P. Tan. ICCV 2015. Linear Global Translation Estimation from Feature Tracks Z. Cui, N. Jiang, C. Tang, P. Tan, BMVC 2015.
Hierarchical SfM Structure-and-Motion Pipeline on a Hierarchical Cluster Tree. A. M.Farenzena, A.Fusiello, R. Gherardi. Workshop on 3-D Digital Imaging and Modeling, 2009. Randomized Structure from Motion Based on Atomic 3D Models from Camera Triplets. M. Havlena, A. Torii, J. Knopp, and T. Pajdla. CVPR 2009. Efficient Structure from Motion by Graph Optimization. M. Havlena, A. Torii, and T. Pajdla. ECCV 2010. Hierarchical structure-and-motion recovery from uncalibrated images. Toldo, R., Gherardi, R., Farenzena, M. and Fusiello, A… CVIU 2015.
Multi-Stage SfM Parallel Structure from Motion from Local Increment to Global Averaging. S. Zhu, T. Shen, L. Zhou, R. Zhang, J. Wang, T. Fang, L. Quan. arXiv 2017. Multistage SFM : Revisiting Incremental Structure from Motion. R. Shah, A. Deshpande, P. J. Narayanan. 3DV 2014. -> Multistage SFM: A Coarse-to-Fine Approach for 3D Reconstruction, arXiv 2016. HSfM: Hybrid Structure-from-Motion. H. Cui, X. Gao, S. Shen and Z. Hu, ICCV 2017.
Non Rigid SfM Robust Structure from Motion in the Presence of Outliers and Missing Data. G. Wang, J. S. Zelek, J. Wu, R. Bajcsy. 2016.
Viewing graph optimization Skeletal graphs for efficient structure from motion. N. Snavely, S. Seitz, R. Szeliski. CVPR 2008
Optimizing the Viewing Graph for Structure-from-Motion. C. Sweeney, T. Sattler, M. Turk, T. Hollerer, M. Pollefeys. ICCV 2015 Graph-Based Consistent Matching for Structure-from-Motion. T. Shen, S. Zhu, T. Fang, R. Zhang, L. Quan. ECCV 2016.
Unordered feature tracking Unordered feature tracking made fast and easy. P. Moulon and P. Monasse. CVMP 2012. Point Track Creation in Unordered Image Collections Using Gomory-Hu Trees. Svärm, Simayijiang, Enqvist, Olsson. ICPR 2012. Fast connected components computation in large graphs by vertex pruning. A. Lulli, E. Carlini, P. Dazzi, C. Lucchese, and L. Ricci. IEEE Transactions on Parallel and Distributed Systems 2016.
Large scale image matching for SfM Video Google: A Text Retrieval Approach to Object Matching in Video. J. Sivic, F. Schaffalitzky and A. Zisserman. ICCV 2003. Scalable Recognition with a Vocabulary Tree. Nister, Stewenius, CVPR 2006. Building Rome in a Day. S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, R. Szeliski. ICCV 2009. Product quantization for nearest neighbor search. H. Jégou, M. Douze and C. Schmid. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011. Fast and Accurate Image Matching with Cascade Hashing for 3D Reconstruction. J. Cheng, C. Leng, J. Wu, H. Cui, H. Lu. CVPR 2014. Recent developments in large-scale tie-point matching. Hartmann, Havlena, Schindler. ISPRS 2016. Graphmatch: Efficient Large-Scale Graph Construction for Structure from Motion. C. Qiaodong, V. Fragoso, C. Sweeney and P. Sen. 3DV 2017.
Localization
Real time localization in SfM reconstructions Real-time Image-based 6-DOF Localization in Large-Scale Environments. Lim, Sinha, Cohen, Uyttendaele. CVPR 2012. Get Out of My Lab: Large-scale, Real-Time Visual-Inertial Localization. Lynen, Sattler, Bosse, Hesch, Pollefeys, Siegwart. RSS 2015. DSAC - Differentiable RANSAC for Camera Localization. E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, C. Rother. CVPR 2017. Learning Less is More - 6D Camera Localization via 3D Surface Regression. E. Brachmann, C. Rother. Submitted to CVPR 2018.
Multiple View Stereovision
Point cloud computation Accurate, Dense, and Robust Multiview Stereopsis. Y. Furukawa, J. Ponce. CVPR 2007. PAMI 2010 State of the art in high density image matching. F. Remondino, M.G. Spera, E. Nocerino, F. Menna, F. Nex . The Photogrammetric Record 29(146), 2014. Progressive prioritized multi-view stereo. A. Locher, M. Perdoch and L. Van Gool. CVPR 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. J. L. Schönberger, E. Zheng, M. Pollefeys, J.-M. Frahm. ECCV 2016. TAPA-MVS: Textureless-Aware PAtchMatch Multi-View Stereo. A. Romanoni, M. Matteucci. ICCV 2019
Surface computation & refinements Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts. P. Labatut, J-P. Pons, R. Keriven. ICCV 2007 Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh. S. N. Sinha, P. Mordohai and M. Pollefeys. ICCV 2007. Towards high-resolution large-scale multi-view stereo. H.-H. Vu, P. Labatut, J.-P. Pons, R. Keriven. CVPR 2009.
Refinement of Surface Mesh for Accurate Multi-View Reconstruction. R. Tylecek and R. Sara. IJVR 2010. High Accuracy and Visibility-Consistent Dense Multiview Stereo. H.-H. Vu, P. Labatut, J.-P. Pons, R. Keriven. Pami 2012. Exploiting Visibility Information in Surface Reconstruction to Preserve Weakly Supported Surfaces M. Jancosek et al. 2014. A New Variational Framework for Multiview Surface Reconstruction. B. Semerjian. ECCV 2014. Photometric Bundle Adjustment for Dense Multi-View 3D Modeling. A. Delaunoy, M. Pollefeys. CVPR2014. Global, Dense Multiscale Reconstruction for a Billion Points. B. Ummenhofer, T. Brox. ICCV 2015. Efficient Multi-view Surface Refinement with Adaptive Resolution Control. S. Li, S. Yu Siu, T. Fang, L. Quan. ECCV 2016. Multi-View Inverse Rendering under Arbitrary Illumination and Albedo, K. Kim, A. Torii, M. Okutomi, ECCV2016. Shading-aware Multi-view Stereo, F. Langguth and K. Sunkavalli and S. Hadap and M. Goesele, ECCV 2016. Scalable Surface Reconstruction from Point Clouds with Extreme Scale and Density Diversity, C. Mostegel, R. Prettenthaler, F. Fraundorfer and H. Bischof. CVPR 2017. Multi-View Stereo with Single-View Semantic Mesh Refinement, A. Romanoni, M. Ciccone, F. Visin, M. Matteucci. ICCVW 2017
Machine Learning based MVS Matchnet: Unifying feature and metric learning for patch-based matching, X. Han, Thomas Leung, Y. Jia, R. Sukthankar, A. C. Berg. CVPR 2015. Stereo matching by training a convolutional neural network to compare image patches, J., Zbontar, and Y. LeCun. JMLR 2016. Efficient deep learning for stereo matching, W. Luo, A. G. Schwing, R. Urtasun. CVPR 2016. Learning a multi-view stereo machine, A. Kar, C. Häne, J. Malik. NIPS 2017. Learned multi-patch similarity, W. Hartmann, S. Galliani, M. Havlena, L. V. Gool, K. Schindler.I CCV 2017.
Surfacenet: An end-to-end 3d neural network for multiview stereopsis, Ji, M., Gall, J., Zheng, H., Liu, Y., Fang, L. ICCV2017. DeepMVS: Learning Multi-View Stereopsis, Huang, P. and Matzen, K. and Kopf, J. and Ahuja, N. and Huang, J. CVPR 2018. RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials, D. Paschalidou and A. O. Ulusoy and C. Schmitt and L. Gool and A. Geiger. CVPR 2018. MVSNet: Depth Inference for Unstructured Multi-view Stereo, Y. Yao, Z. Luo, S. Li, T. Fang, L. Quan. ECCV 2018. Learning Unsupervised Multi-View Stereopsis via Robust Photometric Consistency, T. Khot, S. Agrawal, S. Tulsiani, C. Mertz, S. Lucey, M. Hebert. 2019. DPSNET: END-TO-END DEEP PLANE SWEEP STEREO, Sunghoon Im, Hae-Gon Jeon, Stephen Lin, In So Kweon. 2019. Point-based Multi-view Stereo Network, Rui Chen, Songfang Han, Jing Xu, Hao Su. ICCV 2019.
Multiple View Mesh Texturing Seamless image-based texture atlases using multi-band blending. C. Allène, J-P. Pons and R. Keriven. ICPR 2008. Let There Be Color! - Large-Scale Texturing of 3D Reconstructions. M. Waechter, N. Moehrle, M. Goesele. ECCV 2014.
UAV Trajectory Optimization for model completeness Submodular Trajectory Optimization for Aerial 3D Scanning. M. Roberts, A. Truong, D. Dey, S. Sinha, A. Kapoor, N. Joshi, P. Hanrahan. 2017.
OpenSource resources
OpenSource SfM (Structure from Motion) Project Language License Bundler GNU General Public License - contamination Colmap C++ BSD 3-clause license - Permissive
MAP-Tk BSD 3-Clause license - Permissive MicMac CeCILL-B MVE BSD 3-Clause license + parts under the GPL 3 license MVG C++ MPL2 - Permissive OpenSfM Python Simplified BSD license - Permissive TheiaSfM C++ New BSD license - Permissive
OpenSource Multiple View Geometry Library Solvers Project Language License OpenGV C++ BSD - permissive
OpenSource MVS (Multiple View Stereovision) Project Language License Colmap C++ CUDA BSD 3-clause license - Permissive (Can use CGAL -> GNU General Public License - contamination) GPUIma + fusibile CUDA HPMVS GNU General Public License - contamination MICMAC CeCILL-B MVE C++ BSD 3-Clause license + parts under the GPL 3 license OpenMVS C++ (CUDA optional) AGPL3 PMVS C++ CUDA GNU General Public License - contamination SMVS Shading-aware Multi-view Stereo C++ BSD-3-Clause license
OpenSource SLAM (Simultaneous Localization And Mapping) Project Language License COSLAM C++ GNU General Public License DSO-Direct Sparse Odometry GPLv3 DTSLAM-Deferred Triangulation SLAM modified BSD LSD-SLAM C++/ROS GNU General Public
License MAPLAB-ROVIOLI C++/ROS Apachev2.0 OKVIS: Open Keyframe-based Visual-Inertial SLAM C++ BSD ORB-SLAM C++ GPLv3 REBVO - Realtime Edge Based Visual Odometry for a Monocular Camera C++ GNU General Public License SVO semi-direct Visual Odometry C++/ROS GNU General Public License
Large scale image retrieval / CBIR (Content Based Image Retrieval) Project Language License DBoW2 modified BSD License libvot 3-Clause License VocabTree2 C++ BSD License
OpenSource minimization Project Language License CERES SOLVER GTSAM G2O BSD License + L/GPL3 restriction NLOPT C++ LGPL
Nearest Neighbor Search Project Language License ANN GNU General Public License Annoy Apache License FLANN Libnabo Nanoflann C++ BSD License
Mesh storage processing Project Language License 3DTK GPLv3 CGAL Module dependent GPL/LGPL InstantMesh Mesh Simplification BSD License GEOGRAM Revised BSD License libigl MPL2 Mesh-processing-library 3D MIT License OpenMesh BSD 3 clause license PCL 3-clause BSD license VCG C++ GPL
Features Features detection/Description From handcrafted to deep local features. G. Csurka, C. R. Dance, M. Humenberger. 2018. Project Detection Description AKAZE MSURF/MLDB DART x KAZE x MSURF/MLDB LIOP/MIOP
x LIFT (machine learning) x x MROGH
x SIFT SURF x SFOP x
…
“Real time” oriented methods Project Detection Description BRIEF
x
BRISK x FAST x
FREAK
x FRIF x x HIPS
x LATCH
x MOPS
x PhonySift -scale Fast Reduced Sift grid ORB Multiscale Fast Oriented BRIEF
Datasets with ground truth - Reproducible research Feature detection/description repeatability VGG Oxford 8 dataset with GT homographies + matlab code. Hannover - Region Detector Evaluation Data Set Similar to the previous (5 dataset). Datasets have multiple image resolution & an increased GT homographies precision. DTU - Robot Image Data Sets - Point Feature Data Set 60 scenes with know calibration & different illuminations. Corresponding interest point patches for descriptor learning Corresponding patches, saved with a canonical scale and orientation. Multi-view Stereo Correspondence Dataset HPatches Dataset linked to the ECCV16 workshop “Local Features: State of the art, open problems and performance evaluation” Monocular odometry dataset Mono dataset 50 real-world sequences. Dataset linked to the DSO Visual Odometry paper. MVS - Point Cloud - Surface accuracy Middlebury Multi-view Stereo See “A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms”. CVPR 2006.
Dense MVS See “On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery”. CVPR 2008. DTU - Robot Image Data Sets -MVS Data Set See “Large Scale Multi-view Stereopsis Evaluation“. CVPR 2014. A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos in Unstructured Scenes, TSchöps, J. L. Schönberger, S. Galiani, T. Sattler, K. Schindler, M. Pollefeys, A. Geiger,. CVPR 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction, A. Knapitsch, J. Park, Q.Y. Zhou and V. Koltun. SIGGRAPH 2017.