🔍 iVS3D: intelligent Video Sampler 3D

Max Hermann, Dominik Wüst, Dominic Zahn, Daniel Brommer

@article{DBLP:journals/corr/abs-2110-11810,
  author       = {Max Hermann and Thomas Pollok and Daniel Brommer and Dominic Zahn},
  title        = {{IVS3D:} An Open Source Framework for Intelligent Video Sampling and
                  Preprocessing to Facilitate 3D Reconstruction},
  journal      = {CoRR},
  year         = {2021}
}

📜 Overview

The Intelligent Video Sampler 3D serves as a preprocessing tool to select suitable images primarily for 3D reconstruction. Its pipeline supports importing image sequences or videos, which can be synchronised with metadata, such as GPS poses from EXIF tags or .SRT and .GPX files.

An extensive range of plugins either selects images based on, e.g., visual similarity or sharpness, or creates masks to enhance the provided RGB information further. New algorithms can easily be added by creating new plugins without recompiling the core application.

The resulting image and meta information are exported and can be directly used to start a 3D reconstruction process through the application itself.

overview

🚀 Quick Start

TODO: VIDEO HERE

To get started quickly, download a prebuilt binary from our releases and follow our tutorial.

🔍 Plugins 🧮

There are currently 8 plugins implemented:

Plugin	Description	Supports CUDA
📈 NthFrame	Selects every N-th frame
👓 Blur Detection	Removes images with motion blur	✅
🌐 GeoDistance	Selects images based on the spatial distance between their GPS locations
🌍 GeoMap	Displays an interactive map based on OpenStreetMap to select images using a polygon
🏎 Smooth Camera Movement	Tries to achieve a constant image motion rate by estimating the camera movement using optical flow	✅
🐌 Stationary Camera Removal	Removes redundant images if the camera is not moving	✅
🪞 Deep Visual Similarity	Selects the N-most dissimilar images using K-Means clustering and deep visual embeddings	✅
🤖 Semantic Segmentation	Creates binary masks to exclude objects such as vehicles from the reconstruction by using neural networks for semantic segmentation	✅

These plugins show different approaches to enhance information from an image sequence or video by either selecting images or creating additional masks to improve the 3D reconstruction process. See here for a detailed description of the above mentioned plugins.

iVS3D is built with an open plugin interface for adding new plugins. So feel free to add your own. See here for creating your own plugin.

⛪ 3D Reconstruction

iVS3D does prepare the data for 3D reconstruction. Afterwards, commonly used tools like COLMAP and OpenMVS can be used to perform the reconstruction and meshing. We provide a seamless integration of both in our software. In the Reconstruction tab you can configure desired products such as a dense point cloud or a textured mesh. iVS3D starts the reconstruction with your configuration based on the exported images and masks, either locally on your machine or on a remote server. You can track the progress, view logs and manage the reconstruction jobs all within the Reconstruction tab. For further information see:

📤 Ready to use builds

We provide builds with and without CUDA for multiple platforms and distributions:

OS	CPU only	CUDA enabled
🪟 Windows 11	✅	✅
🐧 Ubuntu 24.04	✅	✅
🐧 Ubuntu 22.04	✅	✅
🐧 Debian 12	✅	✅

Check the latest release to get a build for your platform!

Note that the CUDA builds support RTX series GPUs. Older models or Laptop GPUs might require building iVS3D from source with an OpenCV and CUDA build for that specific GPU.

To use the included plugin for semantic segmentation you can download our deep learning models here: Link to our models

To use other models, they have to be in the .onnx format. In addition, the plug-in requires a file that maps the classes to specific colors.

🔨 Build from source

🔗 Dependencies

iVS3D and the baseline plugins use:

OpenCV 4.11.0 with contrib modules
Qt Framework 5.15.2
FFmpeg latest stable release
Onnxruntime 1.18.0 GPU (if using CUDA, make sure to select the version according to your CUDA version)

For CUDA support:

NVIDIA CUDA Toolkit API 12.0 (or 12.8)
cuDNN 8.9.0 (for CUDA 12.x)

For Windows, we use MSVC 2022 compiler which is shipped with Visual Studio. On Linux, we use GCC 11 compiler.

iVS3D uses the cmake build system and utilizes cmake presets. For detailed instructions on building from source see here:

🧪 Tests

To create the test build add Build_Tests=ON when configuring your build with cmake. Now you can run the tests within the Test Result tab in Qt Creator or use ctest to run the test suite in your terminal.

Link to our test data

🎖️ Acknowledgements

Thanks to everyone that helped bring iVS3D to life.

iVS3D was created as part of the Praxis der Software Entwicklung modul at the Karlsruhe Institue of Technology in the winter term 2020/21, by Patrick Binder, Lennart Ruck, Daniel Brommer, Dominik Wüst and Dominic Zahn. It was build in cooperation with Fraunhofer IOSB, Karlsruhe under supervision of Max Hermann and Thomas Pollok.

Thanks to Boitumelo Ruf for helping with the transition from qmake to cmake.

🔬 Future Work

[x] Add remote colmap execution for windows
[x] Add seemless colmap integration for windows
[ ] Implement metadata reader for gimbal angles
[ ] Create plugin for FOV overlap calculation

📃 Licence

See the [LICENSE](LICENSE) file for details about the license under which this code is made available.