Voxel Carving of 3D Surfaces using AR Markers for Pose Estimation

Final project for "3D Scanning and Motion Capture" class in SS2022 at Technical University of Munich.

Table of Contents

• 1. Abstract
• 2. Setup
  • 2.1. Clone the repository
  • 2.2. Windows
    • 2.2.1. CMake
    • 2.2.2. Git BASH
    • 2.2.3. Eigen
    • 2.2.4. OpenCV
  • 2.3. Other OS
  • 2.4. Build the Project
• 3. Example
• 4. How to run?
  • 4.1. Generate ChArUco Board
  • 4.2. Camera Calibration
  • 4.3. Segmentation
    • 4.3.1. Image Segmentation using Deep Neural Networks & Python
  • 4.4. Carve your own model
  • 4.5. Benchmarking
• References

1. Abstract

Voxel carving is a known 3D reconstruction method for object reconstruction from a set of RGB images. We present our implementation of this method with the goal of achieving a good estimation of a human model placed in a scene. For pre-processing, we used pose estimation using the ChArUco markers set up in our scene, as well as different image segmentation methods. The voxel carving method is then used to extract a voxel representation of the mesh. Finally, post-processing is applied to both color our model according to the input, as well as improve the topology to achieve a convincing mesh.

Our complete written report can be found here. Additionaly, this Video provides a great overview of this project.

2. Setup

2.1. Clone the repository

Repository Structure

• Data

  • demo_material

    • images

    • masks

• Libs

• src

• CMakeLists.txt

• README.adoc

Data

You can use Data to store images, masks, and calibration files of models you want to carve. The subfolder demo_material contains all inputs and outputs used to generate the model as described in section 4.

Libs

If you have never worked with OpenCV or Eigen you can follow the instructions below and install these libraries in the Libs directory.

src

This directory contains the implementation of this project.

2.2. Windows

2.2.1. CMake

1. Download CMake (3.9.1 or newer)

2. Install CMake in your preferred directory.

3. Add CMake to PATH variable.

2.2.2. Git BASH

1. Download and install Git Bash (2.14.1 or newer)

2.2.3. Eigen

Download and extract Eigen (see Eigen - Getting started).

2.2.4. OpenCV

1. Navigate to the directory where you wish to install OpenCV (eg. {directory}/AR_Voxel_Project/Libs)

2. create new file installOCV.sh with the following content:

   #!/bin/bash -e
   
   # taken from: https://docs.opencv.org/3.4/d3/d52/tutorial_windows_install.html
   
   myRepo=$(pwd)
   CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 17 2022"
   #CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 16 2019"
   #CMAKE_GENERATOR_OPTIONS=-G"Visual Studio 15 2017 Win64"
   #CMAKE_GENERATOR_OPTIONS=(-G"Visual Studio 16 2019" -A x64)  # CMake 3.14+ is required
   if [  ! -d "$myRepo/opencv"  ]; then
       echo "cloning opencv"
       git clone https://github.com/opencv/opencv.git
   else
       cd opencv
       git pull --rebase
       cd ..
   fi
   if [  ! -d "$myRepo/opencv_contrib"  ]; then
       echo "cloning opencv_contrib"
       git clone https://github.com/opencv/opencv_contrib.git
   else
       cd opencv_contrib
       git pull --rebase
       cd ..
   fi
   RepoSource=opencv
   mkdir -p build_opencv
   pushd build_opencv
   CMAKE_OPTIONS=(-DBUILD_PERF_TESTS:BOOL=OFF -DBUILD_TESTS:BOOL=OFF -DBUILD_DOCS:BOOL=OFF  -DWITH_CUDA:BOOL=OFF -DBUILD_EXAMPLES:BOOL=OFF -DINSTALL_CREATE_DISTRIB=ON)
   set -x
   cmake "${CMAKE_GENERATOR_OPTIONS[@]}" "${CMAKE_OPTIONS[@]}" -DOPENCV_EXTRA_MODULES_PATH="$myRepo"/opencv_contrib/modules -DCMAKE_INSTALL_PREFIX="$myRepo/install/$RepoSource" "$myRepo/$RepoSource"
   echo "************************* $Source_DIR -->debug"
   cmake --build .  --config debug
   echo "************************* $Source_DIR -->release"
   cmake --build .  --config release
   cmake --build .  --target install --config release
   cmake --build .  --target install --config debug
   popd

3. Adjust the CMAKE_GENERATOR_OPTIONS according to your Visual Studio version.

4. In git command line enter the following command:

   ./installOCV.sh

   This script will generate the required installation of Open CV in the current directory (eg. {directory}/AR_Voxel_Project/Libs/install/opencv).

5. Keep in mind that this will take some time.

6. Add {directory}\Libs\install\opencv\bin to PATH.

7. For further information please check the official OpenCV website.

2.3. Other OS

Please refer to the following instructions:

• Eigen: All OS

• Open CV: Linux MacOS

2.4. Build the Project

After installing both libraries, your repository should look like this.

Repository Structure

• Data

  • demo_material

    • images

    • masks

• Libs

  • Eigen3

  • opencv

  • install

    • opencv

  • opencv_contrib

• src

• CMakeLists.txt

• README.adoc

Now you are ready to generate the project using CMake. If you followed the instructions above, CMake should already have found the installation directories of Eigen and OpenCV. Make sure that Eigen3_DIR and OpenCV_DIR are set correctly.

After compiling the project, you are ready to carve your first model.

3. Example

The following example gives visualize the carved model of our demo object. If you want to reproduce our results, you can use this dataset. For comparison we also provide details about the configuration and the expected output meshes.

Figure 1. Example Input Image (total of 8)

Figure 2. Generated Model: post processing and color reconstruction (Method 2 - color averaging) applied.

4. How to run?

4.1. Generate ChArUco Board

Run

$ ./voxel_project.exe -c=1

This command will generate a new subdirectory out containing the file BoardImage.jpg. This ChArUco Board can be used to calibrate your individual camera and later on carve your own models.

4.2. Camera Calibration

Run

$ ./voxel_project.exe -c=2

Perform camera calibration on previously taken images or take images using a connected camera and perform calibration on those.

After initial calibration, the user can interactively choose to exclude specific images by ID (e.g. outliers). For this, reprojection errors are provided for individual images, as well as the overall error.

The final data is saved to a .yml file that can be specified using -calibration

Flag	Default	Description
-live=true/false	true	Whether the images used for calibration should be taken live using a connected camera. If false, a folder with images for calibration has to be provided to -images.
-cam_id=<cam_id>	0	The camera to use when performing live calibration. 0 should be the system’s default device.
-images="<images-dir>"	Ignored if live==true NonOptional if live==false	Directory of the input images to be used for calibration e.g. ./Data/demo_material/images.
-calibration="<calibration-file>"	out/cameracalibration.yml	Output file for the camera calibration data.

4.3. Segmentation

Run

$ ./voxel_project.exe -c=3

This option provides you with two methods for semantic segmentation: Color segmentation and k-means segmentation. Kmeans image segmentation does not require you to set any hyperparameters. Color segmentation requires you to set scene specific color masks in Segmentation.h of the style:

$ cv::Mat mask;
$ inRange(rgb_img, cv::Scalar(120, 120, 120), cv::Scalar(255, 255, 255), mask);

Multiple masks can be concatenated mask1 + mask2 and masks can be inverted ~mask.

4.3.1. Image Segmentation using Deep Neural Networks & Python

Additionally, it is possible to use the pretrained Mask R-CNN implementation of Pytorch to perform image segmentation. Please, refer to capture_and_segment_images.py. You need to give the dataset you capture a unique name (--dataset_name) and set the id of your camera (--video_id).

4.4. Carve your own model

Run

$ ./voxel_project.exe -c=5 -images="<images-dir>" -masks="<masks-dir>" -calibration="<cameracalibartion.yml-dir>" -carve=<carving-method> -x=<x-dim> -y=<y-dim> -z=<z-dim> -size=<voxel-size> -scale=<model-scale> -dx=<x-offset> -dy=<y-offset> -dz=<z-offset> -color=<color-method> -model_debug=<model_debug-method> -postprocessing=<postprocessing-method> -intermediateMesh=<intermediateMesh-generation> -outFile=<out_file_path>

This command will generate a new file out/mesh.off containing the mesh generated by carving your specified inputs. To understand more about the flags please refer to the table below.

Table 1. Flags

Flag	Default	Description
-images="<images-dir>"	NonOptional	Directory of the input images e.g. ./Data/box_dataset/images.
-masks="<masks-dir>"	NonOptional	Directory of the masks e.g. ./Data/box_dataset/masks.
-calibration="<cameracalibartion.yml-dir>"	NonOptional	Directory of the masks e.g. ./Data/box_dataset/cameracalibration.yml.
-carve=<carving-method>	1	1 - standard carving 2 - greedy carving (fast)
-x=<x-dim>	100	Number of voxels in x direction.
-y=<y-dim>	100	Number of voxels in y direction.
-z=<z-dim>	100	Number of voxels in z direction.
-size=<voxel-size>	0.0028	Side length of a voxel.
-scale=<model-scale>	1.0	Scale factor for the output model.
-dx=<x-offset>	0.0	Move output model in x direction (unscaled).
-dx=<y-offset>	0.0	Move output model in y direction (unscaled).
-dx=<z-offset>	0.0	Move output model in z direction (unscaled).
-color=<color-method>	0	0 - no color reconstruction 1 - nearest observer 2 - average color
-model_debug=<model_debug-method>	false	true - generate debug model (1 cube ~ 1 visible voxel) false - do not generate debug model
-postprocessing=<postprocessing-method>	true	true - apply postprocessing on the model false - do not apply postprocessing
-intermediateMesh=<intermediateMesh-generation>	false	true - Generates a mesh after each processed image. Only works with carving method 1 false - intermediate mesh generation disabled
-outFile=<out_file_path>	./out/mesh.off	Filepath the generated mesh will be written to. Should end with .off.

4.5. Benchmarking

Run

$ ./voxel_project.exe -c=6 -images="<images-dir>" -masks="<masks-dir>" -calibration="<cameracalibartion.yml-dir>"

This command will execute the provided benchmarking cases on the specified data. For each case a mesh file will be generated in out/bench. Additionally, the bash will print a table containing information about the test cases and execution times (see table below). For more information about the flags please refer to the table above.

Table 2. Benchmark output

Column	Description
Name	Short description of the testcase Model size (Small, Medium, Large) Carving method (1 - standard, 2 - greedy) Coloring method
Model size	Model dimensions (x, y, z direction) and voxel size
Carving time	Time needed to execute carving process (in milliseconds)
Coloring time	Time needed to execute coloring process (in milliseconds)
Postprocessing time	Time needed for postprocessing (in milliseconds)
Marching cubes time	Time needed to transform model into .off-format sutiable representation using the marching cubes algorithm (in milliseconds)
Overall time	Time needed for complete reconstruction process (including output file writing and variable initializations, excluding image loading) (in milliseconds)

References

• [1] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision, 38(3):199–218, 2000.

• [2] Johannes Lutz Sch ̈onberger and Jan-Michael Frahm. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

• [3] Alexander Schick and Rainer Stiefelhagen. Real-time gpu-based voxel carving with system- atic occlusion handling. In Joachim Denzler, Gunther Notni, and Herbert S ̈uße, editors, Pattern Recognition, pages 372–381, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg.

• [4] G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000.

• [5] H. Kato and M. Billinghurst. Marker tracking and hmd calibration for a video-based aug- mented reality conferencing system. In Proceedings 2nd IEEE and ACM International Work- shop on Augmented Reality (IWAR’99), pages 85–94, 1999.

• [6] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.

• [7] Kiriakos N. Kutulakos and Steven M. Seitz. A theory of shape by space carving. Interna- tional Journal of Computer Vision, 38(3):199–218, 2000.

• [8] Mathieu Gaillard, Chenyong Miao, James C Schnable, and Bedrich Benes. Voxel carving- based 3d reconstruction of sorghum identifies genetic determinants of light interception efficiency. Plant direct, 4(10):e00255, 2020.

• [9] William E. Lorensen and Harvey E. Cline. Marching cubes: a high resolution 3d surface construction algorithm. Seminal graphics, 1996.

• [10] Felix Wimbauer, Nan Yang, Lukas Von Stumberg, Niclas Zeller, and Daniel Cremers. Monorec: Semi-supervised dense reconstruction in dynamic environments from a single moving camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6112–6122, 2021.

• [11] Adrian Broadhurst, Tom W Drummond, and Roberto Cipolla. A probabilistic framework for space carving. In Proceedings eighth IEEE international conference on computer vision. ICCV 2001, volume 1, pages 388–393. IEEE, 2001