SuperPoint #1603

Dekempsy4 · 2025-10-23T17:52:29Z

Implements the pytorch based Superpoint implementation for keypoint and descriptor extraction:
https://github.com/rpautrat/SuperPoint.

I've also updated the standalone application GUI to include a superpoint rpautrat configuration window with instructions on how to generate the model weights manually.

…ally by the superpoint model

…t at runtime

…ints is lower then the k value (scripting fails in this case)

matlabbe · 2025-10-30T18:02:40Z

corelib/include/rtabmap/core/Features2d.h

 		kFeatureSurfDaisy=14,  //new 0.20.6
-		kFeaturePyDetector=15}; //new 0.20.8
+		kFeaturePyDetector=15, //new 0.20.8
+		kFeatureSuperPointRpautrat=16};


Bump the patch version here

rtabmap/CMakeLists.txt

Line 23 in a307639

SET(RTABMAP_PATCH_VERSION 2)

and add comment with that version here like the other ones.

Bumped the patch and added a comment, (0.23.3)

matlabbe · 2025-10-30T18:06:17Z

corelib/src/python/rtabmap_superpoint.py

These changes seem unrelated to the new approach

I found these changes are necessary to get the old SuperPoint to work. They are unrelated to the focus of this MR so I can remove them if preferred.

matlabbe · 2025-10-30T18:07:02Z

corelib/src/python/rtabmap_superpoint_rpautrat.py

+#
+# Drop this file in the root folder of SuperPoint git: https://github.com/rpautrat/SuperPoint
+# To use with rtabmap:
+#   --Vis/FeatureType 15 --Kp/DetectorStrategy 15 --PyDetector/Path "~/SuperPoint/rtabmap_superpoint_rpautrat.py"


Suggested change

# --Vis/FeatureType 15 --Kp/DetectorStrategy 15 --PyDetector/Path "~/SuperPoint/rtabmap_superpoint_rpautrat.py"

# --Vis/FeatureType 16 --Kp/DetectorStrategy 16 --PyDetector/Path "~/SuperPoint/rtabmap_superpoint_rpautrat.py"

This is the python impl which relies on the detector strategy to be 15 (Pydetector), so I believe that this is correct.

matlabbe · 2025-10-30T18:07:50Z

corelib/src/python/rtabmap_superpoint_rpautrat.py

+    image = np.asarray(imageBuffer)
+    image = (image.astype('float32') / 255.)
+
+    # image_tensor = torch.from_numpy(image[None, None]).float().to(device)


cleanup comments

Whoops, removed the comment.

matlabbe · 2025-10-30T18:33:44Z

corelib/src/superpoint_rpautrat/SuperpointRpautrat.cpp

+    const bool & cuda)
+{
+    // Resolve paths
+    const std::string srcDir = UDirectory::getDir(__FILE__); // folder of this .cpp at build time


That won't work when rtabmap is installed and/or deployed. I think the best would be to include superpoint_to_torchscript.py as a "resource" in rtabmap library. That can be done by appending superpoint_to_torchscript.py to RESOURCES here

rtabmap/corelib/src/CMakeLists.txt

Line 829 in a307639

SET(RESOURCES

(only append if we are building with Torch and python).

The way it works afterwards is that you can include the resource file and decode it into a string:

#include "superpoint_to_torchscript_py.h" [...] std::string file_content = uHex2Str(SUPERPOINT_TO_TORSCRIPT_PY);

Updated to load the file as a resource. I'm generating the loaded .py file in the rtabmap working dir, but it is cleaned up as soon as the model file is generated.

matlabbe · 2025-10-30T18:33:49Z

corelib/src/superpoint_rpautrat/SuperpointRpautrat.cpp

+        << cuda_flag;
+
+    UINFO("Executing: %s", cmd.str().c_str());
+    const int code = std::system(cmd.str().c_str());


Not sure if the windows CI will pass with this, and I would try to stay away from system calls for security reason. Could pybind be used to call the conversion script?

I'm now executing using pybind11 through runpy.

auto runpy = pybind11::module_::import("runpy"); runpy.attr("run_path")(dstScript, pybind11::arg("run_name") = "__main__");

matlabbe · 2025-10-30T18:35:03Z

corelib/src/CMakeLists.txt

 	SET(SRC_FILES
 		${SRC_FILES}
 		superpoint_torch/SuperPoint.cc
+		superpoint_rpautrat/SuperpointRpautrat.cpp


SuperpointRpautrat may be added also only if we build with python too.

Now only add superpoint rpautrat if we have python and torch.

matlabbe · 2025-10-30T18:44:02Z

corelib/include/rtabmap/core/Parameters.h

    RTABMAP_PARAM(SuperPoint, NMSRadius,     int,  4,      uFormat("[%s=true] Minimum distance (pixels) between keypoints.", kSuperPointNMS().c_str()));
    RTABMAP_PARAM(SuperPoint, Cuda,          bool, true,   "Use Cuda device for Torch, otherwise CPU device is used by default.");

+    RTABMAP_PARAM_STR(SuperPointRpautrat, Dir, "",           "[Required] SuperPoint root directory containing scripts and weights.");


I would suggest to use a parameter for the pth file directly: SuperPointRpautrat/modelPath, instead of a directory. See my other comment about which directory to use to output the converted weights.

I've changed it so that we expect the path to the .pth weights file instead of the SuperPoint repo. I originally expected the SuperPoint dir because we need to set the PYTHONPATH there in order for the superpoint_pytorch.py python module import to resolve.

Currently, the code still makes assumptions about the structure of the SuperPoint directory in order to set the PYTHONPATH correctly.

// Add both the weights directory and its parent (repo root) to sys.path // This assumes a certain direcotry structure: /SuperPoint/weights/superpoint_v6_from_tf.pth std::string weightsDir = UDirectory::getDir(weightsPath); std::string repoRoot = UDirectory::getDir(weightsDir); auto sys = pybind11::module_::import("sys"); auto sysPath = sys.attr("path").cast<pybind11::list>(); sysPath.append(repoRoot); sysPath.append(weightsDir);

The most robust solution would be to expect a path to the SuperPoint model class AND the weights file as params. I can implement this if requested.

matlabbe · 2025-10-30T18:57:35Z

corelib/src/Features2d.cpp

+	bool previousCuda = cuda_;
+#endif
+
+	Parameters::parse(parameters, Parameters::kSuperPointRpautratDir(), superpointDir_);


The output file could be generated in the RTABMap's working directory: available by parsing Rtabmap/WorkingDirectory parameter.

superpointDir_ = Parameters::parse(parameters, Parameters::kRtabmapWorkingDirectory(), "");

As suggested above, you may have also to add:

Parameters::parse(parameters, Parameters::kSuperPointRpautratModelPath(), superpointModelPath_);

I'm now forwarding the kRtabmapWorkingDirectory parameter to Features2D and generating the model file there.

matlabbe · 2025-10-30T19:04:42Z

corelib/src/Features2d.cpp

+	else
+	{
+		superPoint_->setSuperpointDir(superpointDir_);
+		superPoint_->setThreshold(threshold_);
+		superPoint_->setNMS(nms_);
+		superPoint_->setMinDistance(minDistance_);
+	}


You may remove these setters because they are not really supported. The model is initialized on the first frame, so any changes here won't matter, the whole model should be re-generated again to use of the new parameters.

It seems that if any parameter changes, we need to delete the previous detector to force it re-initialize on the next frame.

I've removed the setters, we will now regenerate the detector when any parameter changes.

matlabbe · 2025-10-30T19:09:55Z

guilib/src/ui/preferencesDialog.ui

+p, li { white-space: pre-wrap; }
+&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Ubuntu'; font-size:11pt; font-weight:400; font-style:normal;&quot;&gt;
+&lt;p style=&quot; margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;SuperPoint C++ implementation from the &lt;a href=&quot;https://github.com/rpautrat/SuperPoint&quot;&gt;&lt;span style=&quot; text-decoration: underline; color:#0000ff;&quot;&gt;rpautrat/SuperPoint&lt;/span&gt;&lt;/a&gt; project.&lt;/p&gt;
+&lt;p style=&quot; margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;A superpoint_v6_cpp.pt model file will be generated automatically by using the SuperPointRpautrat detector as long as RTAB-Map is built with python support.&lt;/p&gt;


Suggested change

<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">A superpoint_v6_cpp.pt model file will be generated automatically by using the SuperPointRpautrat detector as long as RTAB-Map is built with python support.</p>

<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">A "pt" model file with same name than the "pth" file will be generated automatically in RTAB-Map's working directory so that the model can be called by c++ libtorch directly.</p>

Also: "RTAB-Map should be built with libtorch and python to use this feature."

I updated the info text in the GUI. The MR comment above is updated as well.

matlabbe · 2025-10-30T19:13:26Z

guilib/src/ui/preferencesDialog.ui

+&lt;/style&gt;&lt;/head&gt;&lt;body style=&quot; font-family:'Ubuntu'; font-size:11pt; font-weight:400; font-style:normal;&quot;&gt;
+&lt;p style=&quot; margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;SuperPoint C++ implementation from the &lt;a href=&quot;https://github.com/rpautrat/SuperPoint&quot;&gt;&lt;span style=&quot; text-decoration: underline; color:#0000ff;&quot;&gt;rpautrat/SuperPoint&lt;/span&gt;&lt;/a&gt; project.&lt;/p&gt;
+&lt;p style=&quot; margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;A superpoint_v6_cpp.pt model file will be generated automatically by using the SuperPointRpautrat detector as long as RTAB-Map is built with python support.&lt;/p&gt;
+&lt;p style=&quot; margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;&quot;&gt;To generate the TorchScript model file manually, clone the repository at &lt;a href=&quot;https://github.com/rpautrat/SuperPoint&quot;&gt;&lt;span style=&quot; text-decoration: underline; color:#0000ff;&quot;&gt;https://github.com/rpautrat/SuperPoint.&lt;/span&gt;&lt;/a&gt; Copy the script from &lt;span style=&quot; font-family:'monospace'; background-color:#ffff99;&quot;&gt;corelib/src/superpoint_rpautrat/superpoint_to_torchscript.py&lt;/span&gt; into the SuperPoint repo. Then run the script with the desired parameters, e.g.&lt;br /&gt;&lt;span style=&quot; font-weight:600;&quot;&gt;python superpoint_to_torchscript.py --width WIDTH --height HEIGHT --weights weights/superpoint_v6_from_tf.pth --output weights/superpoint_v6_cpp.pt&lt;/span&gt;&lt;br /&gt;&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>


I would remove this. We won't support the manual way.

matlabbe · 2025-10-30T19:14:35Z

guilib/src/PreferencesDialog.cpp

+	_ui->comboBox_detector_strategy->setItemData(16, 0, Qt::UserRole - 1);
+	_ui->vis_feature_detector->setItemData(16, 0, Qt::UserRole - 1);


#if !defined(RTABMAP_TORCH) || !defined(RTABMAP_PYTHON)

Now checking if RTAB-Map is built with Python and Torch everywhere for SuperPoint rpautrat to be used.

matlabbe · 2025-10-30T19:26:07Z

In the top CMakeLists.txt around here

rtabmap/CMakeLists.txt

Lines 1455 to 1461 in a307639

    
           IF(TORCH_FOUND) 
        
           MESSAGE(STATUS "  With SuperPoint          = YES (License: GPLv3) libtorch=${Torch_VERSION}") 
        
           ELSEIF(NOT WITH_TORCH) 
        
           MESSAGE(STATUS "  With SuperPoint          = NO (WITH_TORCH=OFF)") 
        
           ELSE() 
        
           MESSAGE(STATUS "  With SuperPoint          = NO (libtorch not found)") 
        
           ENDIF()

Can you add a section for SuperPoint Rpautrat? Maybe like:

IF(TORCH_FOUND AND WITH_PYTHON AND Python3_FOUND) 
 MESSAGE(STATUS "  With SuperPoint Rpautrat = YES (License: MIT)") 
 ELSEIF(NOT WITH_TORCH) 
 MESSAGE(STATUS "  With SuperPoint Rpautrat = NO (WITH_TORCH=OFF)") 
 ELSEIF(NOT WITH_PYTHON) 
 MESSAGE(STATUS "  With SuperPoint Rpautrat = NO (WITH_PYTHON=OFF)") 
 ELSE() 
 MESSAGE(STATUS "  With SuperPoint Rpautrat = NO (libtorch and/or python3 not found)") 
 ENDIF()

matlabbe · 2025-10-30T19:32:53Z

Similarly to previous comment: the about dialog could also be updated to show if rtabmap is built with superpoint rpautrat

For Parameters.cpp, we can also add the status to know if rtabmap is built with tht feature:

rtabmap/corelib/src/Parameters.cpp

Lines 659 to 664 in a307639

    
           				str = "With SuperPoint Torch:"; 
        
           #ifdef RTABMAP_TORCH 
        
           				std::cout << str << std::setw(spacing - str.size()) << "true" << std::endl; 
        
           #else 
        
           				std::cout << str << std::setw(spacing - str.size()) << "false" << std::endl; 
        
           #endif

 				str = "With SuperPoint Rpautrat:"; 
 #if defined(RTABMAP_TORCH) and defined(RTABMAP_PYTHON) 
 				std::cout << str << std::setw(spacing - str.size()) << "true" << std::endl; 
 #else 
 				std::cout << str << std::setw(spacing - str.size()) << "false" << std::endl; 
 #endif

…t from source

…cannot support changing params after the model is constructed

Dekempsy4 · 2025-10-31T20:03:22Z

@matlabbe I cannot add comments on the last two threads for some reason, but I addressed both of them. This MR should be ready for another round of review when you have time.

Felix Toft added 24 commits October 23, 2025 10:48

initial python implementation of superpoint rpautrat

96cd52e

working python implementation of superpoint

1b4ae82

small tweaks to try and work around the GIL issue

c8e5fce

fix missing os import

b95d82a

begging cpp impl of superpoint python model

7316bbe

finishing cpp superpoint impl using the same SPDetector interface

49c4ec6

finalizing superpoint cpp impl, working with cpu but needs to be cleaned

28d6be3

fixing feature matching by reworking nms and filtering logic

61ababe

speeding up nms with batched operations and cleaning up

65eeffd

updating conversion script

1509904

adding args for image dimensions and cuda usage to model tracer

76fc0b7

wiring up UI to parse superpoint params

5a4af7a

adding label to superpoint rpautrat ui

8ea0de5

reverting some unintended ui changes

3349f02

typo

c4e9006

oneline revert

0a24816

removing nms and threshold filtering from cpp, this is handled intern…

c007525

…ally by the superpoint model

using python interface to run the superpoint _to_torchscript.py scrip…

aa2ac72

…t at runtime

generate and load model file on the first incoming frame

3b97f44

reverting unintented change

5f1d8bd

ui changes appear mysteriously again, reverting

6e89e3a

remove topk from model to prevent issue when number of detected keypo…

f1bccf0

…ints is lower then the k value (scripting fails in this case)

cleaning up for review

7c8a326

change dest for model file and remove debug logs

19829b7

matlabbe reviewed Oct 30, 2025

View reviewed changes

bump patch and add version comment

54c4602

matlabbe reviewed Oct 30, 2025

View reviewed changes

rm unucessary comment

e2b83d8

matlabbe reviewed Oct 30, 2025

View reviewed changes

Felix Toft added 10 commits October 30, 2025 13:18

use resources to load file to ensure it works when rtabmap isn't buil…

adfcf2a

…t from source

execute with pybind runpy instead of system call

142c797

only build superpoint rpautrat if we have torch and python support

24e4427

changing param to accept a path to the weights .pth file directly

213ba1b

parse the default working directory to save the model file in

3e2dfe7

rm extraneous change

a5ddd82

rm setters and re-initialize the detector whenever params change. We …

e9359dc

…cannot support changing params after the model is constructed

update UI text

04e08ec

only enable superpoint rpautrat when built with python and torch

fe0de7e

more build information regarding superpoint rpautrat

bfa574d

generate temporary python script in the working dir

83d5b0b

	# --Vis/FeatureType 15 --Kp/DetectorStrategy 15 --PyDetector/Path "~/SuperPoint/rtabmap_superpoint_rpautrat.py"
	# --Vis/FeatureType 16 --Kp/DetectorStrategy 16 --PyDetector/Path "~/SuperPoint/rtabmap_superpoint_rpautrat.py"

	<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">A superpoint_v6_cpp.pt model file will be generated automatically by using the SuperPointRpautrat detector as long as RTAB-Map is built with python support.</p>
	<p style=" margin-top:12px; margin-bottom:12px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">A "pt" model file with same name than the "pth" file will be generated automatically in RTAB-Map's working directory so that the model can be called by c++ libtorch directly.</p>

		_ui->comboBox_detector_strategy->setItemData(16, 0, Qt::UserRole - 1);
		_ui->vis_feature_detector->setItemData(16, 0, Qt::UserRole - 1);

SuperPoint #1603

Are you sure you want to change the base?

SuperPoint #1603

Uh oh!

Conversation

Dekempsy4 commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matlabbe Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matlabbe Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matlabbe Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matlabbe commented Oct 30, 2025

Uh oh!

matlabbe commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dekempsy4 commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Dekempsy4 commented Oct 23, 2025 •

edited

Loading

matlabbe Oct 30, 2025 •

edited

Loading

matlabbe Oct 30, 2025 •

edited

Loading

matlabbe Oct 30, 2025 •

edited

Loading

matlabbe commented Oct 30, 2025 •

edited

Loading