[Feature]: EuRoC evaluation harness prototype for kornia-slam benchmarking
#840 opened on Mar 22, 2026
Repository metrics
- Stars
- (675 stars)
- PR merge metrics
- (PR metrics pending)
Description
🚀 Feature Description
Add a backend-agnostic EuRoC/TUM-VI evaluation harness to kornia-slam for benchmarking VI-SLAM systems with ATE and RPE metrics, visualized via Rerun.
📂 Feature Category
Rust Core Library
💡 Motivation
The kornia-slam GSoC 2026 project requires rigorous evaluation across EuRoC and TUM-VI datasets with published ATE/RPE results. Currently kornia-slam has no evaluation utilities — every contributor and user needs to build their own parser and metric computation from scratch.
This harness solves that by providing a ready-to-use, backend-agnostic evaluation pipeline that any SLAM system can plug into. It works with ROS2 bag files directly (no ROS installation needed) and visualizes results in Rerun for easy inspection.
Related to GSoC 2026 project: kornia-slam robust visual-inertial SLAM.
💭 Proposed Solution
Add examples/slam_eval/ to kornia-rs with the following:
-
EurocLoader — parses ROS2 bag files (.db3) and extracts:
- Stereo images (cam0/cam1)
- IMU samples at 200Hz (gyroscope + accelerometer)
- Ground truth positions from /leica/position topic
-
TrajectoryEvaluator — computes standard SLAM metrics:
- ATE (Absolute Trajectory Error) after Umeyama SE(3) alignment
- RPE (Relative Pose Error) over fixed frame intervals
-
RerunVisualizer — logs to Rerun:
- 3D trajectory (ground truth vs estimated)
- Live camera frames from cam0
- IMU accel and gyro signal norms over time
Prototype results on MH_01_easy:
ATE RMSE: 0.0865 m ATE mean: 0.0792 m ATE median: 0.0763 m ATE max: 0.2019 m RPE trans RMSE: 0.1233 m
Prototype branch: https://github.com/NeNish/kornia-rs/tree/slam-eval-euroc
The evaluator is fully backend-agnostic, plug in any list of estimated poses from a SLAM system and get ATE/RPE results instantly. Designed to extend to TUM-VI with minimal changes.
📚 Library Reference
-
EuRoC MAV Dataset: Burri et al., "The EuRoC micro aerial vehicle datasets", International Journal of Robotics Research, 2016. https://doi.org/10.1177/0278364915620033
-
Umeyama SE(3) alignment for ATE: S. Umeyama, "Least-squares estimation of transformation parameters between two point patterns", IEEE TPAMI, 1991.
-
ATE/RPE metric definitions: Sturm et al., "A Benchmark for the Evaluation of RGB-D SLAM Systems", IROS 2012.
-
rosbags (pure Python ROS2 bag reader): https://github.com/rpng/rosbags
-
Rerun visualization SDK: https://rerun.io
-
kornia-rs Python bindings for image I/O: https://github.com/kornia/kornia-rs
🔄 Alternatives Considered
-
evo library (https://github.com/MichaelGrupp/evo) — the standard SLAM evaluation tool. Considered using it directly but it adds a heavy dependency and doesn't integrate with Rerun for visualization. Our implementation keeps ATE/RPE computation self-contained and lightweight.
-
ASL folder format instead of ROS2 bags — the original EuRoC dataset was available as CSV + images. Chose ROS2 bag format instead since it is the current distribution format from ETH Research Collection and aligns better with how kornia-slam will consume sensor data at runtime.
-
Full ROS2 installation — considered using rclpy to read bags but this requires a full ROS2 install which is a major barrier on Windows and macOS. rosbags (pure Python) removes this dependency entirely.
🎯 Use Cases
-
GSoC VI-SLAM development — as the kornia-slam pipeline is built during GSoC, every iteration of the odometry, mapping, and relocalization modules can be benchmarked instantly by piping estimated poses into TrajectoryEvaluator.
-
Regression testing — contributors can run the harness on MH_01_easy after any change to kornia-slam to check whether ATE/RPE improved or degraded.
-
Dataset-wide benchmarking — extend to all 11 EuRoC sequences (MH_01 through MH_05, V1_01 through V2_03) and publish a full benchmark table comparing kornia-slam against state-of-the-art systems like ORB-SLAM3 and VINS-Mono.
-
Stress testing — use the harness on challenging sequences (V2_03_difficult, fast motion, low texture) to identify where the SLAM system degrades and guide robustness improvements.
-
Onboarding new contributors — anyone wanting to test their own SLAM modifications against ground truth can use this harness without needing to understand ROS internals or implement their own metric computation.
📝 Additional Context
No response
🤝 Contribution Intent
- I plan to submit a PR to implement this feature
- I'm requesting this feature but not planning to implement it