kornia/kornia-rs

[Feature]: Add ONNXRuntime-based Vision-Language Model (VLM) inference example to kornia-vlm

Open

#634 opened on Jan 10, 2026

View on GitHub
 (10 comments) (0 reactions) (1 assignee)Rust (188 forks)auto 404
enhancementhelp wantedtriage

Repository metrics

Stars
 (675 stars)
PR merge metrics
 (PR metrics pending)

Description

🚀 Feature Description

Kornia is introducing state-of-the-art Vision-Language Models (VLMs).

This feature proposes adding a minimal, well-documented ONNXRuntime-based VLM inference example to kornia-vlm, demonstrating how exported ONNX models can be executed efficiently in Rust without relying on heavyweight training frameworks.

The example will focus on:

  • Loading an ONNX VLM model via ONNXRuntime
  • Image preprocessing using existing Kornia primitives
  • Running inference in Rust
  • Clear documentation for validation, testing, and edge deployment use cases

📂 Feature Category

Rust Core Library

💡 Motivation

Current Rust-based VLM workflows often rely on training-focused libraries like Candle, which are not ideal for:

  • Inference-only deployments
  • Edge or embedded environments
  • Reproducible validation and benchmarking

ONNXRuntime is a widely adopted, production-grade inference backend with strong cross-platform support.

Providing an official ONNXRuntime-based example in kornia-vlm would:

  • Lower the barrier for users evaluating VLMs in Rust
  • Enable consistent validation and testing of exported models
  • Serve as a foundation for edge deployment and CI-based model verification

This aligns directly with ongoing efforts to integrate SOTA VLMs into Kornia.

💭 Proposed Solution

The proposal is to add a self-contained example under kornia-vlm with the following structure:

  1. ONNXRuntime integration

    • Use the onnxruntime Rust crate as an optional dependency
    • Keep the integration minimal and inference-only
  2. Model loading

    • Load a pre-exported ONNX VLM model
    • No training or conversion logic included
  3. Preprocessing

    • Use existing Kornia image utilities for resizing, normalization, and tensor conversion
    • Avoid re-implementing preprocessing logic
  4. Inference

    • Execute a forward pass via ONNXRuntime
    • Return raw outputs suitable for downstream evaluation
  5. Documentation

    • Explain setup, model requirements, and execution steps
    • Clearly state that this is a reference example
  6. Testing

    • Include a lightweight sanity test to validate model execution

📚 Library Reference

  • ONNXRuntime: production-grade inference engine with Rust bindings
  • onnxruntime Rust crate
  • Existing Kornia image preprocessing utilities
  • Exported ONNX VLM models

🔄 Alternatives Considered

  • Candle-based inference: powerful for training but heavier than necessary for inference-only and deployment scenarios
  • Custom inference engine: high maintenance cost and out of scope for an example

ONNXRuntime provides the best balance between performance, portability, and ecosystem adoption.

🎯 Use Cases

  • Validating exported VLM ONNX models
  • Benchmarking inference performance
  • Edge and embedded deployments
  • CI-based regression testing for model outputs
  • Educational reference for Rust-based VLM inference

📝 Additional Context

This feature is intended to be incremental, reviewable, and aligned with existing Kornia design patterns.

If accepted, the implementation can be expanded in the future to support additional backends or models, while keeping the initial scope minimal and maintainable.

🤝 Contribution Intent

  • I plan to submit a PR to implement this feature
  • I'm requesting this feature but not planning to implement it

Contributor guide