ggml-org/llama.cpp

ggml : add GPU support for Mamba models

Open

#6,758 opened on Apr 19, 2024

View on GitHub
 (39 comments) (32 reactions) (0 assignees)C++ (18,202 forks)batch import
Nvidia GPUenhancementhelp wantedroadmap

Repository metrics

Stars
 (110,169 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

Recently, initial Mamba support (CPU-only) has been introduced in #5328 by @compilade

In order to support running these models efficiently on the GPU, we seem to be lacking kernel implementations for the following 2 ops:

  • GGML_OP_SSM_CONV
  • GGML_OP_SSM_SCAN

Creating this issue to keep track of this and give more visibility of this feature. Help with implementing the missing kernels for CUDA and Metal (and other backends potentially) is welcome. We can also discuss if anything else is required to better support this architecture in llama.cpp

Contributor guide

Research direction
Study the existing CPU implementations of GGML OP SSM CONV and GGML OP SSM SCAN in the source code, then implement equivalent CUDA kernels (and Metal if possible). Reference similar GPU kernel implementations in the repository for patterns.
Tech stack
cpp
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
C++GPU programming
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20