ggml-org/llama.cpp

ggml : add DirectML backend

Open

#7,772 opened on Jun 5, 2024

View on GitHub
 (20 comments) (40 reactions) (0 assignees)C++ (18,202 forks)batch import
help wantedresearch 🔬roadmap

Repository metrics

Stars
 (110,169 stars)
PR merge metrics
 (Avg merge 6d 8h) (389 merged PRs in 30d)

Description

It seems like DirectML supports the upcoming NPU-enabled chips for Windows machines: https://devblogs.microsoft.com/directx/introducing-neural-processor-unit-npu-support-in-directml-developer-preview/

I don't think there is any other way to tap into this hardware, so we should explore if it possible to add this library as a backend in ggml in order to run stuff on the NPUs. There has been some semi-related work in the past that combined ggml and Direct3D: https://github.com/Const-me/Whisper. Not sure if it is relevant at all, maybe just as an inspiration

Contributor guide

Research direction
Study DirectML SDK and existing Direct3D integration in ggml (Const me/Whisper) to design a backend architecture.
Tech stack
cpp
Domain
backendai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Mostly clear
Prerequisites
C++Machine Learning
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20