[Feature] Request to 8-bit Quantization of Attention with SageAttention · sgl-project/sglang#1763

(8 comments) (0 reactions) (0 assignees)Python (6,216 forks)auto 404

good first issue

Repository metrics

Stars: (28,442 stars)
PR merge metrics: (Avg merge 2d 1h) (1,000 merged PRs in 30d)

Description

Checklist

1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
2. Please use English, otherwise it will be closed.

Motivation

As https://github.com/thu-ml/SageAttention mentioned, the quantized 8-bit attention will improvement the speed of inference about 2x and more with the same accuracy, so shall we give it a try or do some verification?

Related resources

github: https://github.com/thu-ml/SageAttention

Contributor guide

Research direction: Investigate the SageAttention repository and understand how to integrate its 8 bit attention quantization into the existing attention modules in sglang. Consider performance benchmarks and accuracy verification.
Tech stack: python
Domain: backend
Issue type: Feature
Difficulty: 3
Estimated time: 1-2 days
Activity status: Active
Clarity: Clear
Prerequisites: Python
Newbie friendliness: 60