NPUgood first issuehelp wantedhigh priority
Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Motivation.
- Performance status:
- Hardware: XX
- Scenario: 2 card, 1024 x 1024, 50 steps, t2i
- Performance: ---> ar 101.6s + dit 36.8 ≈ e2e 143s ---> ar overlay graph mode 31.8s + dit overlay cache 11s ≈ e2e 47.8s ---> ar overlay graph mode + asynchronous scheduling 25.8s + dit overlay cache 11s ≈ e2e 42s
- Currently, there are many optimization PRs for the DIT model, such as cache, SP, quantization, etc. However, the AR module of this model consumes a relatively high proportion of time. This time, we are more inclined to focus on optimizing the AR module to the extreme. At the same time, we have identified several issues: ① Compared to the pure model, the AR module consumes more time than the pure model; ② e2e-ar-dit ≈ 5s, and the reason for this loss needs to be analyzed.
Proposed Change.
TODOLIST：
1.Problem solution:
- Compared to the pure model, the AR module consumes more time than the pure model;
- e2e-ar-dit ≈ 5s, and the reason for this loss needs to be analyzed. @JaredforReal
2.Scheduling optimization
- Streaming inference and asynchronous chunking are supported.
3.AR module optimization.---To be analyzed (In the AR phase, Matmul accounts for 70% of the workload, so we believe that quantization of MM should be prioritized. However, we are currently unsure how to integrate quantization features during the AR phase of vllm-omni, and we need assistance.)
- MM quantization is supported.
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.