good first issue
Repository metrics
- Stars
- (28,442 stars)
- PR merge metrics
- (Avg merge 2d 1h) (1,000 merged PRs in 30d)
Description
Checklist
- 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
- 2. Please use English, otherwise it will be closed.
Motivation
This might be difficult to implement, but I am facing the following issue: When running Qwen2-VL on bigger images, the preprocessor takes a long time to convert the images to tokens.
It would be awesome if we could have a way (OpenAI API with extra parameters) to tell the backend to store the cache of a request and load it by ID for another request, which would make it possible to not reprocess every image (and prompt in general) on each call.
If my problem could be solved in an easier way I would be thankful for any input :)
Related resources
No response