Repository metrics
- Stars
- (2,873 stars)
- PR merge metrics
- (PR metrics pending)
Description
Epic domain
Currently, our inferencing stack primarily supports LlamaCPP for Mac and Windows environments. To enhance our model execution capabilities, ensure consistency between local and production environments, and drive adoption among application developers, we aim to extend our support to vLLM.
Objective:
- Enable vLLM as an alternative inferencing stack alongside LlamaCPP.
- Ensure consistency between local and production environments.
- Promote adoption and usage of vLLM by application developers.
Scope
1. CPU only
In a first approach, we should be providing all the images for supporting vLLM running with CPU for Mac and Windows environments. cf: https://github.com/containers/ramalama/issues/801
2. Research & Feasibility Analysis for GPU support
In order to provide inferecing speed and efficiency, we should be supporting GPU-backed execution for VLLM on Mac/Windows. We'll need to dig into the ways vLLM is executed and analyse the effort required to support backends that can offload operations to the GPU
3. Benchmark
We should be tooling ourselves in order to conduct benchmarking and performance tests as part of our CI. We would in particular, be interested by comparing:
- native, vs container
- llamacpp, vs vLLM
Sub-tasks
- Provide CPU only images for Mac
- Provide CPU only images for Windows
- Research analysis for GPU support Mac
- Benchmarks suite for vLLM
Additional context
No response