[Bug] Kimi VL GPU memory usage too high

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The GPU memory usage when serving Kimi VL is too high.

Reproduction

1. First monitor GPU memory usage using nvitop
2. Launch server with python -m sglang.launch_server --model-path moonshotai/Kimi-VL-A3B-Thinking-2506 --trust-remote-code --reasoning-parser kimi --mem-fraction-static 0.5 The usage should be around 50%.
3. Run python benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 8 in sglang.
4. Continue monitoring the memory usage, it would up to 98%, and sometimes cause OOM

Environment

Python: 3.10.12 (main, May 27 2025, 17:12:29) [GCC 11.4.0] CUDA available: True GPU 0,1,2,3: NVIDIA H100 80GB HBM3 GPU 0,1,2,3 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 12.4, V12.4.131 CUDA Driver Version: 550.127.05 PyTorch: 2.7.1+cu126 sglang: 0.4.7.post1 sgl_kernel: 0.1.9 flashinfer_python: 0.2.6.post1 triton: 3.3.1 transformers: 4.52.3 torchao: 0.9.0 numpy: 2.2.5 aiohttp: 3.11.18 fastapi: 0.115.12 hf_transfer: 0.1.9 huggingface_hub: 0.30.2 interegular: 0.3.3 modelscope: 1.25.0 orjson: 3.10.16 outlines: 0.1.11 packaging: 25.0 psutil: 7.0.0 pydantic: 2.11.3 python-multipart: 0.0.20 pyzmq: 26.4.0 uvicorn: 0.34.2 uvloop: 0.21.0 vllm: Module Not Found xgrammar: 0.1.19 openai: 1.76.0 tiktoken: 0.9.0 anthropic: 0.50.0 litellm: 1.67.4.post1 decord: 0.6.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 48-95,144-191 1 N/A

Legend:

X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks

Hypervisor vendor: KVM ulimit soft: 1048576