moonshotai/Kimi-K2.5

Overview

Kimi K2.5 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms.

Prerequisites

• vLLM version: >= 0.15.0 (speculative decoding with Eagle3 requires >= 0.18.0)
• Hardware (BF16): 8x H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)
• Hardware (NVFP4): 4x Blackwell GPUs (e.g. GB200)
• AMD support: 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12

Install vLLM

Pip (NVIDIA):

uv venv
source .venv/bin/activate
uv pip install vllm --torch-backend auto

Pip (AMD ROCm):

uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm

Docker (NVIDIA):

docker pull vllm/vllm-openai:latest

AMD MI300X/MI325X

On 8x MI300X or MI325X (gfx942), use the standard W4A16 MoE path with AITER and INT4 QuickReduce.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.5 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --mm-encoder-tp-mode data

AMD MI350X/MI355X

On 8x MI350X or MI355X (gfx950), add --moe-backend flydsl to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.5 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --moe-backend flydsl \
  --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

Notes:

• The FlyDSL INT4 MoE path does not support expert parallelism; do not add --enable-expert-parallel.
• Keep --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'; it is required for this FlyDSL path on MI350X / MI355X.
• vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
• Keep vLLM's default block size unless you are tuning long-context throughput; --block-size 64 is safe to try.

AMD MI355X (MXFP4 checkpoint)

The MXFP4 variant serves the AMD Quark checkpoint amd/Kimi-K2.5-MXFP4 — a different quantization from the packed-INT4 moonshotai/Kimi-K2.5 FlyDSL path above. It runs the AITER MXFP4 MoE path and adds an fp8 KV cache.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve amd/Kimi-K2.5-MXFP4 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90

Notes:

• fp8 KV cache is the main MI355X win: it doubles resident KV capacity and halves KV read bandwidth. Measured +27% throughput at 8k/1k input with no accuracy change (GSM8K 0.9697, unchanged from bf16 KV).
• AITER RMSNorm stays on. A local TP4 GSM8K check on the current nightly measured 0.9697.
• Old firmware only: if rocm-smi --showfw reports a MEC firmware version below 177, add export HSA_NO_SCRATCH_RECLAIM=1 — older firmware can't reclaim RCCL scratch and vLLM crashes without it. Current MI355X firmware doesn't need it.
• For maximum throughput on fixed-length benchmark 8k/1k or 1k/1k workloads, see the benchmark reproduction section below. These are throughput-sweep tunings; leave vLLM's defaults for general serving.

MXFP4 benchmark reproduction (InferenceX MI355X sweep)

The command above is the default serving configuration. The SemiAnalysis InferenceX MI355X benchmark lane for this checkpoint is a separate high-concurrency benchmark configuration — not the recipe default. To reproduce that sweep, start from the default command above and apply these deltas:

• Use --tensor-parallel-size 4 (the sweep runs TP4; the default recipe keeps TP8 for KV-cache and multimodal-encoder headroom).
• Add --kv-cache-dtype fp8.
• Use the benchmark block size and scheduler knobs: --block-size 16, --max-num-batched-tokens 16384, --max-num-seqs 512, --async-scheduling, and --no-enable-prefix-caching.
• Run on a current vllm/vllm-openai-rocm:nightly that contains the required AITER Kimi MXFP4 backend.
• Export the AITER and INT4 quantized all-reduce env vars before launch:

# Kernel selection and benchmark runtime knobs.
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ROCM_USE_SKINNY_GEMM=0
export VLLM_ROCM_USE_AITER_RMSNORM=0
export AITER_MXFP4_INTERMEDIATE=1
export AITER_BYPASS_TUNE_CONFIG=0
export AITER_MOE_SORT_BACKEND=auto
export OMP_NUM_THREADS=1

The full reproduction serve command is then:

vllm serve amd/Kimi-K2.5-MXFP4 \
  --port "${PORT:-8000}" \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.85 \
  --max-model-len "$MAX_MODEL_LEN" \
  --kv-cache-dtype fp8 \
  --block-size 16 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 512 \
  --async-scheduling \
  --trust-remote-code \
  --no-enable-prefix-caching \
  --mm-encoder-tp-mode data

The benchmark script enforces MAX_MODEL_LEN >= 9472 so the 1k/1k configuration matches the locally validated server. The env vars and scheduler knobs are benchmark-specific throughput settings, not default serving choices.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Troubleshooting

• OOM errors: Lower --gpu-memory-utilization or adjust TP/EP to match your GPU count.
• Vision encoder performance: Use --mm-encoder-tp-mode data to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain.
• Unique multimodal inputs: Pass --mm-processor-cache-gb 0 to avoid caching overhead. For repeated inputs, --mm-processor-cache-type shm uses host shared memory for better performance at high TP settings.
• MoE kernel tuning: Use the benchmark_moe script from vLLM to tune Triton kernels for your specific hardware.
• Async scheduling: Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM.

References

• Kimi-K2.5 on Hugging Face
• NVIDIA Kimi-K2.5-NVFP4 on Hugging Face
• Eagle3 MLA speculative decoding model
• vLLM multimodal inputs guide
• vLLM Expert Parallelism docs
• vLLM NixlConnector usage guide