vs vLLM deterministic
Up to 2.7x faster
1.5 to 2.7x faster with 16 to 50% less GPU energy per token than vLLM batch-invariant generation.
For inference providers and teams serving LLMs
LLM serving with stable answers, at normal speed.
Bit-identical answers at every batch size while matching default vLLM speed on an H100 (4,898 vs 4,875 tok/s at batch 64). Normal vLLM changed its answer in 12 of 140 test conditions. Luxi changed in 0 of 80.
Measured
Faster than the deterministic modes.
vs SGLang deterministic
Up to 2.86x faster
Generation 1.21 to 2.86x faster; prompts 1.18 to 1.24x faster with 14 to 19% less GPU energy per token.
vs normal vLLM
Same speed, stable answers
Prompts 3 to 7% faster with 2 to 5% less GPU energy per token. Normal generation ran at the same speed with 2 to 7% more GPU energy per token.
Measured on Qwen2-7B-Instruct, H100, offline fixed-batch tests. GPU-board energy from NVML.
Talk to us
Bring us the computation where a changing answer costs you.
Email eric@luxiedge.com or use the form on the contact page.