DEV Community

#vllm

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Six Serving Engines on Four Accelerators: Gemma 4 E2B on an RTX 3050, a T4, an L4 and a TPU v5e
xbill profile

Six Serving Engines on Four Accelerators: Gemma 4 E2B on an RTX 3050, a T4, an L4 and a TPU v5e

#gemma #vllm #nvidia #tpu
14
1
11 min read
Prefill vs Decode: Why Splitting Them Wins
danielsamfdo profile

Prefill vs Decode: Why Splitting Them Wins

#vllm #llm #googlecloud #machinelearning
1
10 min read
Your prefill/decode split has a network bill nobody puts in the diagram
aiapikey profile

Your prefill/decode split has a network bill nobody puts in the diagram

#llm #performance #infrastructure #vllm
3 min read
Day 5: Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution
suresh_kumar_de3920bedd1c profile

Day 5: Local AI Runtimes — Deploying Ollama and vLLM for Bare-Metal Execution

#ai #ollama #vllm #aiintermediateseries
3 min read
PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency
danielsamfdo profile

PagedAttention and the KV Cache, Plainly: Why Memory, Not FLOPs, Caps Your Concurrency

#vllm #llm #googlecloud #machinelearning
11 min read
llm-compressor for vLLM: The FP8/AWQ Deployment Tradeoff
jangwook_kim_e31e7291ad98 profile

llm-compressor for vLLM: The FP8/AWQ Deployment Tradeoff

#quantization #vllm #fp8 #awq
9 min read
vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade
danielsamfdo profile

vLLM 0.33, Plainly: CUDA 13, Model Runner V2, and What Breaks on Upgrade

#vllm #llm #googlecloud #machinelearning
9 min read
Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack
wolfsea2357 profile

Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack

#ai #opensource #llm #vllm
1
1
15 min read
verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment
jangwook_kim_e31e7291ad98 profile

verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment

#rlposttraining #grpo #vllm #gpumemory
12 min read
vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe
wolfsea2357 profile

vLLM 0.33 Adds a Lossy Speculative Verifier: Why Single-Run Accuracy Tables Cannot Tell You It Is Safe

#ai #machinelearning #llm #vllm
15 min read
Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC
rammehta1899 profile

Managing Agent GPU Kernels: vLLM's Kimi K3 Fast-Track RFC

#vllm #gpukernels #cutedsl #codereview
1 min read
vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap
wolfsea2357 profile

vLLM 0.33 Ships an ABI-Stable Wheel, but PyPI Still Pins torch==2.14.0: What You Can Actually Swap

#ai #python #llm #vllm
1
13 min read
vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap
wolfsea2357 profile

vLLM Fast Start Explained: What Preload and Snapshots Save, What They Cannot, and the Fingerprint Gap

#ai #machinelearning #llm #vllm
14 min read
The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...
lifehacknexus profile

The Local vLLM Speculative Decoding Blueprint: How I Quadrupled DeepSeek-R1 Inference to 48 Tokens/Sec on Consumer GPUs with...

#aitoolsproductivity #aitools #deepseekr1 #vllm
14 min read
Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU
xbill profile

Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU

#gemma #vllm #googlecloud #nvidia
20
3
13 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.
⌂