4 × Tesla V100-32GB · NVFP4 · Speculative decoding

四张 V100,单请求 500+ tok/s 写代码,256K 上下文 8 并发

Four V100s. 500+ tok/s of code for one user, 256K context for eight.

1Cat-vLLM 1.5.2 让 2017 年的数据中心显卡流畅运行 Qwen3.8 大模型:投机解码一轮输出多个 token,FP4 权重把显存带宽用到极限,KV 历史放进主机内存撑起 256K 上下文。页面上每个数字都来自同一台 4×V100 服务器的实测。

1Cat-vLLM 1.5.2 runs Qwen3.8 models smoothly on 2017-era datacenter GPUs: speculative decoding emits several tokens per step, FP4 weights push memory bandwidth to its limit, and KV history in host memory carries 256K context. Every number on this page was measured on one 4×V100 server.

578tok/s
27B 单请求写代码(HumanEval,温度 0.6 实测)27B, one request, HumanEval at temperature 0.6 (measured)
1224tok/s
27B 8 并发总吞吐(按 70% 接受率)27B, 8 concurrent, total (70% acceptance)
−18%
Flash-Next 从 1K 到 250K 上下文的速度降幅Flash-Next slowdown from 1K to 250K
1.7s
带 250K 历史的多轮追问首字First token of a follow-up on 250K history
Qwen3.8-27B NVFP4 + DFlash2Qwen3.8-Flash-Next NVFP4 + MTP4OpenAI 兼容接口OpenAI-compatible API128 个 PR128 merged PRs
Live replay

看它写代码

Watch it write code

两个模型在 HumanEval 上的真实输出(19 道题,全部通过单元测试),按 1.5.2 实测的速度和投机解码节奏逐轮输出:每一轮验证一批草稿 token,被接受的会一起出现(高亮)。自动循环播放;这是实测回放,不在你的浏览器里运行模型。

The real HumanEval outputs of both models (19 problems, all passing their tests), replayed at the measured 1.5.2 speed and speculative rhythm: each step verifies a batch of draft tokens, and the accepted ones appear together (highlighted). It loops automatically. This is a replay of measurements; no model runs in your browser.

LIVE  单请求 · 温度 0.6 · 4× V100-32GBOne request · temperature 0.6 · 4× V100-32GB
Qwen3.8-27B NVFP4+ DFlash2 · 每轮验证 8 个 token8 tokens verified per step
0tok/s · 0.0s · 0 tok

      
Qwen3.8-Flash-Next NVFP4+ MTP4 · 每轮验证 5 个 token5 tokens verified per step
0tok/s · 0.0s · 0 tok

      
实测 578 tok/s · 平均每轮出 7.1 个 token · 每轮 12.2 ms · 19 道题约 7.8 sMeasured 578 tok/s · 7.1 tokens per step · 12.2 ms per step · 19 problems in about 7.8 s
实测 252 tok/s · 平均每轮出 4.6 个 token · 每轮 18.4 ms · 19 道题约 19 sMeasured 252 tok/s · 4.6 tokens per step · 18.4 ms per step · 19 problems in about 19 s
0.35 s

27B 写完一个 200 token 的完整函数(带文档和注释)

27B writes a complete 200-token function with docstring and comments

×7.1

写代码时每一轮读一遍权重,平均产出 7.1 个 token

On code, each pass over the weights yields 7.1 tokens on average

4520

27B 约 7.8 秒写完 19 道 HumanEval 题的 token 数,全部通过测试

Tokens 27B writes for 19 HumanEval problems in about 7.8 s, all passing

LIVE  27B · 4 个用户同时写代码 · 温度 0 · 自动循环27B · four users writing code at once · temperature 0 · looping
合计吞吐
Total
0 tok/s
每路
Each
0 tok/s
已写 token
Tokens written
0
4 并发 HumanEval 实测:每路 233 tok/s,合计约 750 tok/s。每路连续写多道题,循环播放。Measured with four concurrent HumanEval requests: 233 tok/s each, about 750 tok/s in total. Each lane writes several problems in a row and loops.
Throughput

速度全景

Speed at a glance

投机解码的速度取决于草稿被接受的比例,而这个比例随内容变化很大。为了让不同场景可以直接比较,并发与长文两张图把实测的每轮耗时统一按 70% 平均接受率折算(27B 每轮 5.9 个 token,Flash-Next 3.8 个);数据集一图是真实题目上的实测速度。

Speculative speed depends on how many drafts are accepted, which varies with content. To compare scenarios, the concurrency and long-context charts convert measured step times at a 70% average acceptance (5.9 tokens per step for 27B, 3.8 for Flash-Next). The dataset chart shows measured speed on real prompts.

并发越多,总吞吐越高

More users, more total throughput

8K 上下文,温度 0,tok/s;柱顶小字为每路速度

8K context, temperature 0, tok/s; per-request speed above each bar

27BFlash-Next

长文几乎不掉速

Long context barely slows it down

单请求 tok/s,1K 到 250K 上下文

One request, tok/s, 1K to 250K context

27BFlash-Next

真实数据集单请求速度

One request on real datasets

实测 tok/s,括号内为草稿接受率

Measured tok/s; draft acceptance in brackets

27BFlash-Next

长对话:第二轮几乎不用等

Long chats: the next turn barely waits

27B 首字时间(秒,对数刻度):首次输入 vs 多轮追加命中缓存

27B time to first token (s, log scale): first prompt vs cached follow-up

首次First prompt追加一轮Follow-up
Quality

压到 4 bit,智商不掉

4-bit weights, same intelligence

与原版 BF16 模型在同一套题、同样的贪心解码下逐题对比。27B 与 BF16 的差距都在统计误差内,所有输出中没有乱码,也没有重复循环。

Compared question by question with the original BF16 model, same prompts, greedy decoding. Every 27B difference is within statistical error, with no garbled text or repetition loops in any output.

准确率

Accuracy

BF1627B NVFP4Flash-Next NVFP4

怎么保证不降智

How precision is kept

  • 只舍入一次:FP4 权重解码到 FP16 只舍入一次,矩阵乘全程 FP32 累加
  • 关键状态高精度:logits、线性注意力状态都用 FP32
  • KV 校准:FP8 E4M3 KV,按层加载校准系数
  • 逐 token 对齐:与 BF16 出现分叉时,BF16 的优势中位数仅 0.2(logprob),从未选到 BF16 前 5 名之外的 token
  • 采样无偏:投机解码与不用草稿模型时的采样分布对比,每个 prompt 600 次采样逐位置检验,未见系统性偏差
  • One rounding: FP4 weights decode to FP16 once; GEMMs accumulate in FP32
  • High-precision state: logits and linear-attention state stay in FP32
  • Calibrated KV: FP8 E4M3 KV with per-layer scales
  • Token alignment: where outputs diverge from BF16, BF16's margin has a median of 0.2 logprob, and the chosen token is always in BF16's top 5
  • Unbiased sampling: speculative sampling compared with sampling without a draft, 600 samples per prompt per position, no systematic bias
Under the hood

为什么这么快

Why it is fast

×6.9

一轮出多个 token

Several tokens per step

DFlash2 草稿模型一次提出 7 个 token,目标模型一轮全部验证。写代码时(温度 0)平均每轮输出 6.9 个 token。

The DFlash2 draft proposes 7 tokens and the target verifies them in one step. On code at temperature 0, each step yields 6.9 tokens on average.

~4.5 bit

权重小,带宽跑满

Small weights, full bandwidth

NVFP4 权重只有 FP16 的约 1/3.5。V100 没有 FP4 硬件,我们用自研算子在寄存器里解码,单请求每轮耗时已贴近只读一遍权重所需的时间。

NVFP4 weights are about 1/3.5 the size of FP16. V100 has no FP4 hardware, so our kernels decode in registers; one request per step is close to the time needed just to read the weights.

理论下限(估算)Floor (estimate)
≈12.0 ms
1.5.2 实测1.5.2 measured
12.1 ms
256K

KV 历史放主机内存

KV history in host memory

Flash-Next 只把最近 32K 的 KV 留在显存,更早的历史以 FP8 放进主机内存,由稀疏注意力按需读取;48 层中 36 层是线性注意力,所以 250K 时只比 1K 慢 18%。

Flash-Next keeps the latest 32K of KV on the GPU and older history in host memory as FP8, read on demand by sparse attention; 36 of 48 layers use linear attention, so 250K is only 18% slower than 1K.

Data room

完整测试数据

Complete test data

发布前在同一台 4×V100 上跑完的全部测试。纯 decode 计时只统计所有请求都在 decode 的步,不含 prefill;所有测试零错误、零显存溢出。

Every test run on the same 4×V100 before release. Pure decode timing counts only steps where all requests are decoding, excluding prefill. Zero errors and zero out-of-memory in all runs.

Deploy

测试环境与部署

Test setup and deployment

测试环境

Test setup

  • 4× Tesla V100-SXM2-32GB(NVLink),张量并行 4
  • 显存 1107 MHz,核心最高 1530 MHz,功耗上限 350 W,ECC 开
  • 主机内存 62 GB
  • Qwen3.8-27B NVFP4(Unsloth)+ DFlash2;Qwen3.8-Flash-Next NVFP4 + MTP4
  • 4× Tesla V100-SXM2-32GB (NVLink), tensor parallel 4
  • Memory 1107 MHz, core up to 1530 MHz, 350 W, ECC on
  • 62 GB host memory
  • Qwen3.8-27B NVFP4 (Unsloth) + DFlash2; Qwen3.8-Flash-Next NVFP4 + MTP4

部署方式

Deployment

  • pip 安装 wheel
  • Docker:一个镜像启动两个模型
  • 1Cat Studio:图形界面一键加载
  • OpenAI 兼容接口:工具调用、结构化输出、思考模式、图片输入(27B)
  • pip-install the wheel
  • Docker: one image serves both models
  • 1Cat Studio: one-click loading in a GUI
  • OpenAI-compatible API: tool calls, structured output, reasoning, image input (27B)

硬件要求

Requirements

  • 4× V100-SXM2-32GB(NVLink)
  • 主机内存 64 GB 以上
  • 最大 256K 上下文、8 并发
  • 27B 启动约 2.5 分钟,Flash-Next 约 6 分钟(有编译缓存)
  • 4× V100-SXM2-32GB (NVLink)
  • 64 GB+ host memory
  • Up to 256K context, 8 concurrent requests
  • Startup about 2.5 min for 27B and 6 min for Flash-Next (warm compile cache)