1.5.4 新特性
What's new in 1.5.4
Flash-Next 是这一版变化最大的模型:KV 容量是 1.5.3 的 4.1 倍,8 并发 64K 和 4 并发 240K 都能放下,8K–64K 各档吞吐高 24–90%。27B 的 MTP4 在 8K 4–8 并发每轮省 9–17%,FP8 checkpoint 的 KV 多 77%,W8A16 从“输出 NaN”变为可用。单请求温度 0 是唯一变慢的场景(INT8 KV,见“已知问题”)。
Flash-Next changes the most: 4.1× the KV of 1.5.3, room for 8 users at 64K and 4 users at 240K, and 24–90% more throughput at 8K–64K. For 27B, MTP4 steps with 4–8 users at 8K are 9–17% cheaper, the FP8 checkpoint gets 77% more KV, and W8A16 goes from NaN output to usable. One request at temperature 0 is the only slower case (INT8 KV, see Known issues).
1.5.3 → 1.5.4 关键对比
1.5.3 → 1.5.4 at a glance
同机同口径;每轮 ms 越低越好,tok/s 越高越好
Same server and method; lower ms per step and higher tok/s are better
Flash-Next:MTP5 + DCP2 + 动态 EntMTP
Flash-Next: MTP5 + DCP2 + adaptive EntMTP
- KV 4.1 倍:INT8 逐 token·head 主机 KV、262K 热区,DCP=2 把历史分到两组卡,378,652 → 1,536,844 token(#1570)
- 每轮草稿数自动调整:MTP5 按熵动态选 2–5 个草稿,8K 单请求温度 0.6 每轮 20.4 → 12.7 ms、合计 108 → 201 tok/s
- 长文几乎不掉速:1K → 250K 每轮只多 6.7%(1.5.3 多 19.7%),250K 单请求 209 → 300 tok/s
- 不加参数也能启动:默认最大并发按显存自动限制,裸
vllm serve不再失败(#1564) - 代价:启动时主机内存最低只剩 15 GiB(1.5.3 为 38 GiB)
- 4.1× KV: per-token-head INT8 host KV with a 262K hot window, and DCP=2 splits history across two GPU groups: 378,652 → 1,536,844 tokens (#1570)
- Adaptive draft count: MTP5 picks 2–5 drafts by entropy; 8K, one request, temperature 0.6: 20.4 → 12.7 ms per step, 108 → 201 tok/s
- Almost flat with context: 1K → 250K costs only 6.7% more per step (19.7% in 1.5.3); 250K, one request: 209 → 300 tok/s
- Starts with no extra flags: the default max concurrency is now sized to memory, so a bare
vllm serveworks (#1564) - Cost: minimum free host memory during startup is 15 GiB (38 GiB in 1.5.3)
27B:并发更快、KV 更多
27B: faster with users, more KV
- MTP4 并发验证:8K 4 / 8 并发每轮 −9% / −15%,64K 8 并发 −14%(#1544、#1518)
- 温度 > 0 不再主机同步:8K 单请求温度 0.6 每轮 15.7 → 14.4 ms,与温度 0 的差距从 2.7 ms 缩到 0.9 ms
- INT8 KV 成为默认:KV +25%(1,598,295 → 1,991,511),得分与 FP8 KV 相同(#1552)
- FP8 / W8A16 原生 QPN8:FP8 KV 1.02M → 1.81M、8K 2–8 并发 +12–34%;W8A16 在 1.5.3 输出 NaN,1.5.4 起可用,多请求速度与 NVFP4 相当(#1524)
- DFlash2:每轮耗时与 1.5.3 持平,温度 0 真实数据集 4 并发合计 +10–35%(prefill / decode 分离调度)
- Concurrent MTP4 verify: 8K with 4 / 8 users −9% / −15% per step, 64K with 8 users −14% (#1544, #1518)
- No host sync above temperature 0: 8K, one request, temperature 0.6: 15.7 → 14.4 ms per step; the gap to temperature 0 shrinks from 2.7 to 0.9 ms
- INT8 KV by default: +25% KV (1,598,295 → 1,991,511) with the same scores as FP8 KV (#1552)
- Native QPN8 for FP8 / W8A16: FP8 KV 1.02M → 1.81M and +12–34% with 2–8 users at 8K; W8A16 printed NaN on 1.5.3 and now matches NVFP4 with several users (#1524)
- DFlash2: same step time as 1.5.3; real datasets at temperature 0 with 4 users +10–35% in total (separate prefill / decode scheduling)
缓存、agent 与 API
Cache, agents and API
- 模型自己的回复也能命中缓存:decode 输出按 64 token 进前缀缓存,agent 多轮后续请求命中 97–98%(#1529、#1559)
- prefill 与 decode 分开调度:64K 新请求 prefill 期间,正在输出的请求能走的步数是 1.5.3 的 2.6–3 倍(#1525)
- 思考 / 非思考动态采样预设;温度 0 不再套默认惩罚项(#1557、#1564)
- 图片输入默认开启且不限张数(#1562)
- 修复:结构化输出草稿 token 跨 prefill 保留(#1530)、跨轮推理内容串扰(#1545)、截断的 required 工具调用校验(#1566);新引擎第一个温度 > 0 的请求不再慢 3 秒
- The model's own replies hit the cache: decode output enters the prefix cache in 64-token blocks; agent follow-ups hit 97–98% (#1529, #1559)
- Prefill and decode scheduled apart: while a new 64K prompt prefills, running requests take 2.6–3× as many steps as on 1.5.3 (#1525)
- Dynamic thinking / non-thinking sampling presets; no default penalties at temperature 0 (#1557, #1564)
- Image input on by default, any number of images (#1562)
- Fixes: structured-output draft tokens survive interleaved prefill (#1530), cross-turn reasoning leakage (#1545), truncated required tool-call validation (#1566); the first sampled request on a fresh engine is no longer 3 s slow
测试口径
How this was measured
- 同机同时钟:与 1.5.3 报告同一台 4×V100,显存 877 MHz、功耗上限 300 W
- 1.5.3 数据可直接对比:NVFP4 三组用 1.5.3 发布数据,并在本机抽点复测(8K 单请求 12.95 vs 发布 12.99 ms,GSM8K 289 vs 290);FP8 / W8A16 的 1.5.3 结果为本次实测
- 同一套工具:测试矩阵、数据集与质量评测与 1.5.3 相同;每组冷启动后先跑一轮不计入的预热
- 新增:FP8 与 W8A16 checkpoint、agent 多轮工具调用、逐 token KLD、长 prompt 干扰、tools + JSON 回归
- 本次未测:双卡(TP2)、思考模式难题集、DeepSeek-V4.1(16 卡)
- Same server and clocks: the 4×V100 of the 1.5.3 report, memory 877 MHz, 300 W
- Directly comparable with 1.5.3: the three NVFP4 configs use the published 1.5.3 data, spot-checked on this server (8K one request 12.95 vs 12.99 ms published, GSM8K 289 vs 290); FP8 / W8A16 on 1.5.3 were measured in this run
- Same tools: matrix, datasets and quality evaluation match 1.5.3; every engine gets an unrecorded warm-up pass after a cold start
- New: FP8 and W8A16 checkpoints, multi-turn agent sessions, per-token KLD, long-prompt interference, tools + JSON regression
- Not run this time: two-GPU (TP2), thinking-mode hard benchmarks, DeepSeek-V4.1 (16 GPUs)
需要注意的问题
What to watch
- tools + json_object 误调工具(1.5.4 回归):27B 温度 0 下 10/10 调用工具(1.5.3 为 0/10)
- INT8 KV 单请求固定开销:每步多约 0.9 ms,27B 单请求温度 0 比 1.5.3 慢 2–8%
- 27B agent 8 并发首字 p90:4.6 → 7.4 s,p50 与吞吐都更好
- Flash-Next 主机内存:64 GB 主机上余量偏小
- tools + json_object calls the tool (1.5.4 regression): 27B at temperature 0 calls the tool 10/10 times (0/10 on 1.5.3)
- INT8 KV fixed cost for one request: about 0.9 ms more per step; one 27B request at temperature 0 is 2–8% slower than 1.5.3
- 27B agent p90 TTFT with 8 sessions: 4.6 → 7.4 s, while p50 and throughput improve
- Flash-Next host memory: little headroom on a 64 GB host
全部问题与 1.5.3 已知限制复测 →All issues and the 1.5.3 limitation re-check →
这一版做了什么
What changed
v1.5.3 之后合并到 main 的 61 个 PR(含 #1527 一次审计合并的 34 个社区 PR),按类别列出;完整清单见“完整测试数据”里的“全部 PR”。
The 61 PRs merged to main since v1.5.3 (including 34 community PRs merged in one audit, #1527), grouped by area; the full list is under "All PRs" in the data room.
Qwen3.8 速度、显存与 KV
Qwen3.8 speed, memory and KV
- MTP decode 逼近 V100 HBM2 / NVLink 上限:并发验证、紧凑采样,温度 > 0 时不再主机同步(#1518、#1544)。
- Flash-Next MTP 提速:无同步采样、并发 decode(#1519、#1521);主机 KV + DCP=2、零衰减 QSA H12 / indexer、MoE zero-cat、MTP5 动态 EntMTP(#1570)。
- INT8 KV:QSA 与 Flash-V100 原生逐 token·head INT8 KV,并设为默认(#1552)。
- TP2 / TP4 通用快速路径:FP8 与 W8A16-FP8 走原生 QPN8,W8A16 每卡省 4.7 GiB(TP4)/ 9.4 GiB(TP2)(#1524)。
- 调度:引擎默认 prefill / decode 分离、部分前缀缓存、Qwen3.8 并发(#1525);NVFP4 MoE 原始 scale 与显存优化(#1544)。
- MTP decode near the V100 HBM2 / NVLink roofline: concurrent verify, compact sampling, no host sync above temperature 0 (#1518, #1544).
- Faster Flash-Next MTP: no-sync sampling and concurrent decode (#1519, #1521); host KV + DCP=2, zero-decay QSA H12 / indexer, MoE zero-cat and MTP5 adaptive EntMTP (#1570).
- INT8 KV: native per-token-head INT8 KV for QSA and Flash-V100, now the default (#1552).
- Shared TP2 / TP4 fast paths: FP8 and W8A16-FP8 run native QPN8; W8A16 saves 4.7 GiB per GPU on TP4 and 9.4 GiB on TP2 (#1524).
- Scheduling: engine-default prefill / decode separation, partial prefix cache, Qwen3.8 concurrency (#1525); raw-scale NVFP4 MoE and memory savings (#1544).
缓存命中、API 与修复
Cache hits, API and fixes
- decode 输出进前缀缓存:64 token 粒度、思考历史恢复、流式工具解析(#1529);27B 多轮 agent 缓存、API 体验与显存(#1559)。
- 采样预设:Qwen3.8 思考 / 非思考动态采样,放宽 SM70 / INT8 路径条件(#1557);温度 0 跳过默认惩罚、裸 serve 的 Qwen4Exp 并发上限(#1564)。
- 视觉:默认开启且不限张数、降低多轮首字(#1562);ViT 保留 TORCH_SDPA Cutlass Tensor Core 后端(#1563)。
- 修复:结构化输出草稿 token(#1530)、跨轮推理缓存串扰(#1545)、混合 prefill 预热计数(#1560)、required 工具调用与 response_format 标签(#1566)。
- Decode output in the prefix cache: 64-token granularity, reasoning history restore, streaming tool parser (#1529); 27B multi-turn agent cache, API polish and memory (#1559).
- Sampling presets: dynamic Qwen3.8 thinking / non-thinking sampling, relaxed SM70 / INT8 path conditions (#1557); no default penalties at temperature 0 and a bounded Qwen4Exp concurrency for bare serve (#1564).
- Vision: on by default with no image limit, lower multi-turn TTFT (#1562); ViT keeps the TORCH_SDPA Cutlass Tensor Core backend (#1563).
- Fixes: structured-output draft tokens (#1530), cross-turn reasoning cache leakage (#1545), mixed-prefill warmup count (#1560), required tool calls and response_format tags (#1566).
社区 PR 审计合并(#1527,34 个)
Community PR audit merge (#1527, 34 PRs)
- SM70 正确性:Marlin bias(#1507)、GDN 验证投影的 FP8 策略(#1508)、grouped e4m3 路由(#1517)、MTP 编译图外的 HC prefill(#1501)。
- 引擎稳定性:缓存漂移自动恢复而不是杀引擎(#1497)、冷编译后恢复栈上限(#1502)、上游 resample 竞争与 shm 卡顿回移(#1408)。
- 工具解析:GLM 非字符串参数流式(#1512)、已完整流出的参数不再补发(#1511)。
- 平台:无 P2P 的双卡 all-reduce(#1216)、拓扑感知 all-reduce(#1245)、Gemma4 V100 支持(#1271)、EAGLE/MTP lookahead 前缀命中(#1276);另含测试与重构。
- SM70 correctness: Marlin bias (#1507), FP8 policy for the GDN verify projection (#1508), grouped e4m3 routing (#1517), HC prefill context outside the compiled MTP forward (#1501).
- Engine stability: recover from cache drift instead of killing the engine (#1497), restore the stack limit after cold compile (#1502), upstream resample-race and shm-stall backports (#1408).
- Tool parsing: GLM non-string argument streaming (#1512), no duplicate args when already streamed (#1511).
- Platforms: two-GPU all-reduce without P2P (#1216), topology-aware all-reduce (#1245), Gemma4 on V100 (#1271), EAGLE/MTP lookahead prefix hits (#1276); plus tests and refactors.
DeepSeek-V4.1(本次未复测)
DeepSeek-V4.1 (not re-tested here)
- PP4 批处理:异步 micro-batch 收尾、prefill 与 IB 重叠、512K indexer 尾部(#1568);采样计数、DSpark 草稿保护、批前缀缓存(#1558)。
- 性能:分布式 IB 对端内存 Engram、key-stationary prefill indexer(#1567);批量 SM70 kernel、prefill 融合、agent 前缀缓存(#1565)。
- 图片输入走原生 PP 路径(#1561);社区 decode / prefill kernel(#1383–#1390,默认关闭)。
- PP4 batching: async micro-batch wind-down, prefill / IB overlap, 512K indexer tail (#1568); sampled counts, DSpark draft guard, batch prefix caching (#1558).
- Performance: distributed IB peer-RAM Engram, key-stationary prefill indexer (#1567); batched SM70 kernels, prefill fusion, agent prefix cache (#1565).
- Image input on the native PP path (#1561); community decode / prefill kernels (#1383–#1390, off by default).
速度全景:1.5.3 vs 1.5.4
Speed: 1.5.3 vs 1.5.4
投机解码的速度随草稿接受率变化,合成测试文本上的接受率每次都有波动。与 1.5.3 报告相同,27B 的并发与长文两张图把实测每轮耗时按 70% 平均接受率折算(DFlash2 每轮 5.9 个 token,MTP4 3.8 个),只比较每轮成本;Flash-Next 1.5.4 每轮草稿数是动态的,按实测每轮 token 计算。数据集一图是真实题目上的实测速度。
Speculative speed follows draft acceptance, which wanders on synthetic text. As in the 1.5.3 report, the 27B concurrency and long-context charts convert measured step times at a 70% average acceptance (5.9 tokens per step for DFlash2, 3.8 for MTP4), so they compare step cost only; Flash-Next 1.5.4 has a dynamic draft count and uses measured tokens per step. The dataset chart is measured speed on real prompts.
并发总吞吐
Total throughput by users
长文掉速
Long-context slowdown
KV 容量
KV capacity
启动日志 GPU KV cache size,万 token;Flash-Next 为主机 + 显存 KV
GPU KV cache size from the startup log, in 10K tokens; Flash-Next counts host + GPU KV
真实数据集 4 并发
Real datasets, 4 users
温度 0,4 并发实测合计 tok/s;单请求两版基本持平(见完整数据)
Temperature 0, measured total tok/s with 4 users; one request is about the same in both versions (see the data room)
agent 多轮工具调用:KV 和缓存决定体验
Multi-turn agents: KV and cache decide the experience
新增测试:类 Claude Code 的多轮 agent 会话,真实执行 Read / Write / Edit / Bash / Grep 工具,混合长回答对话,思考模式、官方采样(温度 1.0、top_p 0.95、top_k 20),每档最长 12 分钟。1.5.3 的 Flash-Next 只有 379K KV,8 个会话(平均 3.7 万、最长 8 万 token)同时跑时历史被挤出缓存,命中跌到 76%、首字 p90 23 秒;1.5.4 命中 95%、吞吐 2.5 倍。
New test: Claude Code-style multi-turn agent sessions that really execute Read / Write / Edit / Bash / Grep tools, mixed with long-answer chats, thinking mode and official sampling (temperature 1.0, top_p 0.95, top_k 20), up to 12 minutes per level. Flash-Next on 1.5.3 has only 379K KV: with 8 sessions (37K tokens on average, up to 80K) history is evicted, hits fall to 76% and p90 TTFT reaches 23 s. 1.5.4 keeps 95% hits and 2.5× the throughput.
合计输出 tok/s
Total output tok/s
1 / 4 / 8 个 agent 会话同时运行,含思考、工具调用与首字等待
1 / 4 / 8 concurrent agent sessions, including thinking, tool calls and TTFT
首字与缓存命中
TTFT and cache hits
后续轮次命中 = 第二轮及以后的请求有多少 prompt token 命中前缀缓存
Follow-up hit = share of prompt tokens served from the prefix cache on the second and later turns
所有会话工具调用格式错误均为 0。27B 8 并发时 1.5.4 的首字 p50 更低、吞吐高 40%,但 p90 从 4.6 s 升到 7.4 s:长 prompt 进来时排队的尾部更长。
No malformed tool call in any session. With 8 sessions, 27B on 1.5.4 has a lower p50 TTFT and 40% more throughput, but p90 rises from 4.6 to 7.4 s: the queue tail is longer when long prompts arrive.
27B 三种权重和 Flash-Next,智商都不掉
Three 27B formats and Flash-Next, same intelligence
与原版 27B BF16 在同一套题、同样的贪心解码下逐题对比。1.5.4 的 NVFP4、FP8、W8A16 与 BF16 的差距都在统计误差内,INT8 KV 与 FP8 KV 得分相同;所有输出没有乱码,也没有重复循环。
Compared question by question with the original 27B BF16, same prompts, greedy decoding. On 1.5.4, NVFP4, FP8 and W8A16 are all within statistical error of BF16, and INT8 KV scores the same as FP8 KV; no garbled text or repetition loops anywhere.
准确率(1.5.4)
Accuracy (1.5.4)
1.5.4 的精度变化
Precision changes in 1.5.4
- INT8 KV 不降分:同一引擎只换 KV 类型,GSM8K / MMLU / 代码 INT8 290 / 331 / 22,FP8 290 / 333 / 22
- 逐 token 分布与 1.5.3 一致:27B NVFP4 对 BF16 的平均 KLD 两版都是 0.0485,Top-1 一致 96.8%
- 8 bit 权重更接近 BF16:FP8 / W8A16 的 KLD 0.022 / 0.021,Top-1 一致 98.4%
- W8A16 修复:1.5.3 丢失 index.json 未列出的 weight_scale,输出 NaN;1.5.4 保留同分片缩放系数(#1524)
- 分叉处差距小:27B NVFP4 与 BF16 分叉时 BF16 的优势中位数 0.23 logprob
- INT8 KV costs no accuracy: same engine, KV type only: GSM8K / MMLU / code INT8 290 / 331 / 22, FP8 290 / 333 / 22
- Per-token distribution unchanged from 1.5.3: 27B NVFP4 mean KLD to BF16 is 0.0485 on both, 96.8% top-1 agreement
- 8-bit weights sit closer to BF16: FP8 / W8A16 KLD 0.022 / 0.021, 98.4% top-1 agreement
- W8A16 fix: 1.5.3 dropped weight_scale tensors missing from index.json and printed NaN; 1.5.4 keeps same-shard scales (#1524)
- Small divergence margins: where 27B NVFP4 diverges from BF16, BF16's median margin is 0.23 logprob
为什么更快
Why it is faster
KV 放主机、分两组卡
KV on the host, split across GPU pairs
Flash-Next 的历史 KV 以 INT8 逐 token·head 放进主机内存,最近 262K 留作热区;DCP=2 让两组卡各存一半历史、各算一半稀疏注意力,所以容量 4 倍而 250K 只比 1K 慢 6.7%。
Flash-Next keeps history KV in host memory as per-token-head INT8 with a 262K hot window; DCP=2 lets each GPU pair hold and attend over half the history, so capacity grows 4× while 250K is only 6.7% slower than 1K.
草稿数按把握调整
Draft count follows confidence
MTP5 + 动态 EntMTP 按预测熵决定每轮提几个草稿:好猜的内容一次 5 个,难猜的只提 2 个、不浪费验证。27B 的 MTP4 则把多个请求的验证合到一次权重读取里。
MTP5 with adaptive EntMTP uses prediction entropy to choose how many drafts to propose: 5 for easy text, 2 for hard text, so verification is not wasted. 27B MTP4 folds several requests' verification into one pass over the weights.
prefill 不再挡住 decode
Prefill no longer blocks decode
prefill 与 decode 分开调度:64K 新请求 prefill 期间,正在输出的请求走了 300 步(1.5.3 同配置 117 步);模型自己的回复也按 64 token 进前缀缓存,agent 下一轮几乎不用重算。
Prefill and decode are scheduled apart: while a new 64K prompt prefills, running requests take 300 steps (117 on 1.5.3); the model's own replies enter the prefix cache in 64-token blocks, so the next agent turn barely recomputes.
完整测试数据
Complete test data
同一台 4×V100 上跑完的全部测试,每张表 1.5.3 与 1.5.4 并排。纯 decode 计时只统计所有请求都在 decode 的步(服务端逐步计时);稳态合计 = 并发 × 每轮 token × 1000 / 每轮 ms。
Every test run on the same 4×V100, with 1.5.3 and 1.5.4 side by side. Pure decode timing counts only steps where every request is decoding (server-side step timing); steady total = users × tokens per step × 1000 / ms per step.
启动参数与 KV 容量
Launch commands and KV capacity
本页每组 1.5.4 数据对应的完整启动参数(全加速路径),复制即可使用,模型路径按实际目录修改。测速时额外设置了服务端逐步计时 VLLM_SM70_ASYNC_CPU_TRACE=1,不影响速度。
The complete launch arguments behind every 1.5.4 result on this page (full acceleration path), ready to copy; adjust model paths. Speed runs also set VLLM_SM70_ASYNC_CPU_TRACE=1 for server-side step timing, which does not change speed.
27B NVFP4 · MTP4 KV 1,991,511 token
1.5.4 发布默认:INT8 逐 token·head KV(不传 --kv-cache-dtype 也是 INT8),前缀缓存网格自动设置。
1.5.4 release default: per-token-head INT8 KV (also the default when --kv-cache-dtype is omitted); prefix-cache geometry is set automatically.
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-mtp \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'27B NVFP4 · DFlash2 KV 1,634,112 token
DFlash2 草稿模型放在本地目录;其余与 MTP4 相同。
The DFlash2 draft sits in a local directory; everything else matches MTP4.
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-dflash2 \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'Flash-Next NVFP4 KV 1,536,844 token
与 Docker docker/sm70_release/serve.sh(main 1fa9c438bc)相同:DCP=2、MTP5 + 动态 EntMTP、INT8 主机 KV、262K 热区。
Same as Docker docker/sm70_release/serve.sh (main 1fa9c438bc): DCP=2, MTP5 with adaptive EntMTP, INT8 host KV, 262K hot window.
vllm serve /models/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen3.8-flash-next-nvfp4 \
--tensor-parallel-size 4 --decode-context-parallel-size 2 --dtype half --max-model-len 262144 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --attention-backend FLASH_ATTN_V100 \
--enable-prefix-caching --partial-prefix-cache --block-size 2048 --mamba-block-size 2048 \
--mixed-prefill-step-latency-ms 0 --enable-prompt-tokens-details --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 --trust-remote-code --seed 0 \
--kv-cache-dtype float16 --gpu-memory-utilization 0.89 --mamba-cache-mode align \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --mamba-ssm-cache-dtype float32 \
--default-chat-template-kwargs '{"enable_thinking":false}' \
--kernel-config '{"sm70_hcx": true, "sm70_qsa_prep": true, "sm70_hcx_local_schedule": true, "hc_weight_storage": "sharded", "qsa_host_kv": true, "qsa_host_kv_dtype": "int8_per_token_head", "qsa_host_kv_draft_dtype": "int8_per_token_head", "qsa_host_kv_hot_dtype": "int8_per_token_head", "qsa_host_kv_hot_tokens": 262144, "qsa_host_kv_prefill_stage_tokens": 262144, "qsa_host_kv_trim_warmup_cache": true, "qsa_host_kv_state_blocks": 176, "qsa_host_kv_history_blocks": 768}' \
--speculative-config '{"method": "mtp", "model": "/models/Qwen3.8-Flash-Next-mtp", "num_speculative_tokens": 5, "draft_sample_method": "greedy"}'27B FP8 · MTP4 KV 1,805,663 token
FP8 checkpoint 走原生 QPN8 快速路径(#1524)。
The FP8 checkpoint runs the native QPN8 fast path (#1524).
vllm serve /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b-fp8-mtp \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'27B FP8 · DFlash2 KV 1,412,800 token
vllm serve /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b-fp8-dflash2 \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'27B W8A16-FP8 · MTP4 KV 1,711,761 token
W8A16-FP8 checkpoint;1.5.3 加载后输出 NaN,1.5.4 起可用。
W8A16-FP8 checkpoint; 1.5.3 outputs NaN, usable from 1.5.4.
vllm serve /models/Qwen3.8-27B-W8A16-FP8 --served-model-name qwen3.8-27b-w8a16-mtp \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'27B W8A16-FP8 · DFlash2 KV 1,319,703 token
vllm serve /models/Qwen3.8-27B-W8A16-FP8 --served-model-name qwen3.8-27b-w8a16-dflash2 \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype int8_per_token_head --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'1.5.3 对比组的启动命令(5 组)1.5.3 comparison launch commands (5)
27B NVFP4 · MTP4 · 1.5.3 KV 1,598,295 token
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-mtp \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype fp8_e4m3 --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'27B NVFP4 · DFlash2 · 1.5.3 KV 1,308,270 token
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-dflash2 \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype fp8_e4m3 --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'Flash-Next NVFP4 · 1.5.3 KV 378,652 token
vllm serve /models/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen3.8-flash-next-nvfp4 \
--tensor-parallel-size 4 --dtype half --kv-cache-dtype float16 --language-model-only \
--max-model-len 262144 --max-num-batched-tokens 4096 --max-num-seqs 8 \
--gpu-memory-utilization 0.89 --attention-backend FLASH_ATTN_V100 --enable-prefix-caching \
--partial-prefix-cache --block-size 2048 --mamba-block-size 2048 \
--mixed-prefill-step-latency-ms 0 --mamba-ssm-cache-dtype float32 --enable-prompt-tokens-details \
--trust-remote-code --seed 0 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking":false}' \
--kernel-config '{"sm70_hcx": true, "sm70_qsa_prep": true, "sm70_hcx_local_schedule": true, "qsa_host_kv": true, "qsa_host_kv_dtype": "fp8_e4m3", "qsa_host_kv_hot_dtype": "fp8_e4m3", "qsa_host_kv_hot_tokens": 32768, "qsa_host_kv_prefill_stage_tokens": 262144}' \
--speculative-config '{"method": "mtp", "model": "/models/Qwen3.8-Flash-Next-mtp", "num_speculative_tokens": 4, "draft_sample_method": "greedy"}'27B FP8 · MTP4 · 1.5.3 KV 1,019,231 token
vllm serve /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b-fp8-mtp \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype fp8_e4m3 --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'27B FP8 · DFlash2 · 1.5.3 KV 748,049 token
vllm serve /models/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b-fp8-dflash2 \
--tensor-parallel-size 4 --dtype half --attention-backend FLASH_ATTN_V100 \
--kv-cache-dtype fp8_e4m3 --max-model-len 262144 --gpu-memory-utilization 0.83 \
--max-num-batched-tokens 4096 --max-num-seqs 8 --mixed-prefill-step-latency-ms 0 \
--block-size 2048 --mamba-block-size 2048 --enable-prefix-caching --mamba-cache-mode align \
--partial-prefix-cache --prefix-cache-retention-interval 8192 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-auto-tool-choice \
--tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":false}' --seed 0 --trust-remote-code \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'已知问题与限制
Known issues and limitations
本次测评发现的问题
Issues found in this evaluation
- 回归tools + json_object 误调工具:同时给 tools(auto)与 response_format=json_object、问题不需要工具时,27B 温度 0 下 1.5.3 0/10 → 1.5.4 10/10 调用工具,默认采样 2/10 → 4/10(Flash-Next 不受影响)。嫌疑在 #1566 / #1557 / #1559。
- 性能INT8 KV 单请求固定开销:每步多约 0.9 ms(250K 多 2.3 ms),并发 ≥ 2 时与 FP8 KV 一致。单请求为主的 27B 部署可先传
--kv-cache-dtype fp8_e4m3,此时比 1.5.3 快 2–4%。 - 性能27B agent 8 并发首字 p90 4.6 → 7.4 s,与长 prompt 干扰同源。
- 性能长 prompt 干扰仍在:64K 新请求进来时正在输出的请求总共被拖慢约 21 s(27B),最长单步约 0.76 s;Flash-Next 约 7 s。
- 资源Flash-Next 主机内存:启动时最低剩余 15 GiB(1.5.3 为 38 GiB),64 GB 主机余量偏小。
- 性能Flash-Next 超长并发偶发长步:240K × 2 并发时纯 decode 步 p99 达 4.2 s(中位 20.8 ms)。
- 资源测完显存上涨约 3 GiB(两版都有,如 27B MTP4 26,964 → 30,204 MiB),未出现 OOM。
- 接口W8A16 无参数工具:tool_choice 指定无参数工具时 arguments 不是合法 JSON(NVFP4 / FP8 正常)。
- 配置发布脚本不一致:Docker serve.sh 的 27B 仍显式 FP8 KV;wheel 自带的 Flash-Next 启动脚本仍为 MTP4、无 DCP。
- regressiontools + json_object calls the tool: with tools (auto) plus response_format=json_object and a question needing no tool, 27B at temperature 0 calls the tool 0/10 → 10/10 times (1.5.3 → 1.5.4), and 2/10 → 4/10 with default sampling (Flash-Next unaffected). Suspects: #1566 / #1557 / #1559.
- speedINT8 KV fixed cost for one request: about 0.9 ms more per step (2.3 ms at 250K); identical to FP8 KV with 2+ users. Single-user 27B deployments can pass
--kv-cache-dtype fp8_e4m3, which is 2–4% faster than 1.5.3. - speed27B agent p90 TTFT with 8 sessions 4.6 → 7.4 s, same root as long-prompt interference.
- speedLong-prompt interference remains: a new 64K prompt slows running requests by about 21 s in total (27B), longest step about 0.76 s; about 7 s on Flash-Next.
- resourcesFlash-Next host memory: 15 GiB minimum free during startup (38 GiB on 1.5.3), little headroom on 64 GB hosts.
- speedOccasional long steps at extreme length: 240K × 2 users, pure-decode p99 step 4.2 s (median 20.8 ms).
- resourcesVRAM grows about 3 GiB after the test suite (both versions, e.g. 27B MTP4 26,964 → 30,204 MiB); no OOM.
- APIW8A16 with a no-argument tool: forcing a tool without parameters returns arguments that are not valid JSON (NVFP4 / FP8 are fine).
- configLaunchers disagree: Docker serve.sh still sets FP8 KV for 27B; the bundled Flash-Next wheel launcher is still MTP4 without DCP.
1.5.3 已知限制的复测
1.5.3 limitations re-checked
最大并发仍为 8(256K 配置的发布默认);Flash-Next 短对话缓存粒度改由 decode 输出 64-token 前缀缓存覆盖,见 agent 命中率。
Max concurrency is still 8 (the release default for 256K); Flash-Next short-chat cache granularity is now covered by the 64-token decode-output prefix cache, see the agent hit rates.
测试环境与部署
Test setup and deployment
测试环境
Test setup
- 4× Tesla V100-SXM2-32GB(NVLink),张量并行 4
- 默认时钟:显存 877 MHz,核心最高 1530 MHz,功耗上限 300 W,ECC 开
- 主机内存 62 GB,28 核
- 1.5.4 wheel = main 1fa9c438bc(#1570 合并后);1.5.3 wheel = cb97f379b;同一套 Python 3.12 / torch 2.10 运行环境
- 模型:Qwen3.8-27B NVFP4 / FP8 / W8A16-FP8 + MTP4 或 DFlash2;Qwen3.8-Flash-Next NVFP4 + MTP
- 4× Tesla V100-SXM2-32GB (NVLink), tensor parallel 4
- Stock clocks: memory 877 MHz, core up to 1530 MHz, 300 W, ECC on
- 62 GB host memory, 28 cores
- 1.5.4 wheel = main 1fa9c438bc (after #1570); 1.5.3 wheel = cb97f379b; one shared Python 3.12 / torch 2.10 environment
- Models: Qwen3.8-27B NVFP4 / FP8 / W8A16-FP8 + MTP4 or DFlash2; Qwen3.8-Flash-Next NVFP4 + MTP
部署方式
Deployment
- pip 安装 wheel
- Docker:一个镜像启动两个模型
- 27B 三种权重:NVFP4、FP8、W8A16-FP8,均支持 MTP4 / DFlash2
- 1Cat Studio:图形界面一键加载
- OpenAI / Anthropic 兼容接口:工具调用、结构化输出、思考模式、图片输入
- pip-install the wheel
- Docker: one image serves both models
- Three 27B weight formats: NVFP4, FP8, W8A16-FP8, each with MTP4 / DFlash2
- 1Cat Studio: one-click loading in a GUI
- OpenAI / Anthropic-compatible API: tool calls, structured output, thinking, image input
硬件要求
Requirements
- 4× V100-SXM2-32GB(NVLink)
- 主机内存 64 GB 以上;Flash-Next 1.5.4 建议更多(启动时占用约 47 GB)
- 最大 256K 上下文、8 并发
- 有编译缓存时 27B 启动约 2 分钟、Flash-Next 约 6.5 分钟;空缓存约 4 / 9 分钟
- 4× V100-SXM2-32GB (NVLink)
- 64 GB+ host memory; more is advisable for Flash-Next 1.5.4 (about 47 GB in use at startup)
- Up to 256K context, 8 concurrent requests
- With a warm compile cache, startup takes about 2 min for 27B and 6.5 min for Flash-Next; about 4 / 9 min cold