1.5.3 新特性
What's new in 1.5.3
27B 现在有两种投机解码可选:DFlash2 需要单独的草稿模型,单请求最快;原生 MTP4 只用模型自带的预测头,不占额外显存,并发和长文更强。两种方式的全部加速路径都默认开启。
27B now has two speculative modes. DFlash2 needs a separate draft model and is fastest for one request. Native MTP4 uses the model's own prediction head, takes no extra memory, and is stronger with concurrency and long context. Every acceleration path is on by default for both.
27B:MTP4 与 DFlash2 每轮耗时
27B: MTP4 vs DFlash2 step time
温度 0,每轮 ms(越低越好),括号内为实测每路 tok/s
Temperature 0, ms per step (lower is better); measured tok/s per request in brackets
MTP4 每轮最多出 5 个 token,DFlash2 最多 8 个:单请求 DFlash2 更快,多请求和长文 MTP4 每轮更省。MTP4 不加载草稿模型,KV 容量 1,598,295 token,比 DFlash2 多 22%。
MTP4 yields up to 5 tokens per step, DFlash2 up to 8: DFlash2 is faster for one request, MTP4 steps are cheaper with many requests and long context. Without a draft model MTP4 has 1,598,295 KV tokens, 22% more than DFlash2.
默认参数更接近发布配置
Defaults closer to the release config
- MTP4 一个参数开全:
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'即启用单图草稿、精修 head、GDN 验证与回放、长文 E4M3 grouped attention - 前缀缓存自动用发布网格:混合架构模型自动设 2048 块、部分前缀命中、8192 保留间隔,FP8 KV 不再出现无法对齐的 1648 块(#1422)
- 27B 直接
vllm serve能启动:编译模式下的设备能力检查不再打断整图编译(#1496、#1498);Flash-Next 仍需--max-num-seqs 8 - 显式参数永远优先:写明的 0 或关闭不会被默认值覆盖(#1500)
- One flag for all of MTP4:
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'turns on the single-graph draft, refined head, GDN verify and replay, and long-context E4M3 grouped attention - Release cache geometry by default: hybrid models get 2048 blocks, partial prefix hits and an 8192 retention interval; FP8 KV no longer ends up with an unalignable 1648 block (#1422)
- 27B starts with a plain
vllm serve: device-capability checks no longer break full-graph compilation (#1496, #1498); Flash-Next still needs--max-num-seqs 8 - Explicit flags always win: an explicit 0 or off is never replaced by a default (#1500)
修掉的崩溃与命中问题
Crashes and lost hits fixed
- 投机解码下发图片崩溃:图片编码结果要等对应 token 被确认后才释放(#1485)
- Flash-Next 主机 KV 预热越界:预热只在有界状态池里分配(#1490)
- 前缀缓存空池崩溃:部分前缀快照按分区配额检查空闲块(#1483)
- Flash-Next 多会话 64K 不命中:主机 KV 模式不再挤掉尾部检查点(#1500)
- 长 prompt 只改末尾时不命中(1.5.2 已知问题):27B 与 Flash-Next 都恢复部分命中(#1401)
- DFlash2 64K 4 并发:新 attention kernel 不再拖慢 8 行校验,每轮 37.1→32.7 ms(#1505)
- Image input crash under speculation: encoder outputs are freed only after their tokens are confirmed (#1485)
- Flash-Next host-KV warmup out of range: warmup allocates only inside the bounded state pool (#1490)
- Empty-pool crash in prefix caching: partial-prefix snapshots check free blocks per partition (#1483)
- Flash-Next multi-session 64K misses: host-KV mode keeps its tail checkpoints (#1500)
- Edited-tail prompts missed the cache (1.5.2 known issue): partial hits are back for 27B and Flash-Next (#1401)
- DFlash2 at 64K with 4 users: the new attention kernel no longer slows the 8-row verifier, 37.1→32.7 ms per step (#1505)
测试口径
How this was measured
- 默认时钟:本页全部数据在显存 877 MHz、功耗上限 300 W 下测得
- 与 1.5.2 不可直接对比:1.5.2 报告在显存 1107 MHz、功耗上限 350 W 下测得,时钟更高,两份报告的速度数字不能直接相比
- 同一套工具:测试矩阵、数据集与质量评测和 1.5.2 完全相同
- 零错误:三个模型配置的全部测试中没有一次错误或显存溢出
- Stock clocks: every number here was measured at 877 MHz memory and a 300 W power limit
- Not directly comparable with 1.5.2: the 1.5.2 report ran at 1107 MHz memory and 350 W, so its speed numbers cannot be compared one to one with this page
- Same tools: the matrix, datasets and quality evaluation are identical to 1.5.2
- Zero errors: no error or out-of-memory in any run across the three configurations
DeepSeek-V4.1-Flash:16 张 V100 跑 1M 上下文
DeepSeek-V4.1-Flash: 1M context on 16 V100s
原版 checkpoint 在 4 台 Dell C4140(每台 4 张 V100)上以流水线 4 段 × 张量并行 4 运行,DSpark 投机解码,最大上下文 1M(#1482)。单请求约 202 tok/s,64 个请求同时输出合计约 1860 tok/s,512K 长文单请求约 165 tok/s。
The original checkpoint runs on four Dell C4140 servers (4 V100s each) as 4 pipeline stages × tensor parallel 4, with DSpark speculative decoding and up to 1M context (#1482). One request decodes at about 202 tok/s, 64 requests at about 1860 tok/s in total, and one request at 512K at about 165 tok/s.
decode 速度
Decode speed
1K 上下文,温度 0。DSpark 每轮 5 个草稿,实测平均每轮产出约 3.9 个 token(各档 3.5–4.4),据此由每轮耗时换算 tok/s
1K context, temperature 0. DSpark proposes 5 drafts per step and yields about 3.9 tokens per step on average (3.5–4.4 across levels); tok/s is converted from step time with that figure
| 并发 | Users | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---|---|---|---|---|---|---|---|
| 每路 tok/s | tok/s each | 202 | 198 | 157 | 111 | 79 | 50 | 29 |
| 合计 tok/s | Total tok/s | 202 | 396 | 628 | 888 | 1264 | 1600 | 1856 |
| 每轮 ms | ms/step | 19.3 | 19.7 | 24.8 | 35.1 | 49.5 | 77.5 | 135.7 |
长文单请求:512K 每轮 23.7 ms,约 165 tok/s(修复前 50.9 ms);512K 首次 prefill 132.7 s。
Long context, one request: 23.7 ms per step at 512K, about 165 tok/s (50.9 ms before the fix); first prefill of 512K takes 132.7 s.
验证与说明
Validation and notes
- 正确性:大海捞针 9/9,GSM8K 297/300(16 并发)
- 稳定性:持续压测 722 个请求,错误 0
- 长文提速:流水线下每步不再复制整段历史与满宽块表,512K 每轮 50.9→23.7 ms
- 测试口径:数据来自 #1482 合并时的验证(11-11,C4140 显存 1053 MHz),这次没有用 1.5.3 wheel 在 C4140 上复测
- Correctness: needle in a haystack 9/9, GSM8K 297/300 (16 users)
- Stability: 722-request soak, 0 errors
- Long-context speedup: pipeline steps no longer copy the whole history and full-width block tables; 512K goes from 50.9 to 23.7 ms per step
- Measurement: numbers come from validating #1482 at merge time (11-11, C4140 memory at 1053 MHz); they were not re-run on C4140 with the 1.5.3 wheel
这一版做了什么
What changed
1.5.2 之后合并的 105 个 PR,按类别列出修复与优化;每一条都链接到对应的 PR 或 issue。完整清单见下方“完整测试数据”里的“全部 PR”。
The 105 PRs merged since 1.5.2, grouped into fixes and optimizations; every item links to its PR or issue. The complete list is under "All PRs" in the data room below.
27B 原生 MTP:不用草稿模型也能投机解码
Native MTP for 27B: speculative decoding without a draft model
- MTP4 大幅提速: 单图草稿、精修 head、CUDA GDN 验证延迟提交、GDN 投机回放、QPN2 投影,TP2/TP4、NVFP4/FP8 共用同一套算子(#1472、#1478、#1491)。
- 长上下文走 E4M3 grouped 快速注意力: 覆盖到服务的整个上下文窗口(#1425),多请求(2–8 路)同样走快速路径,FP32 累加。
- 全部默认开启: 只写
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'即启用全部加速,不需要额外参数或环境变量。 - MTP 精简草稿词表(#1218、#1468)。
- Much faster MTP4: single-graph draft, refined head, CUDA GDN verify with deferred commit, GDN speculative replay and QPN2 projections, sharing one set of operators across TP2/TP4 and NVFP4/FP8 (#1472, #1478, #1491).
- Long context on the E4M3 grouped fast attention: covers the whole served context window (#1425). Concurrent requests (2–8) use the same fast path, with FP32 accumulation.
- All on by default:
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'enables every acceleration, with no extra flags or environment variables. - Reduced MTP draft vocabulary (#1218, #1468).
前缀缓存默认就是发布配置
Prefix caching defaults to the release config
不加缓存参数时,混合架构模型现在会自动用上发布配置:27B 直接 vllm serve <模型> --tensor-parallel-size 4 即可启动;Flash-Next 仍需加 --max-num-seqs 8(见已知限制第 1 条)。
- 混合架构前缀缓存默认使用统一的 2048 网格: 自动设置 block / mamba block 2048、部分前缀缓存、8192 保留间隔,Flash-V100 页对齐到 64 的倍数,FP8 KV 不再出现 1648 这种无法对齐的块(#1422、#1492)。
- 编译模式下的设备能力检查不再打断 torch.compile 全图,不再因此启动失败(#1496、#1498)。
- 显式参数始终优先:
--prefix-cache-retention-interval 0、--no-partial-prefix-cache等用户写明的值不会被默认值覆盖(#1500)。
Without cache flags, hybrid models now get the release configuration: 27B starts with a plain vllm serve <model> --tensor-parallel-size 4; Flash-Next still needs --max-num-seqs 8 (see known limitation 1).
- One 2048 grid for hybrid prefix caching by default: block and mamba block 2048, partial prefix cache and an 8192 retention interval are set automatically. Flash-V100 pages align to a multiple of 64, so FP8 KV no longer ends up with an unalignable 1648-token block (#1422, #1492).
- Device-capability checks no longer break the torch.compile full graph, so startup no longer fails on them (#1496, #1498).
- Explicit flags always win: values you set, such as
--prefix-cache-retention-interval 0or--no-partial-prefix-cache, are never replaced by a default (#1500).
前缀缓存:命中率与稳定性
Prefix caching: hit rate and stability
- Flash-Next 多会话 64K 命中恢复: 主机 KV 模式下不再默认周期保留,尾部检查点不会被挤掉(#1500)。
- 长混合 prompt 的部分前缀命中恢复(27B 与 Flash-Next,#1401)。
- 空池不再崩溃: 部分前缀快照与命中按分区配额检查可用块(#1483、#1492)。
- 释放的哈希块可选保留以便复用,并在分区块池中设上限(#1444、#1466、#1476)。
- 投机回滚时保留超额分配的 mamba 块(#1431)。
- Flash-Next multi-session 64K hits restored: host-KV mode no longer applies periodic retention by default, so tail checkpoints are not evicted (#1500).
- Partial prefix hits restored on long hybrid prompts (27B and Flash-Next, #1401).
- No crash on an empty pool: partial-prefix snapshots and hits check free blocks against partition quotas (#1483, #1492).
- Optional retention of freed hashed blocks for reuse, bounded across partitioned block pools (#1444, #1466, #1476).
- Over-allocated mamba blocks are kept when speculation rolls back (#1431).
稳定性修复
Stability fixes
- 投机解码下图片输入崩溃: 编码器输出要等对应 token 被确认后才释放(#1485、#1494)。
- Flash-Next 主机 KV 启动预热越界: MRV2 预热只在有界状态池内分配(#1490、#1493)。
- MRV2 预热覆盖所有请求数、混合步和满块,服务中不再首次编译(#1402、#1420)。
- TileLang 没有 nvcc 时 GDN prefill 自动用预编译 kernel(#1400)。
- GDN 纯 decode 批次中非投机行的 a/b 门控(#1432),运行时状态、工具解析与 NVFP4 工作区修复(#1464)。
- 派生草稿配置共用目标模型的运行资源(#1418),LoRA 目标 logits 在投机解码中保留(#1261)。
- 退出不再卡住: MessageQueue 的 zmq 资源由 finalizer 关闭(#1406)。
- Image input crash under speculative decoding: encoder outputs are freed only after their tokens are confirmed (#1485, #1494).
- Flash-Next host-KV warmup out of range: MRV2 warmup allocates only inside the bounded state pool (#1490, #1493).
- MRV2 warmup covers every request count, mixed steps and a full chunk, so nothing compiles for the first time while serving (#1402, #1420).
- GDN prefill falls back to the prebuilt kernel when TileLang has no nvcc (#1400).
- GDN a/b gates for non-speculative rows in pure-decode batches (#1432), runtime state, tool parsing and NVFP4 workspace fixes (#1464).
- Derived drafter configs share the target's runtime resources (#1418), LoRA-aware target logits are kept in speculative decoding (#1261).
- No more exit hang: MessageQueue zmq resources are closed by a finalizer (#1406).
API:Anthropic 接口、流式与推理统计
API: Anthropic endpoint, streaming and reasoning usage
- Anthropic Messages 接口:
message_start带 type/role、input_tokens 只计新输入、SSE 帧与 ID 完整保留、返回实测缓存用量(#1428、#1440、#1436)。 - 流式工具调用与推理 usage 补全(#1443),prompt 中已有起始标记时也统计推理 token(#1433)。
- 可选空闲心跳,长思考时连接不被代理断开(#1437),HEAD 存活探测与可选启动日志(#1438)。
- 被拒绝的请求直接返回 4xx,不再打印服务端 traceback(#1399)。
- 单请求阶段耗时统计(#1467),解析器与模板热加载先准备再发布(#1439)。
- Anthropic Messages endpoint:
message_startcarries type and role, input_tokens counts only fresh input, SSE frames and IDs are preserved, and measured cache usage is reported (#1428, #1440, #1436). - Deferred streaming tool calls and reasoning usage are completed (#1443), and reasoning tokens are counted when the start marker is in the prompt (#1433).
- Optional idle heartbeats keep proxies from dropping the connection during long reasoning (#1437); HEAD liveness probe and opt-in startup logs (#1438).
- Rejected requests get a 4xx without a server traceback (#1399).
- Per-request phase timings (#1467); parser and template reloads are prepared before publication (#1439).
速度
Speed
- DFlash2 每请求 8 行的校验继续用 48 行 kernel: MTP 的新 attention kernel 不再拖慢 DFlash2 长文,64K 4 并发每轮 37.1→32.7 ms(#1505)。
- DFlash2 采样图在不同温度之间复用(#1442),带惩罚项时保持紧凑拒绝采样(#1398)。
- 短 prompt 用小块 prefill,长 prompt 保持大块(#1416),长上下文 decode 分桶更细(#1417)。
- 多 KV 头的 grouped FP16 验证(#1345),QPN8 M=9–32 decode 单遍读权重(#1069、#1157、#1469)。
- Flash-Next PLE: 预取下一个 prefill 块的 PLE 收集、限制映射页预取(#1217、#1235、#1477);page4 QSA prefill 原生路径覆盖 12/24 头(#1236)。
- mHC / KDA 投影与验证 kernel(#1159、#1165、#1166、#1325、#1471)。
- The DFlash2 q8 verifier keeps the 48-row kernel: the new MTP attention kernel no longer slows DFlash2 at long context; 64K with 4 users goes from 37.1 to 32.7 ms per step (#1505).
- DFlash2 sampler graphs are reused across temperatures (#1442), and compact target rejection is kept with penalties (#1398).
- Short prompts prefill in smaller chunks, long prompts keep large chunks (#1416); finer decode context buckets for long contexts (#1417).
- Grouped FP16 verify with several KV heads per rank (#1345); QPN8 M=9–32 decode reads the weights once (#1069, #1157, #1469).
- Flash-Next PLE: the next prefill chunk's PLE gather overlaps the current one, and mapped page prefetch is bounded (#1217, #1235, #1477); native page4 QSA prefill covers 12/24-head ranks (#1236).
- mHC / KDA projection and verification kernels (#1159, #1165, #1166, #1325, #1471).
DeepSeek-V4.1-Flash:16 张 V100 原生 PP4 × TP4
DeepSeek-V4.1-Flash: native PP4 × TP4 on 16 V100s
- 原版 checkpoint 在 4 台 C4140(16× V100)上 PP4 × TP4 运行,1M 上下文,含并发与长文 decode 优化(#1482)。
- packed KV 字节恢复、投机回退区预留(#1381),wo_a 单次 grouped GEMM(#1273),融合 KV 压缩的栈用量上限(#1344)。
- 流水线并行: 空步保留常驻请求行、急切发送与缓冲区所有权重叠(#1373、#1294),GGUF TP4 加载与运行内存降低(#1020)。
- The original checkpoint runs PP4 × TP4 on four C4140 servers (16× V100) with 1M context, including concurrency and long-context decode work (#1482).
- Packed KV bytes restored and the speculative rewind span reserved (#1381), one grouped GEMM for wo_a (#1273), bounded stack use in fused KV compression (#1344).
- Pipeline parallel: persistent request rows survive empty steps, and eager sends overlap with explicit buffer ownership (#1373, #1294); lower GGUF TP4 loading and runtime memory (#1020).
上游回移、构建与平台
Upstream backports, build and platforms
- 上游修复回移: 多模态 EXIF 哈希与视觉 RoPE、persistent top-k、GLM-5.3-Flash kpool、PP 下的多模态(#1415、#1411、#1410、#1365)。
- 构建: 修复只编 SM70 时的构建失败(#1322),E4M3 GEMM 支持量化组 64/32(#1412),FP8 bridge 工作区按 KV 头组分配(#1321)。
- Turing(SM75): FlashQLA TileLang prefill(#1228),FA2 固定 split-wave 策略(#1346)。
- 发布: wheel 自带的启动脚本更新到 1.5.2 配置(#1397),Docker 镜像内置 CUDA 12.8 编译器(#1395),DS-V4.1 配置中移除硬编码 API key(#1396)。
- Upstream backports: multimodal EXIF hashing and vision RoPE, persistent top-k, GLM-5.3-Flash kpool, multimodal under PP (#1415, #1411, #1410, #1365).
- Build: SM70-only builds fixed (#1322), E4M3 GEMM at quantization groups 64 and 32 (#1412), FP8 bridge workspace budgeted by KV-head group (#1321).
- Turing (SM75): FlashQLA TileLang prefill (#1228), FA2 pinned to the split-wave heuristic (#1346).
- Release: the bundled launchers match the 1.5.2 profiles (#1397), the Docker image ships the CUDA 12.8 compiler (#1395), and the hard-coded API key is gone from the DS-V4.1 config (#1396).
速度全景
Speed at a glance
投机解码的速度取决于草稿被接受的比例,而这个比例随内容变化很大。为了让不同场景可以直接比较,并发与长文两张图把实测的每轮耗时统一按 70% 平均接受率折算(DFlash2 每轮 5.9 个 token,MTP4 每轮 3.8 个);数据集一图是真实题目上的实测速度。
Speculative speed depends on how many drafts are accepted, which varies with content. To compare scenarios, the concurrency and long-context charts convert measured step times at a 70% average acceptance (5.9 tokens per step for DFlash2, 3.8 for MTP4). The dataset chart shows measured speed on real prompts.
并发越多,总吞吐越高
More users, more total throughput
8K 上下文,温度 0,tok/s;柱顶小字为每路速度
8K context, temperature 0, tok/s; per-request speed above each bar
长文掉速有多少
How much long context costs
单请求 tok/s,1K 到 250K 上下文
One request, tok/s, 1K to 250K context
真实数据集单请求速度
One request on real datasets
实测 tok/s,括号内为草稿接受率
Measured tok/s; draft acceptance in brackets
长对话:第二轮几乎不用等
Long chats: the next turn barely waits
27B 首字时间(秒,对数刻度):首次输入 vs 多轮追加命中缓存
27B time to first token (s, log scale): first prompt vs cached follow-up
压到 4 bit,智商不掉
4-bit weights, same intelligence
与原版 27B BF16 模型在同一套题、同样的贪心解码下逐题对比。两个 NVFP4 模型与 BF16 的差距都在统计误差内或更好,所有输出中没有乱码,也没有重复循环。
Compared question by question with the original 27B BF16 model, same prompts, greedy decoding. Both NVFP4 models are within statistical error of BF16 or better, with no garbled text or repetition loops in any output.
准确率
Accuracy
怎么保证不降智
How precision is kept
- 只舍入一次:FP4 权重解码到 FP16 只舍入一次,矩阵乘全程 FP32 累加
- 关键状态高精度:logits、线性注意力状态都用 FP32
- KV 校准:FP8 E4M3 KV,按层加载校准系数
- 逐 token 对齐:27B 与 BF16 出现分叉时,BF16 的优势中位数仅 0.2(logprob),从未选到 BF16 前 5 名之外的 token
- 缓存不改变输出:同一请求命中前缀缓存与冷算的输出逐字一致
- One rounding: FP4 weights decode to FP16 once; GEMMs accumulate in FP32
- High-precision state: logits and linear-attention state stay in FP32
- Calibrated KV: FP8 E4M3 KV with per-layer scales
- Token alignment: where 27B diverges from BF16, BF16's margin has a median of 0.2 logprob, and the chosen token is always in BF16's top 5
- Cache does not change output: a request served from the prefix cache matches the cold computation token for token
为什么这么快
Why it is fast
一轮出多个 token
Several tokens per step
DFlash2 草稿模型一次提出 7 个 token,目标模型一轮全部验证。写代码时(HumanEval,温度 0)平均每轮输出 6.9 个 token;不想加载草稿模型时,MTP4 每轮平均 4.6 个。
The DFlash2 draft proposes 7 tokens and the target verifies them in one step. On code (HumanEval, temperature 0) each step yields 6.9 tokens on average; without a draft model, MTP4 yields 4.6.
权重小,带宽跑满
Small weights, full bandwidth
NVFP4 权重只有 FP16 的约 1/3.5。V100 没有 FP4 硬件,我们用自研算子在寄存器里解码,FP32 累加,一次读权重就验证一整轮草稿。
NVFP4 weights are about 1/3.5 the size of FP16. V100 has no FP4 hardware, so our kernels decode in registers with FP32 accumulation, and one pass over the weights verifies a whole round of drafts.
KV 历史放主机内存
KV history in host memory
Flash-Next 只把最近 32K 的 KV 留在显存,更早的历史以 FP8 放进主机内存,由稀疏注意力按需读取;48 层中 36 层是线性注意力,所以 250K 时只比 1K 慢 16%。
Flash-Next keeps the latest 32K of KV on the GPU and older history in host memory as FP8, read on demand by sparse attention; 36 of 48 layers use linear attention, so 250K is only 16% slower than 1K.
完整测试数据
Complete test data
发布前在同一台 4×V100 上跑完的全部测试。纯 decode 计时只统计所有请求都在 decode 的步,不含 prefill;所有测试零错误、零显存溢出。
Every test run on the same 4×V100 before release. Pure decode timing counts only steps where all requests are decoding, excluding prefill. Zero errors and zero out-of-memory in all runs.
启动参数
Launch commands
本页每组数据对应的完整启动参数,复制即可使用。模型路径按实际目录修改。
The complete launch arguments behind every result on this page, ready to copy. Adjust the model paths to your directories.
27B · DFlash2
27B · DFlash2
与 Docker(MODEL=27b)和 Studio 预设完全相同的参数。草稿模型放在本地时,把 model 改成本地路径。
Exactly the arguments of Docker (MODEL=27b) and the Studio preset. Point model at a local path for a downloaded draft.
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-dflash2 \
--tensor-parallel-size 4 --dtype half --kv-cache-dtype fp8_e4m3 \
--max-model-len 262144 --max-num-batched-tokens 4096 --max-num-seqs 8 \
--gpu-memory-utilization 0.83 --attention-backend FLASH_ATTN_V100 \
--enable-prefix-caching --partial-prefix-cache --mamba-cache-mode align \
--block-size 2048 --mamba-block-size 2048 --prefix-cache-retention-interval 8192 \
--mixed-prefill-step-latency-ms 0 --enable-prompt-tokens-details \
--limit-mm-per-prompt '{"video": 0, "audio": 0}' --trust-remote-code --seed 0 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--default-chat-template-kwargs '{"enable_thinking":true}' \
--speculative-config '{"method": "dflash", "model": "/models/Qwen3.8-27B-DFlash2", "revision": "dedf8df68adfb1afeaf7b7480c0a0243108177b4", "num_speculative_tokens": 7, "kv_cache_dtype": "auto", "attention_backend": "FLASH_ATTN_V100", "draft_sample_method": "probabilistic", "enforce_eager": false}'
# 或者用 wheel 自带的启动脚本 / or the bundled launcher:
serve_qwen38_27b_nvfp4_v100.sh /models/Qwen3.8-27B-NVFP4 --draft /models/Qwen3.8-27B-DFlash2 --port 800027B · MTP4(不需要草稿模型)
27B · MTP4 (no draft model)
本页 27B MTP4 数据用的就是这组参数。前缀缓存的块大小、部分命中和保留间隔会自动设好。
The arguments behind every 27B MTP4 number on this page. Prefix-cache block sizes, partial hits and the retention interval are set automatically.
vllm serve /models/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b-mtp \
--tensor-parallel-size 4 --dtype half --kv-cache-dtype fp8_e4m3 --max-model-len 262144 \
--max-num-seqs 8 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.83 \
--attention-backend FLASH_ATTN_V100 --enable-prefix-caching --mixed-prefill-step-latency-ms 0 \
--trust-remote-code --limit-mm-per-prompt '{"video": 0, "audio": 0}' --enable-prompt-tokens-details \
--enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 4}'Flash-Next · MTP4 + 主机内存 KV
Flash-Next · MTP4 + host-memory KV
与 Docker(MODEL=flashnext)、Studio 预设和 wheel 自带的 serve_flash_next_nvfp4_v100.sh 相同。--max-num-seqs 8 不能省(已知限制第 1 条)。
Same as Docker (MODEL=flashnext), the Studio preset and the bundled serve_flash_next_nvfp4_v100.sh. Keep --max-num-seqs 8 (known limitation 1).
vllm serve /models/Qwen3.8-Flash-Next-NVFP4 --served-model-name qwen3.8-flash-next-nvfp4 \
--tensor-parallel-size 4 --dtype half --kv-cache-dtype float16 --language-model-only \
--max-model-len 262144 --max-num-batched-tokens 4096 --max-num-seqs 8 \
--gpu-memory-utilization 0.89 --attention-backend FLASH_ATTN_V100 \
--enable-prefix-caching --partial-prefix-cache --block-size 2048 --mamba-block-size 2048 \
--mixed-prefill-step-latency-ms 0 --mamba-ssm-cache-dtype float32 --enable-prompt-tokens-details \
--trust-remote-code --seed 0 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking":false}' \
--kernel-config '{"sm70_hcx": true, "sm70_qsa_prep": true, "sm70_hcx_local_schedule": true, "qsa_host_kv": true, "qsa_host_kv_dtype": "fp8_e4m3", "qsa_host_kv_hot_dtype": "fp8_e4m3", "qsa_host_kv_hot_tokens": 32768, "qsa_host_kv_prefill_stage_tokens": 262144}' \
--speculative-config '{"method": "mtp", "model": "/models/Qwen3.8-Flash-Next-mtp", "num_speculative_tokens": 4, "draft_sample_method": "greedy"}'
# 或者 / or:
serve_flash_next_nvfp4_v100.sh /models/Qwen3.8-Flash-Next-NVFP4 --draft /models/Qwen3.8-Flash-Next-mtp --port 8000Docker
Docker
一个镜像启动两个模型。模型目录名必须与 README-docker.md 一致;--ipc=host 和 --ulimit memlock=-1 不能省。
One image serves both models. Model directory names must match README-docker.en.md; keep --ipc=host and --ulimit memlock=-1.
tar -xzf 1cat-vllm-1.5.3-docker.tar.gz && cd 1cat-vllm-1.5.3-docker sha256sum -c SHA256SUMS docker build -t 1cat/vllm-v100:1.5.3 . # 27B (DFlash2) docker run -d --runtime=nvidia --gpus all --network host --ipc=host --ulimit memlock=-1 \ -e MODEL=27b -e PORT=8000 -v /data/models:/models:ro -v vllm-cache-27b:/cache 1cat/vllm-v100:1.5.3 # Flash-Next docker run -d --runtime=nvidia --gpus all --network host --ipc=host --ulimit memlock=-1 \ -e MODEL=flashnext -e PORT=8000 -v /data/models:/models:ro -v vllm-cache-fn:/cache 1cat/vllm-v100:1.5.3 # 可选 / optional: -e MAX_NUM_SEQS=8 -e MAX_MODEL_LEN=262144 -e VLLM_API_KEY=... -e EXTRA_ARGS="..."
DeepSeek-V4.1-Flash · PP4 × TP4(4 台 × 4 张 V100)
DeepSeek-V4.1-Flash · PP4 × TP4 (4 nodes × 4 V100)
本页 DeepSeek 数据用的完整加速配置。启动器和集群脚本在仓库 benchmarks/deepseek_v41/native/:run_pp4.sh 已带上 InfiniBand/NCCL 设置和其余默认加速开关;下面这组开关在代码里默认关闭,必须显式设置。PIN_* 是 C4140 的 CPU 绑核,换机器请按 NUMA 布局调整。
The full acceleration config behind the DeepSeek numbers on this page. The launcher and cluster scripts live in benchmarks/deepseek_v41/native/: run_pp4.sh already sets InfiniBand/NCCL and the remaining default acceleration switches; the switches below are off in code and must be set explicitly. PIN_* pins CPU cores on the C4140; adjust them to your NUMA layout.
# 1) 每台节点的加速开关 / acceleration switches on every node
export V41_CONFIG=~/v41/serve-native-pp4-long.yaml
export VLLM_V41_BATCH=1 VLLM_V41_WINDOW_RING=0 VLLM_V41_PP_MICRO=1 VLLM_V41_PP_AHEAD=1 \
VLLM_V41_LAZY_OUTPUT_IDS=1 VLLM_V41_PROPOSE_GRAPH=1 VLLM_V41_INDEX_SHARD=1 \
VLLM_V41_PREFILL_PIPELINE=1 VLLM_V41_HC_GEMM=1 VLLM_V41_LIVE_TABLE=1 VLLM_V41_FAST_READD=1 \
VLLM_V41_TOPP_NO_FENCE=1 \
VLLM_V41_PIN_CORES=2,4,3,5 VLLM_V41_PIN_NUMA=0,0,1,1 VLLM_V41_PIN_ENGINE=20,22,60,62
# 2) 依次启动 rank 3、2、1、0(rank 0 = API 节点)/ start ranks 3, 2, 1, then 0 (API node)
~/v41/run_pp4.sh <rank> # 仓库的 cluster/start.sh 可一次启动 4 个节点 / cluster/start.sh starts all four
# 3) serve-native-pp4-long.yaml (1M context)
# DeepSeek-V4.1-Flash, 1M context, PP4 x TP4 (4 nodes x 4 V100). Stage k = node k.
model: /home/ymzx/models/ds41-1017
tokenizer: /home/ymzx/models/ds41-1017
tokenizer-mode: deepseek_v41
served-model-name: deepseek-v4.1-flash
host: 0.0.0.0
port: 18000
dtype: half
tensor-parallel-size: 4
pipeline-parallel-size: 4
distributed-executor-backend: mp
max-model-len: 1048576
max-num-seqs: 64
max-num-batched-tokens: 4096
kv-cache-memory-bytes: 4750000000
gpu-memory-utilization: 0.96
compilation-config:
mode: 0
cudagraph_mode: FULL_DECODE_ONLY
cudagraph_capture_sizes: [6, 12, 18, 24, 30, 36, 42, 48, 60, 72, 84, 96, 120, 144, 192]
max_cudagraph_capture_size: 192
no-async-scheduling: true
no-enable-prefix-caching: true
enable-chunked-prefill: true
no-disable-custom-all-reduce: true
reasoning-parser: deepseek_v4
default-chat-template-kwargs: '{"thinking": true}'
model-loader-extra-config:
engram_path: /home/ymzx/models/ds41-1017/engram
speculative-config:
method: dspark
model: /home/ymzx/models/ds41-1017-dspark
quantization: fp8
num_speculative_tokens: 5
draft_load_config:
load_format: safetensors
enforce_eager: false已知限制
Known limitations
- Flash-Next 不加参数直接 vllm serve 会启动失败:默认最大并发 256 超过能放下的 Mamba 状态块数,请加 --max-num-seqs 8;Docker、Studio 和自带启动脚本已经带上。下一版会根据显存自动压低。
- Flash-Next 250K 历史的多轮追问首字变慢:完全命中 2.8 s、多轮追加 4.6 s(1.5.2 约 2.1 s);128K 及以下比 1.5.2 更快。原因还在排查。
- MTP 在温度大于 0 时每轮更慢:Flash-Next 单请求每轮多约 3.5 ms,8 并发时没有差别;27B MTP4 单请求多约 2.7 ms,8 并发仍多约 3 ms。
- 27B(DFlash2)KV 容量比 1.5.2 少 8.8%:启动时为图片输入预留了更多显存,1,434,850 → 1,308,270 token;MTP4 不加载草稿,有 1,598,295 token。
- Flash-Next 短对话缓存粒度 2048 token:不足一个块的短对话不会命中。
- 最大并发 8:这是 256K 配置的上限,更高并发需要缩短上下文。
- 长 prompt 进来时正在输出的请求会短暂变慢:prefill 和 decode 目前在同一步中混合执行。
- 新引擎第一个温度大于 0 的请求慢约 3 秒:采样相关的 Triton kernel 在这时才首次编译(27B 4.4 s,之后 1.5–1.6 s)。
- Flash-Next fails to start with a bare vllm serve: The default of 256 sequences exceeds the Mamba state blocks that fit; pass --max-num-seqs 8. Docker, Studio and the bundled launcher already do. The next release will lower it automatically.
- Flash-Next follow-ups on 250K history are slower: 2.8 s for a full hit and 4.6 s for a follow-up (about 2.1 s in 1.5.2); 128K and below are faster than 1.5.2. Under investigation.
- MTP steps are slower above temperature 0: Flash-Next adds about 3.5 ms per step for one request and nothing at 8 users; 27B MTP4 adds about 2.7 ms for one request and about 3 ms at 8 users.
- 27B (DFlash2) has 8.8% less KV than 1.5.2: More memory is reserved at startup for image input, 1,434,850 → 1,308,270 tokens; MTP4 loads no draft and has 1,598,295.
- Flash-Next caches short chats at 2048-token granularity: A conversation shorter than one block does not hit.
- At most 8 concurrent requests: This is the limit of the 256K configurations; more requires a shorter context.
- Running requests slow down briefly when a long prompt arrives: Prefill and decode still share a step.
- On a fresh engine the first request above temperature 0 takes about 3 s longer: The sampling Triton kernels compile then (27B: 4.4 s, then 1.5–1.6 s).
测试环境与部署
Test setup and deployment
测试环境
Test setup
- 4× Tesla V100-SXM2-32GB(NVLink),张量并行 4
- 默认时钟:显存 877 MHz,核心最高 1530 MHz,功耗上限 300 W,ECC 开
- 主机内存 62 GB
- Qwen3.8-27B NVFP4(Unsloth)+ DFlash2 或原生 MTP4;Qwen3.8-Flash-Next NVFP4 + MTP4
- wheel 构建自 main
cb97f379b
- 4× Tesla V100-SXM2-32GB (NVLink), tensor parallel 4
- Stock clocks: memory 877 MHz, core up to 1530 MHz, 300 W, ECC on
- 62 GB host memory
- Qwen3.8-27B NVFP4 (Unsloth) + DFlash2 or native MTP4; Qwen3.8-Flash-Next NVFP4 + MTP4
- Wheel built from main
cb97f379b
部署方式
Deployment
- pip 安装 wheel
- Docker:一个镜像启动两个模型
- 27B 两种投机解码:DFlash2 或原生 MTP4
- 1Cat Studio:图形界面一键加载
- OpenAI 兼容接口:工具调用、结构化输出、思考模式、图片输入(27B)
- pip-install the wheel
- Docker: one image serves both models
- Two speculative modes for 27B: DFlash2 or native MTP4
- 1Cat Studio: one-click loading in a GUI
- OpenAI-compatible API: tool calls, structured output, reasoning, image input (27B)
硬件要求
Requirements
- 4× V100-SXM2-32GB(NVLink)
- 主机内存 64 GB 以上
- 最大 256K 上下文、8 并发
- 27B 启动约 2 分钟,Flash-Next 约 6 分钟(有编译缓存)
- 4× V100-SXM2-32GB (NVLink)
- 64 GB+ host memory
- Up to 256K context, 8 concurrent requests
- Startup about 2 min for 27B and 6 min for Flash-Next (warm compile cache)