看它写代码
Watch it write code
两个模型在 HumanEval 上的真实输出(19 道题,全部通过单元测试),按 1.5.2 实测的速度和投机解码节奏逐轮输出:每一轮验证一批草稿 token,被接受的会一起出现(高亮)。自动循环播放;这是实测回放,不在你的浏览器里运行模型。
The real HumanEval outputs of both models (19 problems, all passing their tests), replayed at the measured 1.5.2 speed and speculative rhythm: each step verifies a batch of draft tokens, and the accepted ones appear together (highlighted). It loops automatically. This is a replay of measurements; no model runs in your browser.
27B 写完一个 200 token 的完整函数(带文档和注释)
27B writes a complete 200-token function with docstring and comments
写代码时每一轮读一遍权重,平均产出 7.1 个 token
On code, each pass over the weights yields 7.1 tokens on average
27B 约 7.8 秒写完 19 道 HumanEval 题的 token 数,全部通过测试
Tokens 27B writes for 19 HumanEval problems in about 7.8 s, all passing
速度全景
Speed at a glance
投机解码的速度取决于草稿被接受的比例,而这个比例随内容变化很大。为了让不同场景可以直接比较,并发与长文两张图把实测的每轮耗时统一按 70% 平均接受率折算(27B 每轮 5.9 个 token,Flash-Next 3.8 个);数据集一图是真实题目上的实测速度。
Speculative speed depends on how many drafts are accepted, which varies with content. To compare scenarios, the concurrency and long-context charts convert measured step times at a 70% average acceptance (5.9 tokens per step for 27B, 3.8 for Flash-Next). The dataset chart shows measured speed on real prompts.
并发越多,总吞吐越高
More users, more total throughput
8K 上下文,温度 0,tok/s;柱顶小字为每路速度
8K context, temperature 0, tok/s; per-request speed above each bar
长文几乎不掉速
Long context barely slows it down
单请求 tok/s,1K 到 250K 上下文
One request, tok/s, 1K to 250K context
真实数据集单请求速度
One request on real datasets
实测 tok/s,括号内为草稿接受率
Measured tok/s; draft acceptance in brackets
长对话:第二轮几乎不用等
Long chats: the next turn barely waits
27B 首字时间(秒,对数刻度):首次输入 vs 多轮追加命中缓存
27B time to first token (s, log scale): first prompt vs cached follow-up
压到 4 bit,智商不掉
4-bit weights, same intelligence
与原版 BF16 模型在同一套题、同样的贪心解码下逐题对比。27B 与 BF16 的差距都在统计误差内,所有输出中没有乱码,也没有重复循环。
Compared question by question with the original BF16 model, same prompts, greedy decoding. Every 27B difference is within statistical error, with no garbled text or repetition loops in any output.
准确率
Accuracy
怎么保证不降智
How precision is kept
- 只舍入一次:FP4 权重解码到 FP16 只舍入一次,矩阵乘全程 FP32 累加
- 关键状态高精度:logits、线性注意力状态都用 FP32
- KV 校准:FP8 E4M3 KV,按层加载校准系数
- 逐 token 对齐:与 BF16 出现分叉时,BF16 的优势中位数仅 0.2(logprob),从未选到 BF16 前 5 名之外的 token
- 采样无偏:投机解码与不用草稿模型时的采样分布对比,每个 prompt 600 次采样逐位置检验,未见系统性偏差
- One rounding: FP4 weights decode to FP16 once; GEMMs accumulate in FP32
- High-precision state: logits and linear-attention state stay in FP32
- Calibrated KV: FP8 E4M3 KV with per-layer scales
- Token alignment: where outputs diverge from BF16, BF16's margin has a median of 0.2 logprob, and the chosen token is always in BF16's top 5
- Unbiased sampling: speculative sampling compared with sampling without a draft, 600 samples per prompt per position, no systematic bias
为什么这么快
Why it is fast
一轮出多个 token
Several tokens per step
DFlash2 草稿模型一次提出 7 个 token,目标模型一轮全部验证。写代码时(温度 0)平均每轮输出 6.9 个 token。
The DFlash2 draft proposes 7 tokens and the target verifies them in one step. On code at temperature 0, each step yields 6.9 tokens on average.
权重小,带宽跑满
Small weights, full bandwidth
NVFP4 权重只有 FP16 的约 1/3.5。V100 没有 FP4 硬件,我们用自研算子在寄存器里解码,单请求每轮耗时已贴近只读一遍权重所需的时间。
NVFP4 weights are about 1/3.5 the size of FP16. V100 has no FP4 hardware, so our kernels decode in registers; one request per step is close to the time needed just to read the weights.
KV 历史放主机内存
KV history in host memory
Flash-Next 只把最近 32K 的 KV 留在显存,更早的历史以 FP8 放进主机内存,由稀疏注意力按需读取;48 层中 36 层是线性注意力,所以 250K 时只比 1K 慢 18%。
Flash-Next keeps the latest 32K of KV on the GPU and older history in host memory as FP8, read on demand by sparse attention; 36 of 48 layers use linear attention, so 250K is only 18% slower than 1K.
完整测试数据
Complete test data
发布前在同一台 4×V100 上跑完的全部测试。纯 decode 计时只统计所有请求都在 decode 的步,不含 prefill;所有测试零错误、零显存溢出。
Every test run on the same 4×V100 before release. Pure decode timing counts only steps where all requests are decoding, excluding prefill. Zero errors and zero out-of-memory in all runs.
测试环境与部署
Test setup and deployment
测试环境
Test setup
- 4× Tesla V100-SXM2-32GB(NVLink),张量并行 4
- 显存 1107 MHz,核心最高 1530 MHz,功耗上限 350 W,ECC 开
- 主机内存 62 GB
- Qwen3.8-27B NVFP4(Unsloth)+ DFlash2;Qwen3.8-Flash-Next NVFP4 + MTP4
- 4× Tesla V100-SXM2-32GB (NVLink), tensor parallel 4
- Memory 1107 MHz, core up to 1530 MHz, 350 W, ECC on
- 62 GB host memory
- Qwen3.8-27B NVFP4 (Unsloth) + DFlash2; Qwen3.8-Flash-Next NVFP4 + MTP4
部署方式
Deployment
- pip 安装 wheel
- Docker:一个镜像启动两个模型
- 1Cat Studio:图形界面一键加载
- OpenAI 兼容接口:工具调用、结构化输出、思考模式、图片输入(27B)
- pip-install the wheel
- Docker: one image serves both models
- 1Cat Studio: one-click loading in a GUI
- OpenAI-compatible API: tool calls, structured output, reasoning, image input (27B)
硬件要求
Requirements
- 4× V100-SXM2-32GB(NVLink)
- 主机内存 64 GB 以上
- 最大 256K 上下文、8 并发
- 27B 启动约 2.5 分钟,Flash-Next 约 6 分钟(有编译缓存)
- 4× V100-SXM2-32GB (NVLink)
- 64 GB+ host memory
- Up to 256K context, 8 concurrent requests
- Startup about 2.5 min for 27B and 6 min for Flash-Next (warm compile cache)