SpecForge
在 4 卡昇腾 NPU 上端到端跑通 SpecForge 的投机解码草稿模型训练链路:
安装:huggingface_hub + mooncake + SGLang + specforge + 上游 capture 插件。
端到端 smoke:从 HuggingFace Hub 拉 Qwen3.5-4B → 起 mooncake_master → 起带 capture 插件的 SGLang capture server → 完成 1 步训练。
前置条件
硬件
Atlas 900 A2 / A3 训练系列产品或者 Ascend 950 系列产品,≥ 4 卡(capture server 用卡 0、trainer 用卡 1,卡 2/3 留给 HCCL 通信 buffer)。
基础软件
可用的 Python 环境
可用的 CANN(参考快速安装昇腾环境)
本文档示例使用的版本
配套镜像:
swr.cn-south-1.myhuaweicloud.com/ascendhub/cann:9.1.0-910b-ubuntu22.04-py3.12
软件版本:
组件 |
版本 |
|---|---|
Python |
3.12 |
CANN |
9.1.0 |
torch / torch_npu |
2.10.0+cpu / 2.10.0.post6 |
sglang |
源码构建,commit |
sgl-kernel-npu |
2026.9.0.post12( |
huggingface_hub |
最新稳定版 |
mooncake-transfer-engine |
0.3.14 |
specforge |
上游 |
capture 插件 |
上游 |
模型 |
|
参考样例 |
|
前置安装
确认能看到 NPU 设备(应至少列出 4 张 910B4 且状态 OK):
npu-smi info
输出类似:
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B4 | OK | 89.9 39 0 / 0 |
| 0 | 0000:41:00.0 | 0 0 / 0 2922 / 32768 |
+===========================+===============+====================================================+
| 1 910B4 | OK | 89.9 39 0 / 0 |
| 1 | 0000:42:00.0 | 0 0 / 0 2922 / 32768 |
+===========================+===============+====================================================+
| 2 910B4 | OK | 89.9 39 0 / 0 |
| 2 | 0000:43:00.0 | 0 0 / 0 2922 / 32768 |
+===========================+===============+====================================================+
| 3 910B4 | OK | 89.9 39 0 / 0 |
| 3 | 0000:44:00.0 | 0 0 / 0 2922 / 32768 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| No running processes found in NPU 0 |
| No running processes found in NPU 1 |
| No running processes found in NPU 2 |
| No running processes found in NPU 3 |
+===========================+===============+====================================================+
Note
如果 npu-smi 不存在,请回到 Ascend 官方快速安装指南 补装驱动。
确认 Python 版本:
python --version
输出结果如下:
Python xxx
Note
xxx 表示 Python 的版本号。
确认 CANN 版本:
test -f /usr/local/Ascend/ascend-toolkit/latest/$(uname -m)-linux/ascend_toolkit_install.info && \
grep '^version=' /usr/local/Ascend/ascend-toolkit/latest/$(uname -m)-linux/ascend_toolkit_install.info || \
echo "ascend_toolkit_install.info MISSING"
输出结果如下:
version=9.1.0
安装 torch / torch_npu:
set -euo pipefail
if python - <<'PY' 2>/dev/null; then
import torch, torch_npu
PY
echo "torch stack already present, keeping it" >&2
else
echo "installing torch==2.10.0 torch_npu==2.10.0.post6" >&2
python -m pip install \
-f https://mirrors.aliyun.com/pytorch-wheels/cpu \
--extra-index-url https://repo.huaweicloud.com/ascend/repos/pypi \
"torch==2.10.0" "torch_npu==2.10.0.post6" \
>/tmp/install-torch.log 2>&1 \
|| { echo "install-torch-stack: FAILED - pip install:" >&2; tail -20 /tmp/install-torch.log >&2; exit 1; }
fi
python - <<'PY'
import torch, torch_npu
print('torch stack ready: torch=' + torch.__version__ + ' torch_npu=' + torch_npu.__version__)
PY
输出结果如下:
torch stack ready: torch=2.10.0+cpu torch_npu=2.10.0.post6
确认 torch / torch_npu 能 import 且能看到 4 张卡(下面的命令用 Python 执行):
import torch, torch_npu
print('is_available:', torch.npu.is_available())
print('count:', torch.npu.device_count())
输出结果如下:
is_available: True
count: 4
安装 huggingface_hub
用 uv 安装 huggingface_hub(后面从 HuggingFace Hub 下载模型用):
uv pip install huggingface_hub
安装 mooncake
SpecForge 的在线训练用 mooncake 在 SGLang server 和 trainer 之间传数据。直接用 PyPI 上 Mooncake 官方发布的 NPU prebuilt wheel,wheel 没带的三个系统库用 apt 补齐:
set -euo pipefail
apt-get update -qq >/dev/null 2>&1
apt-get install -qq -y --no-install-recommends \
libibverbs1 libcurl4 libnuma1 >/dev/null 2>&1 \
|| { echo "build-mooncake: FAILED - apt install runtime deps" >&2; exit 1; }
uv pip install --upgrade \
'mooncake-transfer-engine-npu==0.3.14' \
--index-url https://mirrors.aliyun.com/pypi/simple/ \
--extra-index-url https://pypi.org/simple/ \
>/tmp/build-mooncake-pip.log 2>&1 \
|| { echo "build-mooncake: FAILED - pip install:" >&2; tail -20 /tmp/build-mooncake-pip.log >&2; exit 1; }
INSTALLED=$(python - <<'PY'
from importlib.metadata import version
print(version('mooncake-transfer-engine-npu'))
PY
)
echo "build-mooncake: installed mooncake-transfer-engine-npu==${INSTALLED}"
验证两个包都装好了:
python - <<'PY'
import huggingface_hub, mooncake.store
print('huggingface_hub', huggingface_hub.__version__)
print('mooncake.store imports clean')
PY
test -x "$(command -v mooncake_master)" && echo "mooncake_master binary present" || echo "mooncake_master binary MISSING"
输出结果如下:
huggingface_hub xxx
mooncake.store imports clean
mooncake_master binary present
Note
xxx 表示实际安装的版本号。
克隆 SpecForge 源码
PyPI wheel 不带示例配置和 capture 插件,所以从源码装:
set -euo pipefail
GIT_NET_OPTS=(-c http.version=HTTP/1.1 -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=60)
clone_specforge() {
local url="$1" branch="${2:-}"
rm -rf SpecForge
for attempt in 1 2 3 4 5; do
if [ -n "$branch" ]; then
git "${GIT_NET_OPTS[@]}" clone --depth 1 --branch "$branch" "$url" SpecForge && return 0
else
git "${GIT_NET_OPTS[@]}" clone --depth 1 "$url" SpecForge && return 0
fi
echo "clone attempt ${attempt} failed, retrying" >&2
sleep 10
done
echo "FAILED - clone ${branch:+$branch }${url} after 5 attempts" >&2
return 1
}
if [[ "<ref>" =~ ^[0-9a-f]{40}$ ]]; then
clone_specforge https://github.com/sgl-project/SpecForge.git
git -C SpecForge -c http.version=HTTP/1.1 fetch --depth 1 origin "<ref>"
git -C SpecForge checkout FETCH_HEAD
else
clone_specforge https://github.com/sgl-project/SpecForge.git "<ref>"
fi
git -C SpecForge log -1 --format='specforge checkout: %H'
输出结果如下:
specforge checkout: xxx
Note
<ref> 替换为被测的 SpecForge 版本(看护工作流解析上游 main 的 HEAD),xxx 为实际 checkout 的 commit。
安装 SGLang(源码构建)
capture 插件要用的扩展点任何 release wheel 都没有,所以从源码构建;commit 取自上游 SpecForge CI 的 SGLANG_COMMIT:
set -euo pipefail
SGLANG_COMMIT=$(sed -n 's/^[[:space:]]*SGLANG_COMMIT:[[:space:]]*\([0-9a-f]\{40\}\).*/\1/p' \
SpecForge/.github/workflows/test.yaml | head -1)
[ -n "$SGLANG_COMMIT" ] || { echo "sglang-install: FAILED - SGLANG_COMMIT not found" >&2; exit 1; }
rm -rf sglang-src && mkdir sglang-src && cd sglang-src
git init -q
git remote add origin https://github.com/sgl-project/sglang.git
GIT_NET_OPTS=(-c http.version=HTTP/1.1 -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=60)
for attempt in 1 2 3 4 5; do
git "${GIT_NET_OPTS[@]}" fetch --depth 1 origin "$SGLANG_COMMIT" && break
[ "$attempt" = 5 ] && { echo "sglang-install: FAILED - fetch $SGLANG_COMMIT" >&2; exit 1; }
sleep 10
done
git checkout -q FETCH_HEAD
cd python
mv pyproject_npu.toml pyproject.toml
KERNEL_TAG=2026.9.0.post12
PY_TAG=py$(python - <<'PY'
import sys
print(f"{sys.version_info.major}{sys.version_info.minor}")
PY
)
CANN_VER=$(ls -d /usr/local/Ascend/cann-* 2>/dev/null | head -1 | sed 's/.*cann-//')
CANN_VER="${SGL_KERNEL_CANN_VER:-$CANN_VER}"
WHEEL="sgl-kernel-npu-${KERNEL_TAG}-torch2.10.0-${PY_TAG}-cann${CANN_VER}-910b-$(uname -m).zip"
cd .. && mkdir -p npu-ops && cd npu-ops
curl -fSL --retry 5 --retry-all-errors --http1.1 -C - -o "$WHEEL" \
"https://github.com/sgl-project/sgl-kernel-npu/releases/download/${KERNEL_TAG}/${WHEEL}" \
|| { echo "sglang-install: FAILED - download $WHEEL" >&2; exit 1; }
python - "$WHEEL" <<'PYUNZIP'
import sys, zipfile
with zipfile.ZipFile(sys.argv[1]) as z:
z.extractall(".")
PYUNZIP
uv pip install deep_ep*.whl sgl_kernel_npu*.whl torch_memory_saver*.whl
cd ../python
printf 'torch==2.10.0\ntorchvision==0.25.0\ntorchaudio==2.10.0\n' > /tmp/torch-constraint.txt
uv pip install -e . --constraint /tmp/torch-constraint.txt
cd ..
if ! python - <<'PY' 2>/dev/null; then
import triton.language.extra.cann
PY
uv pip install --reinstall --no-deps "triton-ascend==3.6.0" pybind11 \
--extra-index-url https://repo.huaweicloud.com/ascend/repos/pypi \
>/tmp/triton-install.log 2>&1 \
|| { echo "sglang-install: FAILED - install triton-ascend" >&2; exit 1; }
fi
python - <<'PY'
import importlib.metadata as m
print('sglang, version', m.version('sglang'))
print('sgl_kernel_npu, version', m.version('sgl_kernel_npu'))
import triton.language.extra.cann
print('triton-ascend OK')
PY
输出结果如下:
sglang, version xxx
sgl_kernel_npu, version 2026.9.0
triton-ascend OK
Note
xxx 是本次构建出的 SGLang 版本号(源码构建,带上游 commit 的 dev 版本串)。
确认 capture 插件要的扩展点都在(插件启动时会逐项检查,下面的命令用 Python 执行):
from sglang.srt.server_args import ServerArgs
from sglang.srt.model_executor.forward_observer import ForwardObserver
from sglang.srt.managers.deferred_output import ResponseCompletion
from sglang.srt.managers.scheduler import Scheduler
for n in ('enable_return_aux_hidden_states', 'return_aux_hidden_state_layer_ids', 'return_hidden_states_mode'):
assert hasattr(ServerArgs, n), n
for m in ('defer_response', 'get_output_streamer_class', 'get_internal_state'):
assert hasattr(Scheduler, m), m
print('capture API present: aux-hidden-states + forward_observer + defer_response')
输出结果如下:
capture API present: aux-hidden-states + forward_observer + defer_response
安装 specforge 与 capture 插件
capture 插件装进 SGLang server 所在的环境(与 SGLang 同一个 python),SGLang 通过 sglang.srt.plugins entry point 发现它:
set -euo pipefail
cd SpecForge
uv pip install --no-deps .
uv pip install --no-deps accelerate
uv pip install --no-deps plugins/sglang-spec-capture
python - <<'PY'
from importlib.metadata import version
print('specforge, version', version('specforge'))
PY
python - <<'PY'
from importlib.metadata import entry_points
eps = [e.name for e in entry_points(group='sglang.srt.plugins')]
assert 'specforge_spec_capture' in eps, eps
print('capture plugin registered: specforge_spec_capture')
PY
输出结果如下:
specforge, version xxx
capture plugin registered: specforge_spec_capture
Note
xxx 表示实际安装的 specforge 版本号。
CLI 自检
确认 specforge 能在 NPU 环境里正常 import(下面的命令用 Python 执行):
import specforge, torch, torch_npu
print('specforge', getattr(specforge, '__version__', 'unknown'))
print('torch', torch.__version__)
print('torch.npu.is_available', torch.npu.is_available())
输出结果如下:
specforge xxx
torch 2.10.0+cpu
torch.npu.is_available True
Note
xxx 表示 specforge 的实际版本号。
查看 CLI 帮助(三个子命令:train / export / benchmark):
specforge --help
输出结果如下:
Usage: specforge [OPTIONS] COMMAND [ARGS]...
SpecForge: speculative decoding training framework.
Options:
-h, --help Show this message and exit.
Commands:
benchmark benchmark a running SGLang server
export materialize a runtime checkpoint as a model directory
train train a draft model from a typed config
查看 train 子命令的参数(后面训练就用这些 flag):
specforge train --help
输出结果如下:
Usage: specforge train [OPTIONS] [OVERRIDES]...
Train a draft model from a typed run config.
OVERRIDES are dotted ``section.field=value`` assignments applied on top of
the config file, e.g. ``training.learning_rate=1e-4``.
Options:
-c, --config PATH YAML or JSON run config. [required]
--role [auto|all|producer|consumer|both]
Launch selection: offline local 'all' or
online/disaggregated producer+consumer when
'auto'. [default: auto]
--node-rank INTEGER Node-local rank for an explicit multi-node
trainer launch.
--plan Print the resolved process plan without
starting workers.
-h, --help Show this message and exit.
端到端 smoke:1 步训练
按顺序把 mooncake_master → SGLang capture server → specforge train 拉起来跑 1 步训练,每小节一个阶段,失败可定位到具体环节。SPECFORGE_* 环境变量可覆盖各段默认值。
预下载模型
从 HuggingFace Hub 把 Qwen3.5-4B(约 8 GB)拉到本地缓存,路径供后面起 server 和训练用:
set -euo pipefail
MODEL_ID="${SPECFORGE_MODEL_ID:-Qwen/Qwen3.5-4B}"
echo "smoke: downloading model $MODEL_ID from HuggingFace Hub" >&2
python - "$MODEL_ID" <<'PY'
import sys, contextlib
MODEL_ID = sys.argv[1]
with contextlib.redirect_stdout(sys.stderr):
from huggingface_hub import snapshot_download
path = snapshot_download(MODEL_ID)
print(path)
PY
输出的模型缓存目录路径,供「起 SGLang capture server」(--model-path)和「跑 specforge 训练」(model.target_model_path)两节使用。
起 mooncake_master
后台拉起 mooncake_master(metadata + 传输协调服务),等 RPC 端口(默认 35551)就绪。端口探测用 Python socket 而非 nc -z——镜像里没装 nc:
set -euo pipefail
pkill -9 -f '^mooncake_master' 2>/dev/null || true
MOONCAKE_RPC_PORT="${SPECFORGE_MOONCAKE_RPC_PORT:-35551}"
MOONCAKE_HTTP_PORT="${SPECFORGE_MOONCAKE_HTTP_PORT:-35880}"
nohup mooncake_master \
--enable_http_metadata_server=true \
--rpc_port=$MOONCAKE_RPC_PORT \
--http_metadata_server_port=$MOONCAKE_HTTP_PORT \
--metrics_port=35903 \
--enable_metric_reporting=false \
>/tmp/smoke-mooncake.log 2>&1 &
MOONCAKE_PID=$!
mooncake_ready() {
python - <<PY 2>/dev/null
import socket, sys
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(0.5)
try:
s.connect(('127.0.0.1', $MOONCAKE_RPC_PORT))
except Exception:
sys.exit(1)
finally:
s.close()
sys.exit(0)
PY
}
for _ in $(seq 1 30); do
if mooncake_ready; then
echo "smoke: mooncake ready (rpc $MOONCAKE_RPC_PORT, pid=$MOONCAKE_PID)"
exit 0
fi
sleep 1
done
echo "smoke: FAILED - mooncake_master did not bind $MOONCAKE_RPC_PORT in 30s" >&2
tail -50 /tmp/smoke-mooncake.log >&2
exit 1
输出结果如下:
smoke: mooncake ready (rpc 35551, pid=xxx)
Note
xxx 为 mooncake_master 的进程号。
起 SGLang capture server
后台拉起带 capture 插件的 SGLang server(卡 0),开启 dflash spec-capture,等 /health 就绪。首次启动含 NPU graph 编译,等 5-10 分钟属正常。环境变量要逐个 export(前缀链会让 server 拿不到);SPECFORGE_SPEC_CAPTURE=1 是插件开关,不开就不采集;Qwen3.5-4B 是 hybrid Mamba/GDN,--mamba-ssm-dtype bfloat16 让 GDN 的 initial_state 用 bf16(910B4 不支持 fp32),libatb.so 需要手动加进 LD_LIBRARY_PATH:
set -euo pipefail
pkill -9 -f '^python -m sglang\.launch_server' 2>/dev/null || true
CAPTURE_DEVICE="${SPECFORGE_CAPTURE_DEVICE:-0}"
SGLANG_PORT="${SPECFORGE_SGLANG_PORT:-30000}"
SGLANG_HEALTH_TIMEOUT="${SPECFORGE_SGLANG_HEALTH_TIMEOUT:-600}"
MOONCAKE_RPC_PORT="${SPECFORGE_MOONCAKE_RPC_PORT:-35551}"
MOONCAKE_HTTP_PORT="${SPECFORGE_MOONCAKE_HTTP_PORT:-35880}"
export ASCEND_RT_VISIBLE_DEVICES=$CAPTURE_DEVICE
export MOONCAKE_LOCAL_HOSTNAME=127.0.0.1
export MOONCAKE_METADATA_SERVER=http://127.0.0.1:$MOONCAKE_HTTP_PORT/metadata
export MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:$MOONCAKE_RPC_PORT
export MOONCAKE_PROTOCOL=tcp
export MOONCAKE_GLOBAL_SEGMENT_SIZE=$((32<<30))
export SPECFORGE_SPEC_CAPTURE=1
ATB_LIB=/usr/local/Ascend/nnal/atb/9.0.0/atb/cxx_abi_1/lib
if [[ -d "$ATB_LIB" ]]; then
export LD_LIBRARY_PATH="$ATB_LIB:${LD_LIBRARY_PATH:-}"
fi
nohup python -m sglang.launch_server \
--model-path "<MODEL_PATH>" \
--trust-remote-code \
--skip-tokenizer-init \
--tp-size 1 \
--mem-fraction-static 0.5 \
--context-length 1024 \
--chunked-prefill-size -1 \
--attention-backend ascend \
--mamba-ssm-dtype bfloat16 \
--speculative-algorithm DFLASH \
--enable-return-aux-hidden-states \
--return-aux-hidden-state-layer-ids 1 8 15 22 29 \
--return-hidden-states-mode full \
--host 127.0.0.1 --port "$SGLANG_PORT" \
>/tmp/smoke-sglang.log 2>&1 &
SGLANG_PID=$!
echo "smoke: waiting for SGLang /health (up to ${SGLANG_HEALTH_TIMEOUT}s)"
HEALTH_DEADLINE=$((SGLANG_HEALTH_TIMEOUT / 5))
for _ in $(seq 1 "$HEALTH_DEADLINE"); do
if curl -fsS "http://127.0.0.1:$SGLANG_PORT/health" >/dev/null 2>&1; then
echo "smoke: sglang ready (pid=$SGLANG_PID)"
break
fi
sleep 5
done
if ! curl -fsS "http://127.0.0.1:$SGLANG_PORT/health" >/dev/null 2>&1; then
echo "smoke: FAILED - SGLang not healthy after ${SGLANG_HEALTH_TIMEOUT}s" >&2
tail -50 /tmp/smoke-sglang.log >&2
exit 1
fi
echo "smoke: capture backend: $(curl -fsS "http://127.0.0.1:$SGLANG_PORT/server_info" | grep -o '"spec_capture_backend":"[a-z]*"' | head -1)"
输出结果如下:
smoke: waiting for SGLang /health (up to 600s)
smoke: sglang ready (pid=xxx)
smoke: capture backend: "spec_capture_backend":"plugin"
Note
<MODEL_PATH> 在运行时替换为「预下载模型」一节 snapshot_download 命令打印的模型缓存路径;xxx 为 server 进程号。
准备训练数据
写一条 user→assistant 对话的 JSONL 作为训练数据(smoke 只跑 1 步,不追求数据质量;assistant 段够长即可满足 dflash 采样 32 个 anchor 的要求)。正式训练可用上游的 scripts/regenerate_train_data.py 重新生成对齐分布的数据:
set -euo pipefail
SPECFORGE_ROOT="${SPECFORGE_ROOT:-SpecForge}"
if [[ ! -d "$SPECFORGE_ROOT" ]]; then
echo "smoke: FAILED - $SPECFORGE_ROOT/ missing; specforge-install-source first"
exit 1
fi
mkdir -p "$SPECFORGE_ROOT/cache/dataset"
if ! python - "$SPECFORGE_ROOT/cache/dataset/sharegpt_train.jsonl" >/tmp/smoke-prep.log 2>&1 <<'PY'
import json, os, sys
path = sys.argv[1]
os.makedirs(os.path.dirname(path), exist_ok=True)
row = {
"id": "smoke_1",
"conversations": [
{"role": "user", "content": "Explain the difference between supervised and unsupervised learning in machine learning. Cover what types of problems each paradigm is suited for, give at least three concrete algorithm examples for each, and discuss practical considerations for choosing between them when you have a new dataset to model."},
{"role": "assistant", "content": "Supervised learning and unsupervised learning are the two foundational paradigms in machine learning, distinguished by the form of feedback the learning algorithm receives during training. In supervised learning the algorithm is given a labeled dataset, where each training example is paired with a target output, and adjusts its parameters to minimize a loss between its predictions and the ground truth; the two main task families are classification and regression, with concrete algorithms including linear regression for predicting continuous targets, logistic regression for binary classification, decision trees and random forests for tabular data, support vector machines for image classification, and deep neural networks for high-dimensional perceptual inputs. In unsupervised learning the algorithm receives only raw unlabeled inputs and must find structure on its own, with common tasks including clustering, dimensionality reduction, density estimation, and representation learning; concrete algorithms include k-means clustering for customer segmentation, hierarchical clustering for bioinformatics, principal component analysis for visualization, independent component analysis for signal separation, autoencoders for compact latent representations, and generative adversarial networks for synthesizing new samples. Choosing between the two depends on whether you have labeled data and a well defined prediction target: supervised learning delivers higher accuracy on the specific labeled task but requires costly annotation and does not generalize beyond seen labels, while unsupervised learning is preferable when labels are scarce or the goal is exploratory or structural. In modern practice the two are often combined, with unsupervised pretraining learning useful representations from large unlabeled corpora and supervised fine tuning specializing them for downstream tasks, an approach that underlies most large language model training pipelines today."},
],
}
with open(path, "w", encoding="utf-8") as f:
json.dump(row, f, ensure_ascii=False)
f.write("\n")
print(f"wrote {path}")
PY
then
echo "smoke: FAILED - prepare conversations JSONL:" >&2
tail -20 /tmp/smoke-prep.log >&2
exit 1
fi
SHAREGPT_PATH="$(realpath "$SPECFORGE_ROOT/cache/dataset/sharegpt_train.jsonl")"
ls -la "$SHAREGPT_PATH" >&2
echo "smoke: prepare-data OK ($SHAREGPT_PATH)" >&2
echo "$SHAREGPT_PATH"
输出的训练数据文件的绝对路径,供「跑 specforge 训练」(data.train_data_path)一节使用。
跑 specforge 训练(1 步)
看护框架逐段执行本文命令块,单段 stdout/stderr 整段缓冲,只在段结束时才输出到日志。所以 specforge train 运行期间日志长时间没有新输出是正常现象,不是卡死。
因此训练拆成"启动 + 监控"两段:启动段是 ≤30s 的短任务,跑完立刻能看到启动确认和 pid;监控段每 60s 打一行进度,训练完成后收尾。
启动 specforge train(后台)
在卡 1 上后台启动 1 步训练(batch/步数压到最小的 smoke 配置),30 秒后确认进程还活着。trainer 侧的 mooncake client 同样要 MOONCAKE_* 环境变量;deployment.disaggregated.receive_buffers=pageable 是 NPU 必需的覆盖(pinned 默认值在 NPU 上会崩):
set -euo pipefail
SPECFORGE_ROOT="${SPECFORGE_ROOT:-SpecForge}"
TRAINER_DEVICE="${SPECFORGE_TRAINER_DEVICE:-1}"
RECIPE="${SPECFORGE_RECIPE:-examples/configs/online/disaggregated/external/qwen3.5-4b-dflash-npu.yaml}"
pushd "$SPECFORGE_ROOT" >/dev/null
rm -rf outputs/qwen3.5-4b-dflash-npu-online
MOONCAKE_RPC_PORT="${SPECFORGE_MOONCAKE_RPC_PORT:-35551}"
MOONCAKE_HTTP_PORT="${SPECFORGE_MOONCAKE_HTTP_PORT:-35880}"
export MOONCAKE_LOCAL_HOSTNAME=127.0.0.1
export MOONCAKE_METADATA_SERVER=http://127.0.0.1:$MOONCAKE_HTTP_PORT/metadata
export MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:$MOONCAKE_RPC_PORT
export MOONCAKE_PROTOCOL=tcp
export MOONCAKE_GLOBAL_SEGMENT_SIZE=$((32<<30))
PYTHONUNBUFFERED=1 \
ASCEND_RT_VISIBLE_DEVICES=$TRAINER_DEVICE \
HCCL_CONNECT_TIMEOUT=7200 HCCL_EXEC_TIMEOUT=7200 \
PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \
nohup specforge train -c "$RECIPE" \
training.max_steps=1 \
training.batch_size=1 \
training.accumulation_steps=1 \
data.max_length=512 \
data.train_data_path="<SHAREGPT_PATH>" \
training.num_anchors=32 \
training.save_interval=0 \
training.log_interval=1 \
tracking.report_to=none \
deployment.trainer.nproc_per_node=1 \
deployment.disaggregated.receive_buffers=pageable \
model.target_model_path="<MODEL_PATH>" \
model.embedding_key="model.language_model.embed_tokens.weight" \
>/tmp/smoke-train.log 2>&1 &
TRAIN_PID=$!
disown $TRAIN_PID 2>/dev/null || true
echo "$TRAIN_PID" > /tmp/smoke-train.pid
sleep 30
if ! kill -0 "$TRAIN_PID" 2>/dev/null; then
echo "smoke: FAILED - specforge train died within 30s of launch"
tail -50 /tmp/smoke-train.log
exit 1
fi
echo "smoke: specforge train alive after 30s, pid=$TRAIN_PID, log=/tmp/smoke-train.log"
popd >/dev/null
输出结果如下:
smoke: specforge train alive after 30s, pid=xxx, log=/tmp/smoke-train.log
Note
<MODEL_PATH> / <SHAREGPT_PATH> 分别来自「预下载模型」「准备训练数据」两节保存的路径;xxx 为训练进程号。
监控直到退出
轮询等待训练完成:日志出现 step/loss 指标即训练成功,进程随后自行退出(容器的 PID 1 不回收孤儿,正常退出的进程会停在僵尸状态,按"僵尸即退出"处理;产出 step/loss 后 300s 仍不退出则杀掉进程树并按成功收尾):
set -euo pipefail
LOG_FILE=/tmp/smoke-train.log
TRAIN_PID=$(cat /tmp/smoke-train.pid)
WAIT_TIMEOUT="${SPECFORGE_TRAIN_TIMEOUT:-4800}"
GRACE_AFTER_DONE="${SPECFORGE_TRAIN_GRACE:-300}"
WAIT_START=$(date +%s)
DONE_AT=""
trainer_alive() {
kill -0 "$TRAIN_PID" 2>/dev/null || return 1
if grep -q 'State:[[:space:]]*Z' "/proc/$TRAIN_PID/status" 2>/dev/null; then
echo "smoke: train pid=$TRAIN_PID already exited (zombie state; container init does not reap orphans)"
return 1
fi
return 0
}
while trainer_alive; do
ELAPSED=$(( $(date +%s) - WAIT_START ))
if [[ $ELAPSED -ge $WAIT_TIMEOUT ]]; then
echo "smoke: FAILED - specforge train exceeded ${WAIT_TIMEOUT}s, killing pid=$TRAIN_PID"
kill -9 "$TRAIN_PID" 2>/dev/null || true
tail -50 "$LOG_FILE"
exit 1
fi
if [[ -z "$DONE_AT" ]] && grep -qE "step.*loss|loss.*step|step N:|train_runtime" "$LOG_FILE" 2>/dev/null; then
DONE_AT=$ELAPSED
echo "smoke: step/loss markers present at elapsed=${ELAPSED}s; giving the process ${GRACE_AFTER_DONE}s to exit"
fi
if [[ -n "$DONE_AT" ]] && [[ $(( ELAPSED - DONE_AT )) -ge $GRACE_AFTER_DONE ]]; then
echo "smoke: WARNING - train pid=$TRAIN_PID still alive ${GRACE_AFTER_DONE}s after step/loss markers; dumping stacks then killing process tree"
for P in "$TRAIN_PID" $(pgrep -P "$TRAIN_PID" 2>/dev/null || true); do
echo "[ERROR] teardown-hang diagnostic for pid $P:" >&2
awk '/^(State|Threads)/{print " "$0}' /proc/$P/status >&2 2>/dev/null || true
echo " wchan: $(cat /proc/$P/wchan 2>/dev/null)" >&2
echo " kernel stack:" >&2
cat /proc/$P/stack 2>/dev/null | sed 's/^/ /' >&2 || echo " (unreadable)" >&2
done
if uv pip install --quiet py-spy >/dev/null 2>&1; then
for P in "$TRAIN_PID" $(pgrep -P "$TRAIN_PID" 2>/dev/null || true); do
echo "[ERROR] py-spy stack dump for pid $P (teardown did not exit):" >&2
py-spy dump --pid "$P" --native >&2 2>&1 \
|| py-spy dump --pid "$P" >&2 2>&1 \
|| echo "[ERROR] py-spy dump failed for pid $P (ptrace blocked?)" >&2
done
else
echo "[ERROR] py-spy install failed; skipping stack dump" >&2
fi
pkill -9 -P "$TRAIN_PID" 2>/dev/null || true
kill -9 "$TRAIN_PID" 2>/dev/null || true
sleep 2
break
fi
if [[ -f "$LOG_FILE" ]]; then
LOG_SIZE=$(stat -c%s "$LOG_FILE" 2>/dev/null || echo 0)
LAST_LINE=$(tail -1 "$LOG_FILE" 2>/dev/null | head -c 220 || true)
echo "[$(date +%H:%M:%S)] elapsed=${ELAPSED}s pid=$TRAIN_PID log_size=${LOG_SIZE}B last: ${LAST_LINE}"
else
echo "[$(date +%H:%M:%S)] elapsed=${ELAPSED}s pid=$TRAIN_PID log file not yet created"
fi
sleep 60
done
ELAPSED=$(( $(date +%s) - WAIT_START ))
echo "smoke: specforge train process exited, elapsed=${ELAPSED}s"
if ! grep -qE "step.*loss|loss.*step|step N:|train_runtime" "$LOG_FILE"; then
echo "smoke: FAILED - no step/loss output in train log"
tail -30 "$LOG_FILE"
exit 1
fi
if grep -qE "Traceback \(most recent call last\):|RuntimeError:|AssertionError" "$LOG_FILE"; then
echo "smoke: WARNING - log has Traceback/ERROR but step markers also present; treating as success"
grep -nE "Traceback \(most recent call last\):|RuntimeError:|AssertionError" "$LOG_FILE" | head -5
fi
echo "smoke: OK - 1-step training completed"
输出结果如下(首行通配符吸收监控期间的进度行与状态提示;无论哪条路径,最后都以"进程退出 + 训练完成"两行收尾):
...
smoke: specforge train process exited, elapsed=...
smoke: OK - 1-step training completed
Note
... 是通配符:首行的 ... 吸收监控期间的进度行与状态提示,elapsed=... 是本次训练的耗时。
外部链接
GitHub:sgl-project/SpecForge
文档中心:SpecForge Docs