RedHatAI/gemma-4-31B-it-speculator.dflash
Model Overview
- Model Architecture: Gemma4ForConditionalGeneration
- Input: Text
- Output: Text
- Model Optimizations:
- Speculative Decoding Algorithm: DFlash
- Draft Model: 5-layer qwen3 backbone (DFlashDraftModel)
- Release Date: 2026-04-17
- Version: 1.0
- Model Developers: RedHatAI
A DFlash speculator model for google/gemma-4-31B-it. It was trained using the Speculators library on mlabonne/open-perfectblend. Responses were regenerated by google/gemma-4-31B-it (with reasoning).
Training Details
Trained using the Speculators library (version 0.8.0). The published checkpoint comes from a resumed run (run name dflash-gemma4-31b-perfectblend_dpcae_bs8), with --max-anchors 1536 throughout and extended to 6 total epochs.
Commands
Launch vLLM (hidden-state generation)
CUDA_VISIBLE_DEVICES=0,1,2,3 python launch_vllm.py google/gemma-4-31b-it \
--target-layer-ids 1 17 29 47 59 \
--port 8906 \
--tensor-parallel-size 4 \
--data-parallel-size 1 \
--max-model-len 15000 \
--gpu-memory-utilization 0.85
Launch training
Must be run once vLLM has finished launching and is running in the background.
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --standalone --nproc-per-node 4 \
-m speculators.train \
--draft-vocab-size 262144 \
--verifier-name-or-path google/gemma-4-31b-it \
--data-path /path/to/prepared-open-perfectblend-data \
--save-path /path/to/checkpoints \
--hidden-states-path /path/to/hidden-states \
--log-dir /path/to/training-logs \
--speculator-type dflash \
--block-size 8 \
--max-anchors 1536 \
--num-layers 5 \
--target-layer-ids 1 17 29 47 59 \
--sliding-window 2048 \
--epochs 6 \
--lr 6e-4 \
--weight-decay 0.01 \
--scheduler-type cosine \
--scheduler-warmup-ratio 0.05 \
--optimizer muon \
--muon-lr 0.003 \
--dpace-alpha 0.5 \
--per-position-loss-weight dpace \
--loss-fn ce \
--checkpoint-freq 0.1 \
--log-freq 100 \
--num-workers 12 \
--prefetch-factor 2 \
--fsdp-shard \
--vllm-endpoint http://localhost:8906/v1 \
--request-timeout 180 \
--total-seq-len 8192 \
--max-retries 20
4 training GPUs (world size 4) plus 4 separate GPUs for the hidden-state vLLM server, one node.
Model Specifications
| Base Model | google/gemma-4-31B-it |
| Chat Template | google/gemma-4-31B-it (use /chat/completions endpoint) |
| Format | Safetensors |
| License | Apache-2.0 (the Gemma-4 verifier is subject to its own license and terms) |
| Validation Hardware | Not tracked |
Deployment
# Deploy with speculative decoding
vllm serve google/gemma-4-31B-it \
--speculative-config '{
"model": "RedHatAI/gemma-4-31B-it-speculator.dflash",
"num_speculative_tokens": 7,
"method": "dflash"
}'
Use the verifier's chat template via the /chat/completions endpoint.
Evaluation
Per-position token acceptance rates across datasets:
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 93.20% | 84.33% | 76.50% | 69.34% | 63.00% | 57.17% | 51.76% | 5.95 |
| math_reasoning | 96.47% | 90.90% | 85.87% | 80.37% | 74.92% | 69.56% | 64.17% | 6.62 |
| qa | 79.74% | 61.16% | 46.79% | 37.28% | 30.66% | 25.30% | 21.06% | 4.02 |
| question | 82.59% | 63.30% | 49.22% | 38.98% | 31.96% | 26.89% | 22.19% | 4.15 |
| rag | 87.37% | 73.66% | 61.38% | 51.84% | 43.38% | 36.82% | 30.56% | 4.85 |
| summarization | 82.47% | 63.84% | 48.97% | 37.67% | 28.63% | 21.10% | 15.77% | 3.98 |
| tool_call | 84.57% | 66.66% | 52.51% | 42.18% | 34.43% | 28.05% | 22.49% | 4.31 |
| translation | 86.33% | 72.84% | 61.79% | 52.28% | 43.93% | 37.14% | 31.29% | 4.86 |
| writing | 82.53% | 63.48% | 49.18% | 39.09% | 32.10% | 26.90% | 22.37% | 4.16 |
Performance benchmarking
Throughput vs. interactivity, comparing the google/gemma-4-31B-it verifier with no speculator against the verifier served with this DFlash draft model (7 speculative tokens), at matched request concurrency (N=1,2,4,8,16,32,64,128):
At the highest-throughput point measured (N=128 concurrent requests):
| Subset | Baseline (tok/s) | DFlash (tok/s) | Speedup |
|---|---|---|---|
| HumanEval | 2,855 | 3,938 | 1.38x |
| math_reasoning | 3,453 | 4,580 | 1.33x |
Configuration and command
Configuration
- max output tokens: 1024
- concurrency (N): 1, 2, 4, 8, 16, 32, 64, 128
- repetitions: 1 run per point
- vLLM version: 0.31.0
- GuideLLM version: 0.8.1
- Verifier served with:
--tensor-parallel-size 1 --max-model-len 16384 --max-num-seqs 256 --max-num-batched-tokens 16384 --no-enable-prefix-caching
Command
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 1 --max-model-len 16384 --max-num-seqs 256 \
--max-num-batched-tokens 16384 --no-enable-prefix-caching \
--speculative-config '{"model":"RedHatAI/gemma-4-31B-it-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}'
python scripts/evaluate/throughput_interactivity.py collect \
--target http://localhost:8114 --model google/gemma-4-31B-it \
--dataset RedHatAI/speculator_benchmarks --subset HumanEval \
--max-tokens 1024 --repeats 1 \
--streams 1,2,4,8,16,32,64,128
References
Paper: DFlash
- Downloads last month
- 597

