RedHatAI/gemma-4-31B-it-speculator.dflash

Model Overview

  • Model Architecture: Gemma4ForConditionalGeneration
    • Input: Text
    • Output: Text
  • Model Optimizations:
    • Speculative Decoding Algorithm: DFlash
    • Draft Model: 5-layer qwen3 backbone (DFlashDraftModel)
  • Release Date: 2026-04-17
  • Version: 1.0
  • Model Developers: RedHatAI

A DFlash speculator model for google/gemma-4-31B-it. It was trained using the Speculators library on mlabonne/open-perfectblend. Responses were regenerated by google/gemma-4-31B-it (with reasoning).

Training Details

Trained using the Speculators library (version 0.8.0). The published checkpoint comes from a resumed run (run name dflash-gemma4-31b-perfectblend_dpcae_bs8), with --max-anchors 1536 throughout and extended to 6 total epochs.

Commands

Launch vLLM (hidden-state generation)

CUDA_VISIBLE_DEVICES=0,1,2,3 python launch_vllm.py google/gemma-4-31b-it \
  --target-layer-ids 1 17 29 47 59 \
  --port 8906 \
  --tensor-parallel-size 4 \
  --data-parallel-size 1 \
  --max-model-len 15000 \
  --gpu-memory-utilization 0.85

Launch training

Must be run once vLLM has finished launching and is running in the background.

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --standalone --nproc-per-node 4 \
  -m speculators.train \
  --draft-vocab-size 262144 \
  --verifier-name-or-path google/gemma-4-31b-it \
  --data-path /path/to/prepared-open-perfectblend-data \
  --save-path /path/to/checkpoints \
  --hidden-states-path /path/to/hidden-states \
  --log-dir /path/to/training-logs \
  --speculator-type dflash \
  --block-size 8 \
  --max-anchors 1536 \
  --num-layers 5 \
  --target-layer-ids 1 17 29 47 59 \
  --sliding-window 2048 \
  --epochs 6 \
  --lr 6e-4 \
  --weight-decay 0.01 \
  --scheduler-type cosine \
  --scheduler-warmup-ratio 0.05 \
  --optimizer muon \
  --muon-lr 0.003 \
  --dpace-alpha 0.5 \
  --per-position-loss-weight dpace \
  --loss-fn ce \
  --checkpoint-freq 0.1 \
  --log-freq 100 \
  --num-workers 12 \
  --prefetch-factor 2 \
  --fsdp-shard \
  --vllm-endpoint http://localhost:8906/v1 \
  --request-timeout 180 \
  --total-seq-len 8192 \
  --max-retries 20

4 training GPUs (world size 4) plus 4 separate GPUs for the hidden-state vLLM server, one node.

Model Specifications

Base Model google/gemma-4-31B-it
Chat Template google/gemma-4-31B-it (use /chat/completions endpoint)
Format Safetensors
License Apache-2.0 (the Gemma-4 verifier is subject to its own license and terms)
Validation Hardware Not tracked

Deployment

# Deploy with speculative decoding
vllm serve google/gemma-4-31B-it \
    --speculative-config '{
        "model": "RedHatAI/gemma-4-31B-it-speculator.dflash",
        "num_speculative_tokens": 7,
        "method": "dflash"
    }'

Use the verifier's chat template via the /chat/completions endpoint.

Evaluation

Per-position token acceptance rates across datasets:

Dataset Pos 0 Pos 1 Pos 2 Pos 3 Pos 4 Pos 5 Pos 6 Avg. Length
HumanEval 93.20% 84.33% 76.50% 69.34% 63.00% 57.17% 51.76% 5.95
math_reasoning 96.47% 90.90% 85.87% 80.37% 74.92% 69.56% 64.17% 6.62
qa 79.74% 61.16% 46.79% 37.28% 30.66% 25.30% 21.06% 4.02
question 82.59% 63.30% 49.22% 38.98% 31.96% 26.89% 22.19% 4.15
rag 87.37% 73.66% 61.38% 51.84% 43.38% 36.82% 30.56% 4.85
summarization 82.47% 63.84% 48.97% 37.67% 28.63% 21.10% 15.77% 3.98
tool_call 84.57% 66.66% 52.51% 42.18% 34.43% 28.05% 22.49% 4.31
translation 86.33% 72.84% 61.79% 52.28% 43.93% 37.14% 31.29% 4.86
writing 82.53% 63.48% 49.18% 39.09% 32.10% 26.90% 22.37% 4.16

Performance benchmarking

Throughput vs. interactivity, comparing the google/gemma-4-31B-it verifier with no speculator against the verifier served with this DFlash draft model (7 speculative tokens), at matched request concurrency (N=1,2,4,8,16,32,64,128):

HumanEval math_reasoning
HumanEval throughput vs. interactivity math_reasoning throughput vs. interactivity

At the highest-throughput point measured (N=128 concurrent requests):

Subset Baseline (tok/s) DFlash (tok/s) Speedup
HumanEval 2,855 3,938 1.38x
math_reasoning 3,453 4,580 1.33x
Configuration and command

Configuration

  • max output tokens: 1024
  • concurrency (N): 1, 2, 4, 8, 16, 32, 64, 128
  • repetitions: 1 run per point
  • vLLM version: 0.31.0
  • GuideLLM version: 0.8.1
  • Verifier served with: --tensor-parallel-size 1 --max-model-len 16384 --max-num-seqs 256 --max-num-batched-tokens 16384 --no-enable-prefix-caching

Command

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 1 --max-model-len 16384 --max-num-seqs 256 \
  --max-num-batched-tokens 16384 --no-enable-prefix-caching \
  --speculative-config '{"model":"RedHatAI/gemma-4-31B-it-speculator.dflash","num_speculative_tokens":7,"method":"dflash"}'

python scripts/evaluate/throughput_interactivity.py collect \
  --target http://localhost:8114 --model google/gemma-4-31B-it \
  --dataset RedHatAI/speculator_benchmarks --subset HumanEval \
  --max-tokens 1024 --repeats 1 \
  --streams 1,2,4,8,16,32,64,128

References

Paper: DFlash

Downloads last month
597
Safetensors
Model size
3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for RedHatAI/gemma-4-31B-it-speculator.dflash

Finetuned
(385)
this model

Collection including RedHatAI/gemma-4-31B-it-speculator.dflash

Paper for RedHatAI/gemma-4-31B-it-speculator.dflash