Training recipe

Markdown: recipe.md · Data: recipe.json

The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log.

Task

Description: Supervised LoRA fine-tune. Input is the production routing prompt (system + user, ~5.6k-6.3k tokens incl. capability docs and recent context); target is the assistant's single JSON capability call (13-128 tokens, median 26).
Loss: response-only: prompt tokens masked; labels and logits computed only for the last response_len + 1 positions via logits_to_keep
Example length tokens: Min: 6277; Max: 6447; Corpus: v1-v5 era; measured on 2026-09-14

Longest example later corpora tokens

Corpus	Tokens	Measured on
2026-09-15	7037	2026-09-17
2026-09-18 unfiltered	7288	2026-09-18
2026-09-18 filtered	6931	2026-09-18
2026-09-19 filtered	7068	2026-09-20
2026-09-21	7177	2026-09-23

Response span tokens: Min: 13; Max: 128; Median: 26; measured on 2026-09-13

Base model

HF repo: unsloth/gemma-3-4b-it
Text layers: 34
Vocab: 262208
Hidden: 2560
Measured on: 2026-09-14

Quantization for training

Method: bitsandbytes 4-bit QLoRA
Load in 4bit: yes
Bnb 4bit quant type: nf4
Bnb 4bit use double quant: yes
Bnb 4bit compute dtype: bfloat16
Lm head: cast from fp32 to bf16 after get_peft_model
Measured on: 2026-09-18

LoRA

Target modules

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Rank: Shipped: 112; Tried: 16, 64, 96, 112; Probed oom: 128; measured on 2026-09-25
Alpha: equal to rank (scale 1.0) in every shipped version; rsLoRA runs used alpha 32 at r112 and 30 at r96
Dropout: 0 (0.1 tried in v14/v15, closed)
RsLoRA: off (tried v13-v15, closed)
DoRA: off (OOM at r64 and r112)
Bias: none
Measured on: 2026-09-25

Hyperparameters

Epochs: 3
Learning rate: 0.0002
Lr scheduler: linear
Warmup steps: 5
Per device train batch size: 1
Gradient accumulation steps: 8
Effective batch: 8
Optimizer: paged_adamw_8bit
Weight decay: 0.01
Seed: 3407
Bf16: yes
Max seq length: 8192
Gradient checkpointing: on, use_reentrant=False, via prepare_model_for_kbit_training
Attn implementation: sdpa
Save and eval steps: 25
Decoding temperature at inference: 0
Measured on: 2026-09-18

Memory fixes

Fix	Why	Measured on
chunked MLP: Gemma3MLP.forward monkey-patched to run gate/up/down over 2048-token chunks of the sequence	gate_proj(x) and up_proj(x) over ~6300 tokens must coexist for the gated product; that pair was the measured OOM point	2026-09-14
torch.autograd.graph.save_on_cpu(pin_memory=True) around trainer.train()	gradient-checkpoint boundary tensors for 34 layers stayed on GPU; memory climbed 4.5 to 6.99 GB in one forward, 6.39 GB with the offload	2026-09-14
lm_head cast fp32 to bf16	fp32 lm_head was 2.68 GB and forced a mixed-precision matmul; pre-training VRAM fell 4502 MB to 3221 MB	2026-09-14
logits_to_keep = response_len + 1	full [1, 6390, 262208] fp32 logits would be 6.7 GB vs 135 MB for the response span; did not fix the first OOM (that was the MLP) but removes the tensor whenever the forward reaches lm_head	2026-09-14
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True	allocator setting only; bought 3 steps in the v14 dropout runs, not a fix	2026-09-23
pre-tokenize and mask by hand; no datasets.map()	map() crashed dill-pickling a closure over the Unsloth-patched tokenizer	2026-09-14
token_type_ids all zeros	plain transformers Gemma3 requires them during training (clear ValueError)	2026-09-14

Did not help

Lever
expandable_segments allocator tuning
paged_adamw_8bit (kept, but the first OOM was in forward, before any optimizer step)
non-reentrant gradient checkpointing alone
splitting layers across both GPUs with device_map=auto (pipeline split does not shrink one layer's activations)
attention-only LoRA targets (moved the OOM one line; the floor is the base MLP)
Liger Kernel fused GEGLU (fuses only the elementwise step after both projections exist)
flash-attn (attention was never the bottleneck; built from source after installing a CUDA 13.0 toolkit, then abandoned)

Why plain transformers peft not unsloth

Decision: train with transformers.AutoModelForCausalLM + BitsAndBytesConfig + peft.get_peft_model; Unsloth's FastLanguageModel loader dropped on 2026-09-14

Reasons

Unsloth's patched Gemma 4 E2B MoE forward crashed whenever any part of the model sat off-GPU (embedding offloaded to CPU, index_select device mismatch); E2B needs 8-10 GB and does not fit one 8 GB card.
Unsloth multi-GPU was not supported (PR #2074 closed unmerged); spanning both cards loaded cleanly but transformers' own Gemma 4 forward then hit a cuda:0/cuda:1 mismatch.
Forcing device_map={'': 0} silently disabled Unsloth's 4-bit quantization ('Switching to 16bit LoRA') and OOMed.
On gemma-3-4b-it, Unsloth's Gemma3 attention patch raised 'too many values to unpack' on real ~6300-token sequences (a 2048-token trial never reached that path).
datasets.map() via Unsloth's train_on_responses_only crashed pickling the patched tokenizer.

Measured on: 2026-09-14

Libraries

Versions

Python: 3.12.13
Torch: 2.6.0+cu124
Transformers: 5.5.0
Peft: 0.20.0
Trl: 0.24.0
Accelerate: 1.15.0
Bitsandbytes: 0.50.2
Datasets: 4.3.0
Safetensors: 0.8.0
Xformers: 0.0.29.post3
Unsloth: 2026.9.4
Unsloth zoo: 2026.9.3
Liger kernel: 0.8.2
Flash attn: 2.8.3.post1 (built, unused)
Torchao: 0.9.0 (pinned; not present in the venv on 2026-10-15)

Note: Read from the training venv's installed packages on 2026-10-15. Install dates are 2026-09-13; no later upgrade is recorded, so these are taken to be the training versions. torch was upgraded in place from 2.5.1+cu121 to 2.6.0+cu124 on 2026-09-13.
Measured on: 2026-10-15
Status: inferred

Driver and hardware

Nvidia driver: 580.178.04
Cuda toolkit installed: 13.0 (toolkit only, for the flash-attn build)
Training GPU: CUDA device 1 under CUDA_DEVICE_ORDER=PCI_BUS_ID, which is an RTX 3070 Ti 8 GB on this box; torch reports 7.65 GiB total capacity
Other GPU: RTX 3060 Ti 8 GB. Not used for training. The live Ollama server shared the cards and repeatedly reloaded models onto the training GPU (see gotchas).
Note: GPU model names read from nvidia-smi on 2026-10-15; no September log names the card. The program notes confirms 'two 8GB GPUs' and ~7.65 GB usable.
Measured on: 2026-10-15
Status: inferred

Runtime per run

Rank112 full corpus hours: Value: 11.8, 12.5; Runs: v11 (1,167 ex, 438 steps), v17 (1,189 ex, 447 steps); measured on 2026-09-25
Rank112 two epochs hours: Value: 7.9; measured on 2026-09-21
Seconds per optimizer step r16: Value: 81; measured on 2026-09-15
Note: HF Trainer's train_runtime covers only the last resumed segment when a run was killed and resumed (v2, v5).

Export path

Steps

merge: load base in bf16 on CPU, PeftModel.from_pretrained, merge_and_unload, save safetensors (merge_adapter_generic.py)
tensor-name fix: drop the 439 vision/projector tensors and rename the 444 text tensors from model.language_model.* to language_model.* (fix_tensor_names_generic.py; 883 tensors total)
llama.cpp convert_hf_to_gguf.py --outtype f16
llama.cpp llama-quantize ... Q8_0
ollama create from a Modelfile (below)

Tensor fix reason: not stated in the sources; the merged checkpoint carries Gemma 3's vision tower and a nested text-model prefix, and only the text tensors are kept
Llama cpp checkout: commit ad6c668, committed 2026-09-14 (checked 2026-10-15)
GGUF sizes GB: F16: 7.78; Q8 0: 4.14; Q4 0: 2.37
Verification lesson: a sed substitution once built the v3 Ollama tag from v2's GGUF; caught by comparing model-layer sha256 and Ollama model id before baking
Measured on: 2026-09-25
Status: inferred

Ollama modelfile

Text

FROM <path-to>/gemma3-4b-router-<version>.Q8_0.gguf
TEMPLATE """{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 }}
{{- if or (eq .Role "user") (eq .Role "system") }}<start_of_turn>user
{{ .Content }}<end_of_turn>
{{ if $last }}<start_of_turn>model
{{ end }}
{{- else if eq .Role "assistant" }}<start_of_turn>model
{{ .Content }}{{ if not $last }}<end_of_turn>
{{ end }}
{{- end }}
{{- end }}"""
PARAMETER stop <end_of_turn>
PARAMETER temperature 1
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER num_ctx 8192

Notes: system and user turns both render as a Gemma user turn, the router overrides temperature to 0 per request, the client sends num_ctx 9216 per request in production (code default since 2026-09-19); the Modelfile's 8192 is a fallback
Measured on: 2026-09-25

Q8 0 vs q4 0

Model: v5 (rank 16)
Test set: 865
Q8 0: Passed: 773; Accuracy: 0.8936; S per case: 1.506
Q4 0: Passed: 745; Accuracy: 0.8613; S per case: 1.278
Delta pp: -3.23
Speed ratio q4 over q8: 0.849
Verdict: Q8_0 stays: -3.2 pp is not worth ~15% speed.
Measured on: 2026-09-16

Release rule

Rule: veto any model that fails a held-out guard an incumbent passes; rank the rest by held-out accuracy; break ties within 0.5 pp by overall accuracy. For v17 the scripted rule was: beat the incumbent on the fair slice and add no new held-out guard breach.
Why: the earlier 'highest overall accuracy wins' rule picked v13 (rsLoRA) on 22 Sep and had to be overruled
Measured on: 2026-09-25