About: method and scope
Method
Each version was trained, served through Ollama and scored against a routing test set. Dates are UTC. Every record carries a measured-on date.
Definitions
In short: accuracy is the share of routing test cases answered with the right capability and args. Held-out and fair-slice cases are those with no training label.
- Accuracy
- passed / runnable cases. A case passes when the router returns the asserted capability and arguments, and none of the case's forbidden capabilities.
- Held out
- cases with no training label in the corpus the model was trained on. The held-out slice changed size as the case file grew, so compare held-out numbers only within one test-set era.
- Fair slice
- cases carrying no train key at all, which neither compared model ever saw (used for v16 and v17).
- Guard
- must-not-fire cases: utterances that must NOT route to a named capability (negations, quoted or pasted content, vague phrasing). Counts only; case text is excluded.
- Speed
- mean seconds per case through the production routing prompt (~5.6k tokens), model fully on GPU unless noted.
- Dates
- UTC. measured_on is the file timestamp of the result, or the date the program notes records.
Scope notes from the data
- findings.json
- Lessons from 17+ LoRA fine-tunes of gemma-3-4b-it as a command router on one 8 GB consumer GPU. Each claim is limited to this task, this base model and this recipe. Accuracy numbers are passed / runnable cases; see versions.json for definitions.
- recipe.json
- The training recipe shared by every fresh-from-base version (v1-v3, v5-v17). v4 differs only in continuing from v3's adapter at 8 epochs. Values come from the training scripts and the run headers printed in each training log.
- vram.json
- Every VRAM measurement from the program. The cross_gpu section adds one-step probes on an AMD RX 7900 XTX (24 GB) and a second NVIDIA card. Training GPU: one 8 GB NVIDIA consumer card (an RTX 3070 Ti by today's nvidia-smi; torch reports 7.65 GiB total capacity). 'mode' is probe (one forward+backward on the single longest example, no optimizer step) or real (a full Trainer run; peak = torch.cuda.max_memory_allocated at the end). All training configs are 4-bit nf4 QLoRA on unsloth/gemma-3-4b-it with the recipe.json memory fixes unless noted.
- baselines.json
- Untuned (off-the-shelf) models, measured before and during the fine-tune program through Ollama. 'routing' = the production command-routing prompt (~5.6k tokens); 'content' = 11 generation workloads (summaries, tagging, narration and similar) scored by a judge model. 'floor' tier = unambiguous cases every acceptable model must pass; 'hard' tier = deliberately ambiguous cases. Tournament numbers below were recomputed from the raw per-cycle score records by time window and checked against the program's written notes where they quote them.
- gotchas.json
- Environment and tooling failures hit during the program, with symptom, cause, fix and versions. Versions are those recorded at the time; see recipe.json for the final stack.
What is not published
The training corpus, the text of the test cases and the exact text of the safety-guard cases are deliberately not published. They appear only as counts and broad categories.
Corrections
Numbers marked unverified or inferred are flagged in the data. An inferred number was derived by the authors, not read straight from a result file. An unverified number could not be checked against a primary result.
Contact: [email protected]
License
The data and text on this site are licensed under CC BY 4.0. Use them for anything, with credit to MarvinLabs and a link to marvinlabs.org.