Inference speed

Markdown: inference.md · Data: inference.json

Ollama inference speed for seven small models on one AMD Radeon RX 7900 XTX (24 GB, Windows 11, Ollama 0.30.10). Each model was unloaded first, then given synthetic prompts of about 1,550 and 4,550 tokens, 3 trials each, with num_ctx 9216, temperature 0 and 200 output tokens. Each trial changes the first line, so no prompt prefix is reused from cache. Cold load is the first call's load time. The speed medians leave out that first cold trial, because its prompt speed includes warm-up. Every model was fully resident in VRAM (size_vram equals size), and no prompt was truncated. In the summary, speeds are median tokens per second; short means a prompt of about 1,550 tokens and long about 4,550. Gen speed is output generation.

GPU

Model: AMD Radeon RX 7900 XTX
Arch: gfx1100
Total MB: 24560
OS: Windows 11 (build 26200)
Driver: AMD Adrenalin 26.7.1 (32.0.31035.1003)
Ollama: 0.30.10

Summary

Model	Resident MB	Cold load s	Prompt speed short	Prompt speed long	Gen speed short	Gen speed long	Measured on
gemma3:4b	2891	7.74	4411.4	5039.9	132.6	128.6	2026-10-15
gemma4:e2b-it-qat	1762	6.17	6982.2	7458.3	154.6	152.2	2026-10-15
granite4.2:3b-q8_0	4313	4.84	5707.4	5219.5	127.0	121.3	2026-10-15
gemma4:e4b-it-qat	3133	7.69	3941.8	4258.9	116.5	113.5	2026-10-15
qwen3.5:9b	5642	8.82	2588.9	2704.7	84.9	84.2	2026-10-15
gemma4:12b-it-qat	7601	9.46	1526.1	1661.9	68.4	66.4	2026-10-15
gpt-oss:20b	12261	14.69	3730.2	4118.1	123.0	116.2	2026-10-15

Trials

Model	Prompt words	Trial	Prompt tokens	Load s	Prompt s	Prompt tok per s	Gen tokens	Gen s	Gen tok per s	Total s	Size VRAM MB	Context length	Measured on
gemma3:4b	1500	0	1551	7.74	0.532	2915.4	200	1.659	120.6	9.94	2891	9216	2026-10-15
gemma3:4b	1500	1	1551	0.35	0.364	4261.0	200	1.511	132.4	2.45	2891	9216	2026-10-15
gemma3:4b	1500	2	1551	0.32	0.34	4561.8	200	1.506	132.8	2.28	2891	9216	2026-10-15
gemma3:4b	4500	0	4551	0.32	0.893	5096.3	200	1.547	129.3	2.89	2891	9216	2026-10-15
gemma3:4b	4500	1	4551	0.33	0.93	4893.5	200	1.555	128.6	2.99	2891	9216	2026-10-15
gemma3:4b	4500	2	4551	0.32	0.903	5039.9	200	1.571	127.3	2.97	2891	9216	2026-10-15
gemma4:e2b-it-qat	1500	0	1557	6.17	0.336	4633.9	200	1.298	154.1	7.81	1762	9216	2026-10-15
gemma4:e2b-it-qat	1500	1	1557	0.32	0.224	6950.9	200	1.29	155.0	1.86	1762	9216	2026-10-15
gemma4:e2b-it-qat	1500	2	1557	0.32	0.222	7013.5	200	1.298	154.1	1.86	1762	9216	2026-10-15
gemma4:e2b-it-qat	4500	0	4557	0.32	0.609	7482.8	200	1.314	152.2	2.29	1762	9216	2026-10-15
gemma4:e2b-it-qat	4500	1	4557	0.32	0.611	7458.3	200	1.308	152.9	2.29	1762	9216	2026-10-15
gemma4:e2b-it-qat	4500	2	4557	0.31	0.612	7446.1	200	1.316	152.0	2.3	1762	9216	2026-10-15
granite4.2:3b-q8_0	1500	0	1541	4.84	0.294	5241.5	200	1.564	127.9	6.71	4313	9216	2026-10-15
granite4.2:3b-q8_0	1500	1	1541	0.09	0.27	5707.4	200	1.567	127.6	2.0	4313	9216	2026-10-15
granite4.2:3b-q8_0	1500	2	1541	0.08	0.27	5707.4	200	1.582	126.4	2.0	4313	9216	2026-10-15
granite4.2:3b-q8_0	4500	0	4541	0.08	0.868	5231.6	200	1.65	121.2	2.68	4313	9216	2026-10-15
granite4.2:3b-q8_0	4500	1	4541	0.09	0.87	5219.5	200	1.649	121.3	2.78	4313	9216	2026-10-15
granite4.2:3b-q8_0	4500	2	4541	0.08	0.878	5172.0	200	1.649	121.3	2.79	4313	9216	2026-10-15
gemma4:e4b-it-qat	1500	0	1557	7.69	0.496	3139.1	200	1.732	115.5	9.93	3133	9216	2026-10-15
gemma4:e4b-it-qat	1500	1	1557	0.31	0.395	3941.8	200	1.718	116.4	2.48	3133	9216	2026-10-15
gemma4:e4b-it-qat	1500	2	1557	0.31	0.395	3941.8	200	1.716	116.6	2.47	3133	9216	2026-10-15
gemma4:e4b-it-qat	4500	0	4557	0.31	1.065	4278.9	200	1.778	112.5	3.23	3133	9216	2026-10-15
gemma4:e4b-it-qat	4500	1	4557	0.31	1.07	4258.9	200	1.762	113.5	3.23	3133	9216	2026-10-15
gemma4:e4b-it-qat	4500	2	4557	0.32	1.071	4254.9	200	1.762	113.5	3.25	3133	9216	2026-10-15
qwen3.5:9b	1500	0	1552	8.82	0.68	2282.4	200	2.383	83.9	11.89	5642	9216	2026-10-15
qwen3.5:9b	1500	1	1552	0.26	0.597	2599.7	200	2.359	84.8	3.3	5642	9216	2026-10-15
qwen3.5:9b	1500	2	1552	0.23	0.602	2578.1	200	2.35	85.1	3.25	5642	9216	2026-10-15
qwen3.5:9b	4500	0	4552	0.23	1.673	2720.9	200	2.375	84.2	4.37	5642	9216	2026-10-15
qwen3.5:9b	4500	1	4552	0.24	1.683	2704.7	200	2.378	84.1	4.41	5642	9216	2026-10-15
qwen3.5:9b	4500	2	4552	0.24	1.683	2704.7	200	2.374	84.2	4.41	5642	9216	2026-10-15
gemma4:12b-it-qat	1500	0	1557	9.46	0.972	1601.9	200	2.851	70.2	13.29	7601	9216	2026-10-15
gemma4:12b-it-qat	1500	1	1557	0.33	1.037	1501.4	200	2.851	70.2	4.5	7601	9216	2026-10-15
gemma4:12b-it-qat	1500	2	1557	0.32	1.004	1550.8	200	3.003	66.6	4.62	7601	9216	2026-10-15
gemma4:12b-it-qat	4500	0	4557	0.45	2.839	1605.1	200	2.964	67.5	6.68	7601	9216	2026-10-15
gemma4:12b-it-qat	4500	1	4557	0.34	2.742	1661.9	200	3.119	64.1	6.57	7601	9216	2026-10-15
gemma4:12b-it-qat	4500	2	4557	0.33	2.74	1663.1	200	3.01	66.4	6.47	7601	9216	2026-10-15
gpt-oss:20b	1500	0	1604	14.69	0.81	1980.2	200	1.617	123.7	17.13	12261	9216	2026-10-15
gpt-oss:20b	1500	1	1604	0.28	0.429	3738.9	200	1.623	123.2	2.37	12261	9216	2026-10-15
gpt-oss:20b	1500	2	1604	0.27	0.431	3721.6	200	1.63	122.7	2.38	12261	9216	2026-10-15
gpt-oss:20b	4500	0	4604	0.28	1.121	4107.0	200	1.721	116.2	3.19	12261	9216	2026-10-15
gpt-oss:20b	4500	1	4604	0.28	1.117	4121.8	200	1.732	115.5	3.22	12261	9216	2026-10-15
gpt-oss:20b	4500	2	4604	0.28	1.118	4118.1	200	1.715	116.6	3.21	12261	9216	2026-10-15

Measured on

2026-10-15