From e367e9e14b97b10a895ba5984efb4f5c783ba450 Mon Sep 17 00:00:00 2001 From: Hyperling Date: Wed, 21 Oct 2026 05:23:57 -0700 Subject: [PATCH] llm, begin populating thread benchmark data. --- tasks/workstation/linux/software/llm.md | 126 ++++++++++++++++++++++-- 1 file changed, 117 insertions(+), 9 deletions(-) diff --git a/tasks/workstation/linux/software/llm.md b/tasks/workstation/linux/software/llm.md index a74e68b..2a11736 100644 --- a/tasks/workstation/linux/software/llm.md +++ b/tasks/workstation/linux/software/llm.md @@ -82,15 +82,123 @@ developer when it happens. Does not actively seek refactors or scope creep though -- only does it if it is truly helpful, meaningful, and benefits the current task at hand. Can easily handle multi-file projects. -## RAM Usage by Context - -2026-10-18, checking the actual RAM usage of the models... +# Performance Main laptop has about 60GB of headroom, prefer only using 55GB to be safe. Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated cores are not very useful to LLMs. Try only using between 6 and 8 when possible. Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. +## Tokens per CPU Count + +Verifying that 8 or less cores gives better performance (less memory issues). + +"options":{"num_thread":'$t'} + +### Smaller Models + +#### qwen3.5:4b + +TBD + +#### ministral-3:14b + +TBD + +#### ministral-3:3b + +TBD + +### Larger Models + +#### qwen3.8:27b + +```console +$ ollama-cpu qwen3.8:27b + Stopping all ollama usage... +Wed Oct 21 04:42:04 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 0.7746121685803296 @ context=257000 +Wed Oct 21 04:43:28 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.6705309718150672 @ context=257000 +Wed Oct 21 04:44:12 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.104312894492908 @ context=257000 +Wed Oct 21 04:44:54 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 2.024676406965962 @ context=257000 +Wed Oct 21 04:45:39 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.0480156821567044 @ context=257000 +Wed Oct 21 04:46:19 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.184879704982382 @ context=257000 +Wed Oct 21 04:46:58 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.4228105619148526 @ context=257000 +Wed Oct 21 04:47:36 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.472564250465107 @ context=257000 +Wed Oct 21 04:48:23 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.399352715711734 @ context=257000 +Wed Oct 21 04:49:02 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.7001138401057605 @ context=257000 +Wed Oct 21 04:49:36 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.216459555312583 @ context=257000 +Wed Oct 21 04:50:27 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.228007424406279 @ context=257000 +``` + +```console +$ ollama-cpu qwen3.8:27b 32000 + Stopping all ollama usage... +Wed Oct 21 04:54:29 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 1.1388177022378907 @ context=32000 +Wed Oct 21 04:55:29 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.388676019472746 @ context=32000 +Wed Oct 21 04:56:22 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.3505103465458905 @ context=32000 +Wed Oct 21 04:56:51 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 1.9083129860906882 @ context=32000 +Wed Oct 21 04:57:30 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.727908205888872 @ context=32000 +Wed Oct 21 04:58:00 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.133796334310243 @ context=32000 +Wed Oct 21 04:58:35 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.6835810717324833 @ context=32000 +Wed Oct 21 04:59:04 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.5231895597861493 @ context=32000 +Wed Oct 21 04:59:38 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.512777473452506 @ context=32000 +Wed Oct 21 05:00:06 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.3125143809488065 @ context=32000 +Wed Oct 21 05:00:48 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.700281667801969 @ context=32000 +Wed Oct 21 05:01:20 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.6798394650721127 @ context=32000 +``` + +#### qwen3.6:27b + +TBD + +#### devstral-small-2:24b + +TBD + +#### mistral-small-3.2:??b + +TBD + +### MoE Models + +#### qwen3.6:35b + +```console +$ for c in 16000 32000 64000 96000 128000 192000 256000; do ollama_threads qwen3.6:35b $c; done + Stopping all ollama usage... +Wed Oct 21 05:02:48 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.247414587381804 @ context=16000 +Wed Oct 21 05:03:37 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.940551196032734 @ context=16000 +Wed Oct 21 05:04:09 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 13.337110328883416 @ context=16000 +Wed Oct 21 05:04:37 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 14.679419453925597 @ context=16000 +Wed Oct 21 05:05:11 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.831044087619176 @ context=16000 +Wed Oct 21 05:05:43 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 13.752831594816088 @ context=16000 +Wed Oct 21 05:06:09 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 14.603026210396049 @ context=16000 +Wed Oct 21 05:06:37 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 14.376794631809172 @ context=16000 +Wed Oct 21 05:07:05 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 8.313997206415827 @ context=16000 +Wed Oct 21 05:07:43 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.6791421803383955 @ context=16000 +Wed Oct 21 05:08:20 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.554472381570082 @ context=16000 +Wed Oct 21 05:09:04 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.6568033831630835 @ context=16000 + Stopping all ollama usage... +Wed Oct 21 05:09:46 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.558900659595539 @ context=32000 +Wed Oct 21 05:10:28 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.676325434530082 @ context=32000 +Wed Oct 21 05:11:11 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 12.007117299908693 @ context=32000 +Wed Oct 21 05:11:39 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 13.683089387469174 @ context=32000 +Wed Oct 21 05:12:04 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.447667625607968 @ context=32000 +Wed Oct 21 05:12:31 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 11.170529760956333 @ context=32000 +Wed Oct 21 05:12:57 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 13.448480713710353 @ context=32000 +Wed Oct 21 05:13:27 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 13.143672445847205 @ context=32000 +Wed Oct 21 05:13:50 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 7.9688801054121505 @ context=32000 +Wed Oct 21 05:14:21 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.064418756844208 @ context=32000 +Wed Oct 21 05:14:53 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.387867650986557 @ context=32000 +Wed Oct 21 05:15:30 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.661822236694878 @ context=32000 + Stopping all ollama usage... +... +``` + +## RAM Usage by Context + +Checking the actual RAM usage of the models... + ### `ollama_sizes m 16000` | Model Name | HDD (ls) | Context | RAM (ps) | @@ -178,7 +286,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 64000 | 18GB | | qwen3.8:27b | 17GB | 64000 | 18GB | -### 80000 +### ollama_sizes m 80000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: | @@ -207,7 +315,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 80000 | 18GB | | qwen3.8:27b | 17GB | 80000 | 18GB | -### 96000 +### ollama_sizes m 96000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: | @@ -236,7 +344,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 96000 | 18GB | | qwen3.8:27b | 17GB | 96000 | 18GB | -### 128000 +### ollama_sizes m 128000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: | @@ -265,7 +373,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 128000 | 19GB | | qwen3.8:27b | 17GB | 128000 | 19GB | -### 160000 +### ollama_sizes m 160000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: | @@ -294,7 +402,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 160000 | 19GB | | qwen3.8:27b | 17GB | 160000 | 19GB | -### 192000 +### ollama_sizes m 192000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: | @@ -323,7 +431,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. | qwen3.6:27b | 17GB | 192000 | 19GB | | qwen3.8:27b | 17GB | 192000 | 19GB | -### 224000 +### ollama_sizes m 224000 | Model Name | HDD (ls) | Context | RAM (ps) | | :------------------------ | -----------: | -------: | -----------: |