llm, begin populating thread benchmark data.

2026-10-21 05:23:57 -07:00
1 parent c604859947
commit e367e9e14b
@@ -82,15 +82,123 @@ developer when it happens. Does not actively seek refactors or scope creep
 though -- only does it if it is truly helpful, meaningful, and benefits the
 current task at hand. Can easily handle multi-file projects.

-## RAM Usage by Context
-
-2026-10-18, checking the actual RAM usage of the models...
+# Performance

 Main laptop has about 60GB of headroom, prefer only using 55GB to be safe.
 Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated
 cores are not very useful to LLMs. Try only using between 6 and 8 when possible.
 Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.

+## Tokens per CPU Count
+
+Verifying that 8 or less cores gives better performance (less memory issues).
+
+"options":{"num_thread":'$t'}
+
+### Smaller Models
+
+#### qwen3.5:4b
+
+TBD
+
+#### ministral-3:14b
+
+TBD
+
+#### ministral-3:3b
+
+TBD
+
+### Larger Models
+
+#### qwen3.8:27b
+
+```console
+$ ollama-cpu qwen3.8:27b
+ Stopping all ollama usage...
+Wed Oct 21 04:42:04 AM MST 2026 - Model qwen3.8:27b w/  1 Thread(s), tokens/sec: 0.7746121685803296 @ context=257000
+Wed Oct 21 04:43:28 AM MST 2026 - Model qwen3.8:27b w/  2 Thread(s), tokens/sec: 1.6705309718150672 @ context=257000
+Wed Oct 21 04:44:12 AM MST 2026 - Model qwen3.8:27b w/  3 Thread(s), tokens/sec: 2.104312894492908 @ context=257000
+Wed Oct 21 04:44:54 AM MST 2026 - Model qwen3.8:27b w/  4 Thread(s), tokens/sec: 2.024676406965962 @ context=257000
+Wed Oct 21 04:45:39 AM MST 2026 - Model qwen3.8:27b w/  5 Thread(s), tokens/sec: 2.0480156821567044 @ context=257000
+Wed Oct 21 04:46:19 AM MST 2026 - Model qwen3.8:27b w/  6 Thread(s), tokens/sec: 2.184879704982382 @ context=257000
+Wed Oct 21 04:46:58 AM MST 2026 - Model qwen3.8:27b w/  7 Thread(s), tokens/sec: 2.4228105619148526 @ context=257000
+Wed Oct 21 04:47:36 AM MST 2026 - Model qwen3.8:27b w/  8 Thread(s), tokens/sec: 2.472564250465107 @ context=257000
+Wed Oct 21 04:48:23 AM MST 2026 - Model qwen3.8:27b w/  9 Thread(s), tokens/sec: 2.399352715711734 @ context=257000
+Wed Oct 21 04:49:02 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.7001138401057605 @ context=257000
+Wed Oct 21 04:49:36 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.216459555312583 @ context=257000
+Wed Oct 21 04:50:27 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.228007424406279 @ context=257000
+```
+
+```console
+$ ollama-cpu qwen3.8:27b 32000
+ Stopping all ollama usage...
+Wed Oct 21 04:54:29 AM MST 2026 - Model qwen3.8:27b w/  1 Thread(s), tokens/sec: 1.1388177022378907 @ context=32000
+Wed Oct 21 04:55:29 AM MST 2026 - Model qwen3.8:27b w/  2 Thread(s), tokens/sec: 1.388676019472746 @ context=32000
+Wed Oct 21 04:56:22 AM MST 2026 - Model qwen3.8:27b w/  3 Thread(s), tokens/sec: 2.3505103465458905 @ context=32000
+Wed Oct 21 04:56:51 AM MST 2026 - Model qwen3.8:27b w/  4 Thread(s), tokens/sec: 1.9083129860906882 @ context=32000
+Wed Oct 21 04:57:30 AM MST 2026 - Model qwen3.8:27b w/  5 Thread(s), tokens/sec: 2.727908205888872 @ context=32000
+Wed Oct 21 04:58:00 AM MST 2026 - Model qwen3.8:27b w/  6 Thread(s), tokens/sec: 2.133796334310243 @ context=32000
+Wed Oct 21 04:58:35 AM MST 2026 - Model qwen3.8:27b w/  7 Thread(s), tokens/sec: 2.6835810717324833 @ context=32000
+Wed Oct 21 04:59:04 AM MST 2026 - Model qwen3.8:27b w/  8 Thread(s), tokens/sec: 2.5231895597861493 @ context=32000
+Wed Oct 21 04:59:38 AM MST 2026 - Model qwen3.8:27b w/  9 Thread(s), tokens/sec: 2.512777473452506 @ context=32000
+Wed Oct 21 05:00:06 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.3125143809488065 @ context=32000
+Wed Oct 21 05:00:48 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.700281667801969 @ context=32000
+Wed Oct 21 05:01:20 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.6798394650721127 @ context=32000
+```
+
+#### qwen3.6:27b
+
+TBD
+
+#### devstral-small-2:24b
+
+TBD
+
+#### mistral-small-3.2:??b
+
+TBD
+
+### MoE Models
+
+#### qwen3.6:35b
+
+```console
+$ for c in 16000 32000 64000 96000 128000 192000 256000; do ollama_threads qwen3.6:35b $c; done
+ Stopping all ollama usage...
+Wed Oct 21 05:02:48 AM MST 2026 - Model qwen3.6:35b w/  1 Thread(s), tokens/sec:  6.247414587381804 @ context=16000
+Wed Oct 21 05:03:37 AM MST 2026 - Model qwen3.6:35b w/  2 Thread(s), tokens/sec: 10.940551196032734 @ context=16000
+Wed Oct 21 05:04:09 AM MST 2026 - Model qwen3.6:35b w/  3 Thread(s), tokens/sec: 13.337110328883416 @ context=16000
+Wed Oct 21 05:04:37 AM MST 2026 - Model qwen3.6:35b w/  4 Thread(s), tokens/sec: 14.679419453925597 @ context=16000
+Wed Oct 21 05:05:11 AM MST 2026 - Model qwen3.6:35b w/  5 Thread(s), tokens/sec: 13.831044087619176 @ context=16000
+Wed Oct 21 05:05:43 AM MST 2026 - Model qwen3.6:35b w/  6 Thread(s), tokens/sec: 13.752831594816088 @ context=16000
+Wed Oct 21 05:06:09 AM MST 2026 - Model qwen3.6:35b w/  7 Thread(s), tokens/sec: 14.603026210396049 @ context=16000
+Wed Oct 21 05:06:37 AM MST 2026 - Model qwen3.6:35b w/  8 Thread(s), tokens/sec: 14.376794631809172 @ context=16000
+Wed Oct 21 05:07:05 AM MST 2026 - Model qwen3.6:35b w/  9 Thread(s), tokens/sec:  8.313997206415827 @ context=16000
+Wed Oct 21 05:07:43 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec:  7.6791421803383955 @ context=16000
+Wed Oct 21 05:08:20 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec:  7.554472381570082 @ context=16000
+Wed Oct 21 05:09:04 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec:  5.6568033831630835 @ context=16000
+ Stopping all ollama usage...
+Wed Oct 21 05:09:46 AM MST 2026 - Model qwen3.6:35b w/  1 Thread(s), tokens/sec:  6.558900659595539 @ context=32000
+Wed Oct 21 05:10:28 AM MST 2026 - Model qwen3.6:35b w/  2 Thread(s), tokens/sec: 10.676325434530082 @ context=32000
+Wed Oct 21 05:11:11 AM MST 2026 - Model qwen3.6:35b w/  3 Thread(s), tokens/sec: 12.007117299908693 @ context=32000
+Wed Oct 21 05:11:39 AM MST 2026 - Model qwen3.6:35b w/  4 Thread(s), tokens/sec: 13.683089387469174 @ context=32000
+Wed Oct 21 05:12:04 AM MST 2026 - Model qwen3.6:35b w/  5 Thread(s), tokens/sec: 13.447667625607968 @ context=32000
+Wed Oct 21 05:12:31 AM MST 2026 - Model qwen3.6:35b w/  6 Thread(s), tokens/sec: 11.170529760956333 @ context=32000
+Wed Oct 21 05:12:57 AM MST 2026 - Model qwen3.6:35b w/  7 Thread(s), tokens/sec: 13.448480713710353 @ context=32000
+Wed Oct 21 05:13:27 AM MST 2026 - Model qwen3.6:35b w/  8 Thread(s), tokens/sec: 13.143672445847205 @ context=32000
+Wed Oct 21 05:13:50 AM MST 2026 - Model qwen3.6:35b w/  9 Thread(s), tokens/sec:  7.9688801054121505 @ context=32000
+Wed Oct 21 05:14:21 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec:  7.064418756844208 @ context=32000
+Wed Oct 21 05:14:53 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec:  7.387867650986557 @ context=32000
+Wed Oct 21 05:15:30 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec:  5.661822236694878 @ context=32000
+ Stopping all ollama usage...
+...
+```
+
+## RAM Usage by Context
+
+Checking the actual RAM usage of the models...
+
 ### `ollama_sizes m 16000`

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
@@ -178,7 +286,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |    64000 |         18GB |
 | qwen3.8:27b               |         17GB |    64000 |         18GB |

-### 80000
+### ollama_sizes m 80000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |
@@ -207,7 +315,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |    80000 |         18GB |
 | qwen3.8:27b               |         17GB |    80000 |         18GB |

-### 96000
+### ollama_sizes m 96000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |
@@ -236,7 +344,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |    96000 |         18GB |
 | qwen3.8:27b               |         17GB |    96000 |         18GB |

-### 128000
+### ollama_sizes m 128000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |
@@ -265,7 +373,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |   128000 |         19GB |
 | qwen3.8:27b               |         17GB |   128000 |         19GB |

-### 160000
+### ollama_sizes m 160000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |
@@ -294,7 +402,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |   160000 |         19GB |
 | qwen3.8:27b               |         17GB |   160000 |         19GB |

-### 192000
+### ollama_sizes m 192000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |
@@ -323,7 +431,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
 | qwen3.6:27b               |         17GB |   192000 |         19GB |
 | qwen3.8:27b               |         17GB |   192000 |         19GB |

-### 224000
+### ollama_sizes m 224000

 | Model Name                |     HDD (ls) |  Context |     RAM (ps) |
 | :------------------------ | -----------: | -------: | -----------: |