llm, begin populating thread benchmark data.
This commit is contained in:
1 parent
c604859947
commit
e367e9e14b
1 file changed
+117
-9
@@ -82,15 +82,123 @@ developer when it happens. Does not actively seek refactors or scope creep
|
|||||||
though -- only does it if it is truly helpful, meaningful, and benefits the
|
though -- only does it if it is truly helpful, meaningful, and benefits the
|
||||||
current task at hand. Can easily handle multi-file projects.
|
current task at hand. Can easily handle multi-file projects.
|
||||||
|
|
||||||
## RAM Usage by Context
|
# Performance
|
||||||
|
|
||||||
2026-10-18, checking the actual RAM usage of the models...
|
|
||||||
|
|
||||||
Main laptop has about 60GB of headroom, prefer only using 55GB to be safe.
|
Main laptop has about 60GB of headroom, prefer only using 55GB to be safe.
|
||||||
Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated
|
Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated
|
||||||
cores are not very useful to LLMs. Try only using between 6 and 8 when possible.
|
cores are not very useful to LLMs. Try only using between 6 and 8 when possible.
|
||||||
Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
||||||
|
|
||||||
|
## Tokens per CPU Count
|
||||||
|
|
||||||
|
Verifying that 8 or less cores gives better performance (less memory issues).
|
||||||
|
|
||||||
|
"options":{"num_thread":'$t'}
|
||||||
|
|
||||||
|
### Smaller Models
|
||||||
|
|
||||||
|
#### qwen3.5:4b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
#### ministral-3:14b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
#### ministral-3:3b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
### Larger Models
|
||||||
|
|
||||||
|
#### qwen3.8:27b
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ ollama-cpu qwen3.8:27b
|
||||||
|
Stopping all ollama usage...
|
||||||
|
Wed Oct 21 04:42:04 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 0.7746121685803296 @ context=257000
|
||||||
|
Wed Oct 21 04:43:28 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.6705309718150672 @ context=257000
|
||||||
|
Wed Oct 21 04:44:12 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.104312894492908 @ context=257000
|
||||||
|
Wed Oct 21 04:44:54 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 2.024676406965962 @ context=257000
|
||||||
|
Wed Oct 21 04:45:39 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.0480156821567044 @ context=257000
|
||||||
|
Wed Oct 21 04:46:19 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.184879704982382 @ context=257000
|
||||||
|
Wed Oct 21 04:46:58 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.4228105619148526 @ context=257000
|
||||||
|
Wed Oct 21 04:47:36 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.472564250465107 @ context=257000
|
||||||
|
Wed Oct 21 04:48:23 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.399352715711734 @ context=257000
|
||||||
|
Wed Oct 21 04:49:02 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.7001138401057605 @ context=257000
|
||||||
|
Wed Oct 21 04:49:36 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.216459555312583 @ context=257000
|
||||||
|
Wed Oct 21 04:50:27 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.228007424406279 @ context=257000
|
||||||
|
```
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ ollama-cpu qwen3.8:27b 32000
|
||||||
|
Stopping all ollama usage...
|
||||||
|
Wed Oct 21 04:54:29 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 1.1388177022378907 @ context=32000
|
||||||
|
Wed Oct 21 04:55:29 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.388676019472746 @ context=32000
|
||||||
|
Wed Oct 21 04:56:22 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.3505103465458905 @ context=32000
|
||||||
|
Wed Oct 21 04:56:51 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 1.9083129860906882 @ context=32000
|
||||||
|
Wed Oct 21 04:57:30 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.727908205888872 @ context=32000
|
||||||
|
Wed Oct 21 04:58:00 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.133796334310243 @ context=32000
|
||||||
|
Wed Oct 21 04:58:35 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.6835810717324833 @ context=32000
|
||||||
|
Wed Oct 21 04:59:04 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.5231895597861493 @ context=32000
|
||||||
|
Wed Oct 21 04:59:38 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.512777473452506 @ context=32000
|
||||||
|
Wed Oct 21 05:00:06 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.3125143809488065 @ context=32000
|
||||||
|
Wed Oct 21 05:00:48 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.700281667801969 @ context=32000
|
||||||
|
Wed Oct 21 05:01:20 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.6798394650721127 @ context=32000
|
||||||
|
```
|
||||||
|
|
||||||
|
#### qwen3.6:27b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
#### devstral-small-2:24b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
#### mistral-small-3.2:??b
|
||||||
|
|
||||||
|
TBD
|
||||||
|
|
||||||
|
### MoE Models
|
||||||
|
|
||||||
|
#### qwen3.6:35b
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ for c in 16000 32000 64000 96000 128000 192000 256000; do ollama_threads qwen3.6:35b $c; done
|
||||||
|
Stopping all ollama usage...
|
||||||
|
Wed Oct 21 05:02:48 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.247414587381804 @ context=16000
|
||||||
|
Wed Oct 21 05:03:37 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.940551196032734 @ context=16000
|
||||||
|
Wed Oct 21 05:04:09 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 13.337110328883416 @ context=16000
|
||||||
|
Wed Oct 21 05:04:37 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 14.679419453925597 @ context=16000
|
||||||
|
Wed Oct 21 05:05:11 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.831044087619176 @ context=16000
|
||||||
|
Wed Oct 21 05:05:43 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 13.752831594816088 @ context=16000
|
||||||
|
Wed Oct 21 05:06:09 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 14.603026210396049 @ context=16000
|
||||||
|
Wed Oct 21 05:06:37 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 14.376794631809172 @ context=16000
|
||||||
|
Wed Oct 21 05:07:05 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 8.313997206415827 @ context=16000
|
||||||
|
Wed Oct 21 05:07:43 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.6791421803383955 @ context=16000
|
||||||
|
Wed Oct 21 05:08:20 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.554472381570082 @ context=16000
|
||||||
|
Wed Oct 21 05:09:04 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.6568033831630835 @ context=16000
|
||||||
|
Stopping all ollama usage...
|
||||||
|
Wed Oct 21 05:09:46 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.558900659595539 @ context=32000
|
||||||
|
Wed Oct 21 05:10:28 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.676325434530082 @ context=32000
|
||||||
|
Wed Oct 21 05:11:11 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 12.007117299908693 @ context=32000
|
||||||
|
Wed Oct 21 05:11:39 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 13.683089387469174 @ context=32000
|
||||||
|
Wed Oct 21 05:12:04 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.447667625607968 @ context=32000
|
||||||
|
Wed Oct 21 05:12:31 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 11.170529760956333 @ context=32000
|
||||||
|
Wed Oct 21 05:12:57 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 13.448480713710353 @ context=32000
|
||||||
|
Wed Oct 21 05:13:27 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 13.143672445847205 @ context=32000
|
||||||
|
Wed Oct 21 05:13:50 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 7.9688801054121505 @ context=32000
|
||||||
|
Wed Oct 21 05:14:21 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.064418756844208 @ context=32000
|
||||||
|
Wed Oct 21 05:14:53 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.387867650986557 @ context=32000
|
||||||
|
Wed Oct 21 05:15:30 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.661822236694878 @ context=32000
|
||||||
|
Stopping all ollama usage...
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
## RAM Usage by Context
|
||||||
|
|
||||||
|
Checking the actual RAM usage of the models...
|
||||||
|
|
||||||
### `ollama_sizes m 16000`
|
### `ollama_sizes m 16000`
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
@@ -178,7 +286,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 64000 | 18GB |
|
| qwen3.6:27b | 17GB | 64000 | 18GB |
|
||||||
| qwen3.8:27b | 17GB | 64000 | 18GB |
|
| qwen3.8:27b | 17GB | 64000 | 18GB |
|
||||||
|
|
||||||
### 80000
|
### ollama_sizes m 80000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
@@ -207,7 +315,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 80000 | 18GB |
|
| qwen3.6:27b | 17GB | 80000 | 18GB |
|
||||||
| qwen3.8:27b | 17GB | 80000 | 18GB |
|
| qwen3.8:27b | 17GB | 80000 | 18GB |
|
||||||
|
|
||||||
### 96000
|
### ollama_sizes m 96000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
@@ -236,7 +344,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 96000 | 18GB |
|
| qwen3.6:27b | 17GB | 96000 | 18GB |
|
||||||
| qwen3.8:27b | 17GB | 96000 | 18GB |
|
| qwen3.8:27b | 17GB | 96000 | 18GB |
|
||||||
|
|
||||||
### 128000
|
### ollama_sizes m 128000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
@@ -265,7 +373,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 128000 | 19GB |
|
| qwen3.6:27b | 17GB | 128000 | 19GB |
|
||||||
| qwen3.8:27b | 17GB | 128000 | 19GB |
|
| qwen3.8:27b | 17GB | 128000 | 19GB |
|
||||||
|
|
||||||
### 160000
|
### ollama_sizes m 160000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
@@ -294,7 +402,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 160000 | 19GB |
|
| qwen3.6:27b | 17GB | 160000 | 19GB |
|
||||||
| qwen3.8:27b | 17GB | 160000 | 19GB |
|
| qwen3.8:27b | 17GB | 160000 | 19GB |
|
||||||
|
|
||||||
### 192000
|
### ollama_sizes m 192000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
@@ -323,7 +431,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
|
|||||||
| qwen3.6:27b | 17GB | 192000 | 19GB |
|
| qwen3.6:27b | 17GB | 192000 | 19GB |
|
||||||
| qwen3.8:27b | 17GB | 192000 | 19GB |
|
| qwen3.8:27b | 17GB | 192000 | 19GB |
|
||||||
|
|
||||||
### 224000
|
### ollama_sizes m 224000
|
||||||
|
|
||||||
| Model Name | HDD (ls) | Context | RAM (ps) |
|
| Model Name | HDD (ls) | Context | RAM (ps) |
|
||||||
| :------------------------ | -----------: | -------: | -----------: |
|
| :------------------------ | -----------: | -------: | -----------: |
|
||||||
|
|||||||
Reference in new issue
Block a user