me/env-ansible
1
1 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: me/env-ansible:85e60ef60c36a233be88b8195e6ff27982ff186c
me/env-ansible:prod me/env-ansible:dev me/env-ansible:stage me/env-ansible:main
..
compare: me/env-ansible:e367e9e14b97b10a895ba5984efb4f5c783ba450
me/env-ansible:dev me/env-ansible:prod me/env-ansible:stage me/env-ansible:main
5 Commits
85e60ef60c .. e367e9e14b
Author SHA1 Message Date
me e367e9e14b llm, begin populating thread benchmark data. 2026-10-21 05:23:57 -07:00
me c604859947 llm, add elapsed time to ollama_threads. 2026-10-21 05:23:41 -07:00
me 7f7f2a2edb llm, fix context sending in ollama_threads. 2026-10-21 04:55:01 -07:00
me 92be32345a llm, add context param to ollama_threads. 2026-10-21 04:51:46 -07:00
me 5719ec6b60 llm, add optional open-terminal package list, helpful but required Internet, may have to do an Internet check and then choose which docker run command to execute. Also fix and enhance the ollama_thread checker. 2026-10-21 04:47:16 -07:00
2 changed files with 146 additions and 13 deletions
Expand all files Collapse all files

No files matched your search

tasks/workstation/linux/software/llm.md
+117 -9
View File
@@ -82,15 +82,123 @@ developer when it happens. Does not actively seek refactors or scope creep
though -- only does it if it is truly helpful, meaningful, and benefits the though -- only does it if it is truly helpful, meaningful, and benefits the
current task at hand. Can easily handle multi-file projects. current task at hand. Can easily handle multi-file projects.
## RAM Usage by Context # Performance
2026-10-18, checking the actual RAM usage of the models...
Main laptop has about 60GB of headroom, prefer only using 55GB to be safe. Main laptop has about 60GB of headroom, prefer only using 55GB to be safe.
Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated
cores are not very useful to LLMs. Try only using between 6 and 8 when possible. cores are not very useful to LLMs. Try only using between 6 and 8 when possible.
Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
## Tokens per CPU Count
Verifying that 8 or less cores gives better performance (less memory issues).
"options":{"num_thread":'$t'}
### Smaller Models
#### qwen3.5:4b
TBD
#### ministral-3:14b
TBD
#### ministral-3:3b
TBD
### Larger Models
#### qwen3.8:27b
```console
$ ollama-cpu qwen3.8:27b
Stopping all ollama usage...
Wed Oct 21 04:42:04 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 0.7746121685803296 @ context=257000
Wed Oct 21 04:43:28 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.6705309718150672 @ context=257000
Wed Oct 21 04:44:12 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.104312894492908 @ context=257000
Wed Oct 21 04:44:54 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 2.024676406965962 @ context=257000
Wed Oct 21 04:45:39 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.0480156821567044 @ context=257000
Wed Oct 21 04:46:19 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.184879704982382 @ context=257000
Wed Oct 21 04:46:58 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.4228105619148526 @ context=257000
Wed Oct 21 04:47:36 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.472564250465107 @ context=257000
Wed Oct 21 04:48:23 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.399352715711734 @ context=257000
Wed Oct 21 04:49:02 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.7001138401057605 @ context=257000
Wed Oct 21 04:49:36 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.216459555312583 @ context=257000
Wed Oct 21 04:50:27 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.228007424406279 @ context=257000
```
```console
$ ollama-cpu qwen3.8:27b 32000
Stopping all ollama usage...
Wed Oct 21 04:54:29 AM MST 2026 - Model qwen3.8:27b w/ 1 Thread(s), tokens/sec: 1.1388177022378907 @ context=32000
Wed Oct 21 04:55:29 AM MST 2026 - Model qwen3.8:27b w/ 2 Thread(s), tokens/sec: 1.388676019472746 @ context=32000
Wed Oct 21 04:56:22 AM MST 2026 - Model qwen3.8:27b w/ 3 Thread(s), tokens/sec: 2.3505103465458905 @ context=32000
Wed Oct 21 04:56:51 AM MST 2026 - Model qwen3.8:27b w/ 4 Thread(s), tokens/sec: 1.9083129860906882 @ context=32000
Wed Oct 21 04:57:30 AM MST 2026 - Model qwen3.8:27b w/ 5 Thread(s), tokens/sec: 2.727908205888872 @ context=32000
Wed Oct 21 04:58:00 AM MST 2026 - Model qwen3.8:27b w/ 6 Thread(s), tokens/sec: 2.133796334310243 @ context=32000
Wed Oct 21 04:58:35 AM MST 2026 - Model qwen3.8:27b w/ 7 Thread(s), tokens/sec: 2.6835810717324833 @ context=32000
Wed Oct 21 04:59:04 AM MST 2026 - Model qwen3.8:27b w/ 8 Thread(s), tokens/sec: 2.5231895597861493 @ context=32000
Wed Oct 21 04:59:38 AM MST 2026 - Model qwen3.8:27b w/ 9 Thread(s), tokens/sec: 2.512777473452506 @ context=32000
Wed Oct 21 05:00:06 AM MST 2026 - Model qwen3.8:27b w/ 10 Thread(s), tokens/sec: 2.3125143809488065 @ context=32000
Wed Oct 21 05:00:48 AM MST 2026 - Model qwen3.8:27b w/ 11 Thread(s), tokens/sec: 2.700281667801969 @ context=32000
Wed Oct 21 05:01:20 AM MST 2026 - Model qwen3.8:27b w/ 12 Thread(s), tokens/sec: 2.6798394650721127 @ context=32000
```
#### qwen3.6:27b
TBD
#### devstral-small-2:24b
TBD
#### mistral-small-3.2:??b
TBD
### MoE Models
#### qwen3.6:35b
```console
$ for c in 16000 32000 64000 96000 128000 192000 256000; do ollama_threads qwen3.6:35b $c; done
Stopping all ollama usage...
Wed Oct 21 05:02:48 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.247414587381804 @ context=16000
Wed Oct 21 05:03:37 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.940551196032734 @ context=16000
Wed Oct 21 05:04:09 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 13.337110328883416 @ context=16000
Wed Oct 21 05:04:37 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 14.679419453925597 @ context=16000
Wed Oct 21 05:05:11 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.831044087619176 @ context=16000
Wed Oct 21 05:05:43 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 13.752831594816088 @ context=16000
Wed Oct 21 05:06:09 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 14.603026210396049 @ context=16000
Wed Oct 21 05:06:37 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 14.376794631809172 @ context=16000
Wed Oct 21 05:07:05 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 8.313997206415827 @ context=16000
Wed Oct 21 05:07:43 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.6791421803383955 @ context=16000
Wed Oct 21 05:08:20 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.554472381570082 @ context=16000
Wed Oct 21 05:09:04 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.6568033831630835 @ context=16000
Stopping all ollama usage...
Wed Oct 21 05:09:46 AM MST 2026 - Model qwen3.6:35b w/ 1 Thread(s), tokens/sec: 6.558900659595539 @ context=32000
Wed Oct 21 05:10:28 AM MST 2026 - Model qwen3.6:35b w/ 2 Thread(s), tokens/sec: 10.676325434530082 @ context=32000
Wed Oct 21 05:11:11 AM MST 2026 - Model qwen3.6:35b w/ 3 Thread(s), tokens/sec: 12.007117299908693 @ context=32000
Wed Oct 21 05:11:39 AM MST 2026 - Model qwen3.6:35b w/ 4 Thread(s), tokens/sec: 13.683089387469174 @ context=32000
Wed Oct 21 05:12:04 AM MST 2026 - Model qwen3.6:35b w/ 5 Thread(s), tokens/sec: 13.447667625607968 @ context=32000
Wed Oct 21 05:12:31 AM MST 2026 - Model qwen3.6:35b w/ 6 Thread(s), tokens/sec: 11.170529760956333 @ context=32000
Wed Oct 21 05:12:57 AM MST 2026 - Model qwen3.6:35b w/ 7 Thread(s), tokens/sec: 13.448480713710353 @ context=32000
Wed Oct 21 05:13:27 AM MST 2026 - Model qwen3.6:35b w/ 8 Thread(s), tokens/sec: 13.143672445847205 @ context=32000
Wed Oct 21 05:13:50 AM MST 2026 - Model qwen3.6:35b w/ 9 Thread(s), tokens/sec: 7.9688801054121505 @ context=32000
Wed Oct 21 05:14:21 AM MST 2026 - Model qwen3.6:35b w/ 10 Thread(s), tokens/sec: 7.064418756844208 @ context=32000
Wed Oct 21 05:14:53 AM MST 2026 - Model qwen3.6:35b w/ 11 Thread(s), tokens/sec: 7.387867650986557 @ context=32000
Wed Oct 21 05:15:30 AM MST 2026 - Model qwen3.6:35b w/ 12 Thread(s), tokens/sec: 5.661822236694878 @ context=32000
Stopping all ollama usage...
...
```
## RAM Usage by Context
Checking the actual RAM usage of the models...
### `ollama_sizes m 16000` ### `ollama_sizes m 16000`
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
@@ -178,7 +286,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 64000 | 18GB | | qwen3.6:27b | 17GB | 64000 | 18GB |
| qwen3.8:27b | 17GB | 64000 | 18GB | | qwen3.8:27b | 17GB | 64000 | 18GB |
### 80000 ### ollama_sizes m 80000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
@@ -207,7 +315,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 80000 | 18GB | | qwen3.6:27b | 17GB | 80000 | 18GB |
| qwen3.8:27b | 17GB | 80000 | 18GB | | qwen3.8:27b | 17GB | 80000 | 18GB |
### 96000 ### ollama_sizes m 96000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
@@ -236,7 +344,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 96000 | 18GB | | qwen3.6:27b | 17GB | 96000 | 18GB |
| qwen3.8:27b | 17GB | 96000 | 18GB | | qwen3.8:27b | 17GB | 96000 | 18GB |
### 128000 ### ollama_sizes m 128000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
@@ -265,7 +373,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 128000 | 19GB | | qwen3.6:27b | 17GB | 128000 | 19GB |
| qwen3.8:27b | 17GB | 128000 | 19GB | | qwen3.8:27b | 17GB | 128000 | 19GB |
### 160000 ### ollama_sizes m 160000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
@@ -294,7 +402,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 160000 | 19GB | | qwen3.6:27b | 17GB | 160000 | 19GB |
| qwen3.8:27b | 17GB | 160000 | 19GB | | qwen3.8:27b | 17GB | 160000 | 19GB |
### 192000 ### ollama_sizes m 192000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
@@ -323,7 +431,7 @@ Research says it is not worth enabling the iGPU for Intel XE, CPUs are better.
| qwen3.6:27b | 17GB | 192000 | 19GB | | qwen3.6:27b | 17GB | 192000 | 19GB |
| qwen3.8:27b | 17GB | 192000 | 19GB | | qwen3.8:27b | 17GB | 192000 | 19GB |
### 224000 ### ollama_sizes m 224000
| Model Name | HDD (ls) | Context | RAM (ps) | | Model Name | HDD (ls) | Context | RAM (ps) |
| :------------------------ | -----------: | -------: | -----------: | | :------------------------ | -----------: | -------: | -----------: |
tasks/workstation/linux/software/llm.yml
+29 -4
View File
@@ -560,6 +560,9 @@
$image run --port {{ open_terminal_port }} $image run --port {{ open_terminal_port }}
# -p {{ open_terminal_port }}:8000 # -p {{ open_terminal_port }}:8000
# This requires internet access to be successful...
#-e OPEN_TERMINAL_PACKAGES="xz-utils libglu1-mesa libglu1-mesa ninja-build libnotify-bin libgtk-3-dev liblzma-dev libstdc++-12-dev libc6:amd64 libstdc++6 lib32z1 libbz2-1.0 default-jdk libncurses-dev android-udev-rules" \
mode: "0755" mode: "0755"
- name: Workstation | Linux | Software | LLMs | runit | open-terminal | Script | Finish - name: Workstation | Linux | Software | LLMs | runit | open-terminal | Script | Finish
@@ -906,16 +909,38 @@
alias model-sizes="ollama_sizes " alias model-sizes="ollama_sizes "
function ollama_threads() { function ollama_threads() {
# VAR1 - Model
model="$1" model="$1"
if [[ -z $model ]]; then if [[ -z $model ]]; then
model="qwen3.5:4b" model="qwen3.5:4b"
fi fi
# VAR2 - Context
typeset -i num_ctx="$2"
context=""
if [[ -n $num_ctx ]]; then
context=',"num_ctx":'$num_ctx''
fi
echo " Stopping all ollama usage..."
sudo sv restart ollama >/dev/null 2>&1
# Benchmark Thread Count # Benchmark Thread Count
for t in {1..{{ ansible_processor_vcpus | int }}}; do for t in {1..{{ ansible_processor_vcpus | int }}}; do
echo "`date` - Model $model w/ $t Threads`" start=$SECONDS
curl -s http://localhost:11434/api/generate \ echo -n "`date` - Model $model w/ $t Thread(s): "
-d '{"model":"$model","prompt":"Hello","stream":false,"options":{"num_thread":$t}}' \
| jq .eval_count / .eval_duration speed=$(curl -s http://localhost:11434/api/generate \
-d '{"model":"'$model'","prompt":"Hello","stream":false,"keep_alive":1,"options":{"num_thread":'$t$context'}}' \
| jq '.eval_count * 1000000000 / .eval_duration'
)
finish=$SECONDS
ctx="`ollama ps | grep $model | awk '{print $7}'`"
time=$(( $finish - $start ))
echo ", tokens/sec: $speed @ context=$ctx, time: $time secs"
ollama stop $model >/dev/null 2>&1
done done
} }
alias ollama-threads="ollama_threads " alias ollama-threads="ollama_threads "