diff --git a/tasks/workstation/linux/software/llm.md b/tasks/workstation/linux/software/llm.md index 343b90d..13b4e4c 100644 --- a/tasks/workstation/linux/software/llm.md +++ b/tasks/workstation/linux/software/llm.md @@ -105,6 +105,64 @@ Number of physical cores is 8, hyperthreaded to 16, but supposedly the emulated cores are not very useful to LLMs. Try only using between 6 and 8 when possible. Research says it is not worth enabling the iGPU for Intel XE, CPUs are better. +## Benchmark + +### System + +``` +$ fastfetch + ' ling@good-rhino + 'o' --------------- + 'ooo' OS: Artix Linux x86_64 + 'ooxoo' Host: NV4xPZ (v2.1) + 'ooxxxoo' Kernel: Linux 7.2.11-artix1-1 + 'oookkxxoo' Uptime: 10 hours, 3 mins + 'oiioxkkxxoo' Packages: 42 (flatpak), 1502 (pacman) + ':;:iiiioxxxoo' Shell: bash 5.3.20 + `'.;::ioxxoo' Display (AUO683D): 1920x1080 in 14", 60 Hz [Built-in] + '-. `':;jiooo' Window Manager: i3 4.25.1 (X11) + 'oooio-.. `'i:io' Theme: Adwaita-dark [GTK2/3/4] + 'ooooxxxxoio:,. `'-;' Icons: Adwaita [GTK2/3/4] + 'ooooxxxxxkkxoooIi:-. `' Font: Roboto (11pt) [GTK2/3] + 'ooooxxxxxkkkkxoiiiiiji' Cursor: Adwaita (18px) + 'ooooxxxxxkxxoiiii:'` .i' Terminal: kitty 0.48.2 + 'ooooxxxxxoi:::'` .;ioxo' Terminal Font: DejaVuSansMono (11pt) + 'ooooxooi::'` .:iiixkxxo' CPU: 12th Gen Intel(R) Core(TM) i7-1260P (8+8) @ 4.70 GHz + 'ooooi:'` `'';ioxxo' GPU: Intel Iris Xe Graphics @ 1.40 GHz [Integrated] + 'i:'` '':io' Memory: 28.11 GiB / 62.63 GiB (45%) +'` `' Swap: 1.12 GiB / 68.89 GiB (2%) + Disk (/): 552.03 GiB / 847.71 GiB (65%) - ext4 + Local IP (wg0-mullvad): 10.131.3.55/32 + Battery (BAT): 95% [AC Connected] + Locale: en_US.UTF-8 +``` + +### Main Models and Parameters + +``` +$ export QUIET=true +$ ollama-test "64000 128000" "4 7 0" 0 "qwen3.5:27b qwen3.5:35b qwen3.6:27b qwen3.6:35b qwen3.8:27b devstral-small-2:24b" +| Model | HDD | Context | RAM | Threads | Processor | Speed (t/s) | Time (s) | +| :------------------------ | -----: | -------: | -----: | ------: | --------: | ----------: | -------: | +| qwen3.5:27b | 17GB | 64000 | 22GB | 4 | 100%CPU | 2.12 | 256 | +| qwen3.5:35b | 22GB | 64000 | 23GB | 4 | 100%CPU | 13.26 | 42 | +| qwen3.6:27b | 17GB | 64000 | 18GB | 4 | 100%CPU | 2.97 | 102 | +| qwen3.6:35b | 22GB | 64000 | 22GB | 4 | 100%CPU | 13.91 | 26 | +| qwen3.8:27b | 17GB | 64000 | 19GB | 4 | 100%CPU | 1.76 | 49 | +| devstral-small-2:24b | 15GB | 64000 | 26GB | 4 | 100%CPU | 2.76 | 93 | +| qwen3.5:27b | 17GB | 64000 | 22GB | 7 | 100%CPU | 2.08 | 167 | +| qwen3.5:35b | 22GB | 64000 | 23GB | 7 | 100%CPU | 13.61 | 20 | +| qwen3.6:27b | 17GB | 64000 | 18GB | 7 | 100%CPU | 3.08 | 56 | +| qwen3.6:35b | 22GB | 64000 | 22GB | 7 | 100%CPU | 11.13 | 36 | +| qwen3.8:27b | 17GB | 64000 | 19GB | 7 | 100%CPU | 2.23 | 43 | +| devstral-small-2:24b | 15GB | 64000 | 26GB | 7 | 100%CPU | 2.36 | 88 | +| qwen3.5:27b | 17GB | 64000 | 22GB | 0 | 100%CPU | 2.11 | 156 | +| qwen3.5:35b | 22GB | 64000 | 23GB | 0 | 100%CPU | 13.68 | 37 | +| qwen3.6:27b | 17GB | 64000 | 18GB | 0 | 100%CPU | 2.68 | 85 | +| qwen3.6:35b | 22GB | 64000 | 22GB | 0 | 100%CPU | 13.83 | 39 | +| qwen3.8:27b | 17GB | . . . . +``` + ## Tokens per CPU Count Verifying that 8 or less cores gives better performance (less memory issues).