Skip to content

Commit eaa3192

Browse files
mixa3607mixa3607
committed
llamacpp: 7.2.3
1 parent f234276 commit eaa3192

6 files changed

Lines changed: 15 additions & 89 deletions

File tree

‎.github/workflows/llamacpp-daily-build.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ jobs:
1010
runs-on: ubuntu-latest
1111
strategy:
1212
matrix:
13-
rocm_version: [ "6.3.3", "7.2.1" ]
13+
rocm_version: [ "6.3.3", "7.2.3" ]
1414
steps:
1515
- name: Checkout repo
1616
uses: actions/checkout@v4

‎llama.cpp/preset.b9180-rocm-7.2.1.sh‎

Lines changed: 0 additions & 6 deletions
This file was deleted.

‎llama.cpp/preset.b9180-rocm-7.2.2.sh‎

Lines changed: 0 additions & 6 deletions
This file was deleted.

‎llama.cpp/readme.md‎

Lines changed: 11 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,87 +1,25 @@
11
# llama.cpp GFX906
2+
23
LLM inference in C/C++ https://github.com/ggml-org/llama.cpp
34

4-
Recommend use `docker.io/mixa3607/llama.cpp-gfx906:7.0.0-complete`
5+
Recommend use `docker.io/mixa3607/llama.cpp-gfx906:7.2.3-complete`
56

67
Also see [llamacpp-offload-calculator](./llamacpp-offload-calculator/readme.md)
78

8-
## Benchmarks
9-
```shell
10-
export PATH="/app:$PATH"
11-
export LD_LIBRARY_PATH="/app:$LD_LIBRARY_PATH"
12-
13-
MODEL=/root/.cache/huggingface/hub/models--ggml-org--gemma-3n-E4B-it-GGUF/snapshots/ee0f0cb58a4b9d5b48dd55b576db22eeeeecdd7e/gemma-3n-E4B-it-Q8_0.gguf
14-
MODEL=/root/.cache/huggingface/hub/models--unsloth--gemma-3-12b-it-GGUF/snapshots/a5592d885c8a933e824f80d2eeda84db95ad2712/gemma-3-12b-it-Q8_0.gguf
15-
MODEL=/root/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3-14B-GGUF/snapshots/bd080f768a6401c2d5a7fa53a2e50cd8218a9ce2/Qwen_Qwen3-14B-Q4_K_S.gguf
16-
MODEL=/root/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3-14B-GGUF/snapshots/bd080f768a6401c2d5a7fa53a2e50cd8218a9ce2/Qwen_Qwen3-14B-Q4_0.gguf
17-
MODEL=/root/.cache/huggingface/hub/models--bartowski--Qwen_Qwen3-14B-GGUF/snapshots/bd080f768a6401c2d5a7fa53a2e50cd8218a9ce2/Qwen_Qwen3-14B-bf16.gguf
18-
MODEL=/root/.cache/huggingface/hub/models--ggml-org--gemma-3-27b-it-GGUF/snapshots/f94c25afed0072339c5fa3b705a7b4222afe5f62/gemma-3-27b-it-f16-00001-of-00002.gguf
19-
20-
llama-bench --model $MODEL -t 16 --flash-attn 0
21-
```
22-
23-
```
24-
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
25-
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
26-
ggml_cuda_init: found 2 ROCm devices:
27-
Device 0: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
28-
Device 1: AMD Radeon Graphics, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
29-
load_backend: loaded ROCm backend from /app/libggml-hip.so
30-
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
31-
```
32-
33-
| rocm | llama.cpp | model | size | params | backend | ngl | test | t/s |
34-
| ----- | --------- | ------------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
35-
| 6.3.4 | 982e3472 | gemma3n E4B Q8_0 | 6.84 GiB | 6.87 B | ROCm | 99 | pp512 | 483.29 ± 0.68 |
36-
| 6.3.4 | 982e3472 | gemma3n E4B Q8_0 | 6.84 GiB | 6.87 B | ROCm | 99 | tg128 | 33.48 ± 0.43 |
37-
| 6.3.4 | 982e3472 | gemma3 12B Q8_0 | 11.64 GiB | 11.77 B | ROCm | 99 | pp512 | 246.66 ± 0.07 |
38-
| 6.3.4 | 982e3472 | gemma3 12B Q8_0 | 11.64 GiB | 11.77 B | ROCm | 99 | tg128 | 28.41 ± 0.12 |
39-
| 6.3.4 | 982e3472 | qwen3 14B Q4_K - Small | 7.98 GiB | 14.77 B | ROCm | 99 | pp512 | 242.34 ± 0.15 |
40-
| 6.3.4 | 982e3472 | qwen3 14B Q4_K - Small | 7.98 GiB | 14.77 B | ROCm | 99 | tg128 | 35.87 ± 0.15 |
41-
| 6.3.4 | 982e3472 | qwen3 14B Q4_0 | 7.95 GiB | 14.77 B | ROCm | 99 | pp512 | 574.13 ± 0.28 |
42-
| 6.3.4 | 982e3472 | qwen3 14B Q4_0 | 7.95 GiB | 14.77 B | ROCm | 99 | tg128 | 39.02 ± 0.23 |
43-
| 6.3.4 | 982e3472 | qwen3 14B BF16 | 27.51 GiB | 14.77 B | ROCm | 99 | pp512 | 118.01 ± 0.24 |
44-
| 6.3.4 | 982e3472 | qwen3 14B BF16 | 27.51 GiB | 14.77 B | ROCm | 99 | tg128 | 19.33 ± 0.08 |
45-
| 6.3.4 | 982e3472 | gemma3 27B F16 | 50.31 GiB | 27.01 B | ROCm | 99 | pp512 | 236.51 ± 0.14 |
46-
| 6.3.4 | 982e3472 | gemma3 27B F16 | 50.31 GiB | 27.01 B | ROCm | 99 | tg128 | 10.37 ± 0.04 |
47-
| 6.3.4 | 982e3472 | llama4 17Bx16E (Scout) Q3_K - Medium | 48.19 GiB | 107.77 B | ROCm | 99 | pp512 | 160.50 ± 0.81 |
48-
| 6.3.4 | 982e3472 | llama4 17Bx16E (Scout) Q3_K - Medium | 48.19 GiB | 107.77 B | ROCm | 99 | tg128 | 22.75 ± 0.07 |
49-
| 6.4.1 | 982e3472 | gemma3n E4B Q8_0 | 6.84 GiB | 6.87 B | ROCm | 99 | pp512 | 606.83 ± 0.97 |
50-
| 6.4.1 | 982e3472 | gemma3n E4B Q8_0 | 6.84 GiB | 6.87 B | ROCm | 99 | tg128 | 33.36 ± 0.23 |
51-
| 6.4.1 | 982e3472 | gemma3 12B Q8_0 | 11.64 GiB | 11.77 B | ROCm | 99 | pp512 | 329.70 ± 0.30 |
52-
| 6.4.1 | 982e3472 | gemma3 12B Q8_0 | 11.64 GiB | 11.77 B | ROCm | 99 | tg128 | 28.58 ± 0.15 |
53-
| 6.4.1 | 982e3472 | qwen3 14B Q4_K - Small | 7.98 GiB | 14.77 B | ROCm | 99 | pp512 | 286.58 ± 0.15 |
54-
| 6.4.1 | 982e3472 | qwen3 14B Q4_K - Small | 7.98 GiB | 14.77 B | ROCm | 99 | tg128 | 36.48 ± 0.11 |
55-
| 6.4.1 | 982e3472 | qwen3 14B Q4_0 | 7.95 GiB | 14.77 B | ROCm | 99 | pp512 | 570.15 ± 0.23 |
56-
| 6.4.1 | 982e3472 | qwen3 14B Q4_0 | 7.95 GiB | 14.77 B | ROCm | 99 | tg128 | 38.94 ± 0.16 |
57-
| 6.4.1 | 982e3472 | qwen3 14B BF16 | 27.51 GiB | 14.77 B | ROCm | 99 | pp512 | 119.03 ± 0.31 |
58-
| 6.4.1 | 982e3472 | qwen3 14B BF16 | 27.51 GiB | 14.77 B | ROCm | 99 | tg128 | 19.46 ± 0.10 |
59-
| 6.4.1 | 982e3472 | gemma3 27B F16 | 50.31 GiB | 27.01 B | ROCm | 99 | pp512 | 238.38 ± 0.26 |
60-
| 6.4.1 | 982e3472 | gemma3 27B F16 | 50.31 GiB | 27.01 B | ROCm | 99 | tg128 | 10.41 ± 0.03 |
61-
| 6.4.1 | 982e3472 | llama4 17Bx16E (Scout) Q3_K - Medium | 48.19 GiB | 107.77 B | ROCm | 99 | pp512 | 190.52 ± 0.84 |
62-
| 6.4.1 | 982e3472 | llama4 17Bx16E (Scout) Q3_K - Medium | 48.19 GiB | 107.77 B | ROCm | 99 | tg128 | 22.96 ± 0.10 |
9+
## Docs
6310

11+
http://arkprojects.space/wiki/AMD_GFX906/llamacpp
6412

65-
## Run
66-
### Docker
67-
See https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md + https://github.com/ROCm/vllm/blob/main/docs/deployment/docker.md
13+
## Build
6814

69-
### Kubernetes
70-
Helm chart and samples [mixa3607 charts](https://github.com/mixa3607/charts)
15+
See build vars in `./env.sh`
7116

72-
## Build
73-
See build vars in `./env.sh`. You also may use presetis `./preset.rocm-*.sh`. Exec `./build-and-push.rocm.sh`:
7417
```bash
75-
$ . preset.rocm-7.0.0.sh
76-
$ ./build-and-push.rocm.sh
77-
~/REPOS/mixa3607/llama.cpp-gfx906/rocm ~/REPOS/mixa3607/llama.cpp-gfx906/rocm
78-
~/REPOS/mixa3607/llama.cpp-gfx906/rocm
79-
~/REPOS/mixa3607/llama.cpp-gfx906/llama.cpp ~/REPOS/mixa3607/llama.cpp-gfx906/rocm
80-
~/REPOS/mixa3607/llama.cpp-gfx906/rocm
81-
~/REPOS/mixa3607/llama.cpp-gfx906/comfyui ~/REPOS/mixa3607/llama.cpp-gfx906/rocm
82-
~/REPOS/mixa3607/llama.cpp-gfx906/rocm
83-
~/REPOS/mixa3607/llama.cpp-gfx906/vllm ~/REPOS/mixa3607/llama.cpp-gfx906/rocm
84-
~/REPOS/mixa3607/llama.cpp-gfx906/rocm
18+
$ . preset.b9180-rocm-7.2.3.sh && ./build-and-push.rocm.sh
19+
~/REPOS/mixa3607/ML-gfx906/llama.cpp ~/REPOS/mixa3607/ML-gfx906/llama.cpp
20+
~/REPOS/mixa3607/ML-gfx906/llama.cpp
21+
~/REPOS/mixa3607/ML-gfx906/rocm ~/REPOS/mixa3607/ML-gfx906/llama.cpp
22+
~/REPOS/mixa3607/ML-gfx906/llama.cpp
8523
#0 building with "remote" instance using remote driver
8624
#...............
8725
#14 DONE 583.8s

‎llama.cpp/upd2last-release.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,12 @@ export LLAMA_PRESET_NAME=\"\${LLAMA_BRANCH}-rocm-\${LLAMA_ROCM_VERSION}\"
1313
" > "$PRESET"
1414
fi
1515

16-
PRESET=preset.$RELEASE_TAG-rocm-7.2.1.sh
16+
PRESET=preset.$RELEASE_TAG-rocm-7.2.3.sh
1717
if ! [ -f "$PRESET" ]; then
1818
echo "Creating preset $PRESET"
1919
echo "#!/bin/bash
2020
21-
export LLAMA_ROCM_VERSION='7.2.1'
21+
export LLAMA_ROCM_VERSION='7.2.3'
2222
export LLAMA_BRANCH='$RELEASE_TAG'
2323
export LLAMA_PRESET_NAME=\"\${LLAMA_BRANCH}-rocm-\${LLAMA_ROCM_VERSION}\"
2424
" > "$PRESET"

‎readme.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ https://arkprojects.space/wiki/AMD_GFX906
5050
| | [`docker.io/mixa3607/comfyui-gfx906:<ver>-rocm-7.2.1`](https://hub.docker.com/r/mixa3607/comfyui-gfx906/tags?name=rocm-7.2.1)\* |
5151
| --------- | ----------------------------------------------------------- |
5252
| llama.cpp | [`docker.io/mixa3607/llama.cpp-gfx906:<ver>-rocm-6.3.3`](https://hub.docker.com/r/mixa3607/llama.cpp-gfx906/tags?name=rocm-6.3.3)\* |
53-
| | [`docker.io/mixa3607/llama.cpp-gfx906:<ver>-rocm-7.2.1`](https://hub.docker.com/r/mixa3607/llama.cpp-gfx906/tags?name=rocm-7.2.1)\* |
53+
| | [`docker.io/mixa3607/llama.cpp-gfx906:<ver>-rocm-7.2.3`](https://hub.docker.com/r/mixa3607/llama.cpp-gfx906/tags?name=rocm-7.2.3)\* |
5454

5555
> \* llama.cpp and ComfyUI have daily builds. See last tag on dockerhub
5656

0 commit comments

Comments
 (0)