Skip to content

DeepSeek V4: native HIP tier and --gpu/--vram on it (RX 7600M XT, 8 GB) - #2603

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
kflirik:feat/v4-hip-rx7600m
Nov 14, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
kflirik:feat/v4-hip-rx7600m

Conversation

@kflirik

@kflirik kflirik commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

The native DeepSeek V4 engine had no HIP build, and coli --gpu / --vram never reached it (--vram was a no-op for it). This runs it on an AMD GPU through HIP and makes the two flags work there, in 212 added and 12 removed lines in one commit. Developed on a Radeon RX 7600M XT (gfx1102, 8 GB, ROCm 7.16) with DeepSeek-V4-Flash-0731-reap-150b (79 GiB, 132 experts).

The new revision no longer touches backend_loader_dsv4.c; the overlap mentioned in #2752 and #2760 is gone.

  • HIP build. make deepseek-v4 HIP=1 HIP_ARCH=gfx1102. backend_cuda_dsv4.cu compiles with hipcc through backend_gpu_compat.h (name mappings) and links into the engine like CUDA=1. The header is included only under HIP, so the CUDA build keeps the includes it has on dev. __shfl_*_sync without a width maps to a 32-lane group (not warpSize), like bd_shfl and cc_shfl_xor; run on RDNA only. The cooperative fused-MoE kernel (DSV4_CUDA_MOE_FUSED6) is CUDA-only and compiled out. It needs ROCm 6.3+ (the OCP fp8 type in <hip/hip_fp8.h>); the fp8 names live in backend_cuda_dsv4_quant.cuh, so the ROCm 6.2 HIP syntax check of backend_cuda.cu never sees them. coli recognizes the HIP build by libamdhip64.
  • --gpu / --vram. On the HIP build the engine reads the names the other engines take on HIP too (GPU_BACKENDS.md: the runtime configuration is identical for both vendors): COLI_CUDA=0 (--gpu none) turns the tier off, COLI_GPUS (--gpu N, the first device) picks the device, and CUDA_EXPERT_GB (--vram N) is the VRAM the engine may use. The expert mirrors grow until the card has that much less free than at open; this is a floor for the existing mirror-guard reserve (v4_gpu_mirror_reserve_mb), so the engine's own headroom stays, and an explicit DSV4_CUDA_VRAM_RESERVE_MB is a floor too. coli_env.h gains CE_DSV4 on two rows.
  • Dense in RAM below 10 GiB. On an 8 GB card the 6.2 GiB of dense/attention mirrors leave too little VRAM for expert mirrors, so they stay in RAM and the VRAM goes to the mirrors. The resident-experts preload then falls back with a message instead of failing. A long prompt is prefilled on the CPU.
  • With the mirror live, rows16 repacking turns itself off (the mirror refuses a packed view).

Everything in the engine is under -DCOLI_V4_HIP, which only the HIP=1 Makefile section defines: a CUDA build keeps its placement.

Validation

  • make check (check-env, a clean portable build, the C and Python tests), run with PYTHONNOUSERSITE=1 on the previous revision of this branch merged into dev at b92eccd: 3372 tests, everything passes except test_serve_listen_backlog, which fails the same way on a clean dev here (its kernel-backlog probe reads 0). This commit differs from that tree by three include/macro/doc edits and the rebase; on it I re-ran make check-env, the HIP build and the model runs below, not the full make check.
  • CUDA changes were tested with make -C c cuda-test — not run: there is no NVIDIA card or nvcc here. For a CUDA build backend_cuda_dsv4.cu and backend_cuda_dsv4_quant.cuh differ from dev only by #if guards (the includes, sticky(), the fused-MoE kernel) whose CUDA arm is the code that was there; backend_gpu_compat.h changes only inside its HIP arm; the V4 Makefile's HIP section needs HIP=1. CI's cuda-dsv4-dll job compiles the CUDA arm with nvcc.
  • Performance claims: none. This PR makes no throughput claim, so there is no experiment manifest.
  • Also run here: make -C c deepseek-v4 HIP=1 (this commit), make -C c gpu-compile HIP=1 HIP_ARCH=gfx1100 (before the rebase, with the same three edits; ROCm 7.16, not CI's 6.2 image) and deepseek-v4-tiny-check (an earlier revision).
  • test_env_defaults: a Linux build linking libamdhip64 now counts as a GPU-capable V4 build.

What --vram did on the card: "The capital of France is", 40 tokens allowed, greedy; the answer is **Paris**. at every budget. Peak card memory is read from sysfs and includes the desktop's share (about 0.6 to 0.95 GiB at the start):

--vram peak card use above start min. free VRAM
4 4.2 GiB 2.8 GiB
5 5.3 GiB 2.2 GiB
6 6.2 GiB 1.3 GiB

So the process stays within about 0.25 GiB of --vram: the budget caps the mirrors, the engine's working buffers come on top. The rows were measured on earlier revisions of this branch with the same budget logic. Re-run on this commit (on dev with DSV4_CUDA_HEAD_LAZY, desktop at 1.2 to 1.6 GiB, DSV4_CUDA_VRAM_RESERVE_MB=1024) at --vram 5: 5.3 GiB above start with the lazy head on and off, and the exact head stays on the card (v4_gpu head=resident) in both, also after a 2535-token prompt (prefilled on the CPU, 511 s to the first token; peak 5.3 to 5.8 GiB above start, the desktop's share moved by 0.4 GiB between those two runs). --vram 6 was not re-run on this commit. --vram 7 does not work on this card: with a free card the free VRAM fell to 0.5 GiB and the run was stopped by a guard. About 6 is the practical maximum with a desktop on the same GPU; the docs say so.

Compatibility

  • The default CPU build remains dependency-free
  • No model files, generated binaries, or benchmark artifacts are included

Not covered: Windows and macOS are not built or run. Multi-GPU: the HIP placement applies only with one device; several devices keep the existing contiguous-layer placement. The card's VRAM is shared with every other client: --vram caps what this process's mirrors take, it does not reserve memory against the desktop or a game; leave the display room (for example DSV4_CUDA_VRAM_RESERVE_MB=1024; despite the name it applies to the HIP build too, like the other *_CUDA_* variables). On the HIP tier a later request in one process can differ from the first by a token flip, because which experts are mirrored decides whether an expert is computed on the GPU or the host, and the two round differently.

@kflirik
kflirik force-pushed the feat/v4-hip-rx7600m branch from b8893c2 to 5c894cc Compare November 11, 2026 10:43
@kflirik
kflirik force-pushed the feat/v4-hip-rx7600m branch from 3861191 to 092a88e Compare November 12, 2026 16:35
@kflirik kflirik changed the title DeepSeek V4: native HIP tier and a single GPU placement plan (RX 7600M XT, 8 GB) DeepSeek V4: native HIP tier and --gpu/--vram on it (RX 7600M XT, 8 GB) Nov 12, 2026
@kflirik
kflirik force-pushed the feat/v4-hip-rx7600m branch from 092a88e to 8e3e616 Compare November 12, 2026 17:14
backend_cuda_dsv4.cu builds with hipcc through backend_gpu_compat.h (name
mappings) and links into the engine like CUDA=1: make deepseek-v4 HIP=1
HIP_ARCH=gfx1102. The cooperative fused-MoE kernel is CUDA-only and compiled
out. It needs ROCm 6.3+ for the OCP fp8 type; the fp8 names live in
backend_cuda_dsv4_quant.cuh so the ROCm 6.2 HIP syntax check of
backend_cuda.cu does not see them.

The native engine read none of the GPU variables coli exports for V4. On the
HIP build it now reads the names the other engines take on HIP too
(GPU_BACKENDS.md): COLI_CUDA=0 (--gpu none), COLI_GPUS (--gpu N) and
CUDA_EXPERT_GB (--vram N). --vram is the VRAM the engine may use: the expert
mirrors grow until the card has that much less free than at open. The hold is
a floor for the mirror guard's reserve, so the engine's own headroom stays,
and an explicit DSV4_CUDA_VRAM_RESERVE_MB is a floor too. Below 10 GiB the
dense/attention mirrors stay in RAM and the VRAM goes to the mirrors; a long
prompt is prefilled on the CPU. With the mirror live, rows16 repacking turns
itself off.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kflirik
kflirik force-pushed the feat/v4-hip-rx7600m branch from 8e3e616 to b5ff0e4 Compare November 13, 2026 08:49
@JustVugg
JustVugg merged commit 6e489f7 into JustVugg:dev Nov 14, 2026
4 of 117 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants