Repository navigation
DeepSeek V4: native HIP tier and --gpu/--vram on it (RX 7600M XT, 8 GB) - #2603
Merged
Merged
Conversation
This was referenced Nov 6, 2026
kflirik
force-pushed
the
feat/v4-hip-rx7600m
branch
from
November 11, 2026 10:43
b8893c2 to
5c894cc
Compare
kflirik
force-pushed
the
feat/v4-hip-rx7600m
branch
from
November 12, 2026 16:35
3861191 to
092a88e
Compare
kflirik
force-pushed
the
feat/v4-hip-rx7600m
branch
from
November 12, 2026 17:14
092a88e to
8e3e616
Compare
backend_cuda_dsv4.cu builds with hipcc through backend_gpu_compat.h (name mappings) and links into the engine like CUDA=1: make deepseek-v4 HIP=1 HIP_ARCH=gfx1102. The cooperative fused-MoE kernel is CUDA-only and compiled out. It needs ROCm 6.3+ for the OCP fp8 type; the fp8 names live in backend_cuda_dsv4_quant.cuh so the ROCm 6.2 HIP syntax check of backend_cuda.cu does not see them. The native engine read none of the GPU variables coli exports for V4. On the HIP build it now reads the names the other engines take on HIP too (GPU_BACKENDS.md): COLI_CUDA=0 (--gpu none), COLI_GPUS (--gpu N) and CUDA_EXPERT_GB (--vram N). --vram is the VRAM the engine may use: the expert mirrors grow until the card has that much less free than at open. The hold is a floor for the mirror guard's reserve, so the engine's own headroom stays, and an explicit DSV4_CUDA_VRAM_RESERVE_MB is a floor too. Below 10 GiB the dense/attention mirrors stay in RAM and the VRAM goes to the mirrors; a long prompt is prefilled on the CPU. With the mirror live, rows16 repacking turns itself off. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
kflirik
force-pushed
the
feat/v4-hip-rx7600m
branch
from
November 13, 2026 08:49
8e3e616 to
b5ff0e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The native DeepSeek V4 engine had no HIP build, and
coli --gpu/--vramnever reached it (--vramwas a no-op for it). This runs it on an AMD GPU through HIP and makes the two flags work there, in 212 added and 12 removed lines in one commit. Developed on a Radeon RX 7600M XT (gfx1102, 8 GB, ROCm 7.16) withDeepSeek-V4-Flash-0731-reap-150b(79 GiB, 132 experts).The new revision no longer touches
backend_loader_dsv4.c; the overlap mentioned in #2752 and #2760 is gone.make deepseek-v4 HIP=1 HIP_ARCH=gfx1102.backend_cuda_dsv4.cucompiles with hipcc throughbackend_gpu_compat.h(name mappings) and links into the engine likeCUDA=1. The header is included only under HIP, so the CUDA build keeps the includes it has ondev.__shfl_*_syncwithout a width maps to a 32-lane group (notwarpSize), likebd_shflandcc_shfl_xor; run on RDNA only. The cooperative fused-MoE kernel (DSV4_CUDA_MOE_FUSED6) is CUDA-only and compiled out. It needs ROCm 6.3+ (the OCP fp8 type in<hip/hip_fp8.h>); the fp8 names live inbackend_cuda_dsv4_quant.cuh, so the ROCm 6.2 HIP syntax check ofbackend_cuda.cunever sees them.colirecognizes the HIP build bylibamdhip64.--gpu/--vram. On the HIP build the engine reads the names the other engines take on HIP too (GPU_BACKENDS.md: the runtime configuration is identical for both vendors):COLI_CUDA=0(--gpu none) turns the tier off,COLI_GPUS(--gpu N, the first device) picks the device, andCUDA_EXPERT_GB(--vram N) is the VRAM the engine may use. The expert mirrors grow until the card has that much less free than at open; this is a floor for the existing mirror-guard reserve (v4_gpu_mirror_reserve_mb), so the engine's own headroom stays, and an explicitDSV4_CUDA_VRAM_RESERVE_MBis a floor too.coli_env.hgainsCE_DSV4on two rows.Everything in the engine is under
-DCOLI_V4_HIP, which only theHIP=1Makefile section defines: a CUDA build keeps its placement.Validation
make check(check-env, a clean portable build, the C and Python tests), run withPYTHONNOUSERSITE=1on the previous revision of this branch merged intodevat b92eccd: 3372 tests, everything passes excepttest_serve_listen_backlog, which fails the same way on a cleandevhere (its kernel-backlog probe reads 0). This commit differs from that tree by three include/macro/doc edits and the rebase; on it I re-ranmake check-env, the HIP build and the model runs below, not the fullmake check.make -C c cuda-test— not run: there is no NVIDIA card or nvcc here. For a CUDA buildbackend_cuda_dsv4.cuandbackend_cuda_dsv4_quant.cuhdiffer fromdevonly by#ifguards (the includes,sticky(), the fused-MoE kernel) whose CUDA arm is the code that was there;backend_gpu_compat.hchanges only inside its HIP arm; the V4 Makefile's HIP section needsHIP=1. CI'scuda-dsv4-dlljob compiles the CUDA arm with nvcc.make -C c deepseek-v4 HIP=1(this commit),make -C c gpu-compile HIP=1 HIP_ARCH=gfx1100(before the rebase, with the same three edits; ROCm 7.16, not CI's 6.2 image) anddeepseek-v4-tiny-check(an earlier revision).test_env_defaults: a Linux build linkinglibamdhip64now counts as a GPU-capable V4 build.What
--vramdid on the card: "The capital of France is", 40 tokens allowed, greedy; the answer is**Paris**.at every budget. Peak card memory is read from sysfs and includes the desktop's share (about 0.6 to 0.95 GiB at the start):--vramSo the process stays within about 0.25 GiB of
--vram: the budget caps the mirrors, the engine's working buffers come on top. The rows were measured on earlier revisions of this branch with the same budget logic. Re-run on this commit (ondevwithDSV4_CUDA_HEAD_LAZY, desktop at 1.2 to 1.6 GiB,DSV4_CUDA_VRAM_RESERVE_MB=1024) at--vram 5: 5.3 GiB above start with the lazy head on and off, and the exact head stays on the card (v4_gpu head=resident) in both, also after a 2535-token prompt (prefilled on the CPU, 511 s to the first token; peak 5.3 to 5.8 GiB above start, the desktop's share moved by 0.4 GiB between those two runs).--vram 6was not re-run on this commit.--vram 7does not work on this card: with a free card the free VRAM fell to 0.5 GiB and the run was stopped by a guard. About 6 is the practical maximum with a desktop on the same GPU; the docs say so.Compatibility
Not covered: Windows and macOS are not built or run. Multi-GPU: the HIP placement applies only with one device; several devices keep the existing contiguous-layer placement. The card's VRAM is shared with every other client:
--vramcaps what this process's mirrors take, it does not reserve memory against the desktop or a game; leave the display room (for exampleDSV4_CUDA_VRAM_RESERVE_MB=1024; despite the name it applies to the HIP build too, like the other*_CUDA_*variables). On the HIP tier a later request in one process can differ from the first by a token flip, because which experts are mirrored decides whether an expert is computed on the GPU or the host, and the two round differently.