Skip to content

windows: DeepSeek V4 built with clang and LLVM's libomp, whose barriers spin (#2286) - #2800

Merged
JustVugg merged 3 commits into
JustVugg:devfrom
KairaCode:kaira/issue-2286-windows-libomp
Nov 14, 2026
Merged

JustVugg merged 3 commits into
JustVugg:devfrom
KairaCode:kaira/issue-2286-windows-libomp

Conversation

@KairaCode

Copy link
Copy Markdown

Base: dev @ b92eccd · Depends on: none

Summary

#2286 measured the shipped Windows deepseek_v4.exe (MSYS2 gcc, libgomp) decoding DeepSeek V4 Flash 1.5x slower
than a clang + libomp build of the same source, with the same output. The reporter's diagnosis holds: MinGW's
libgomp never spins at a barrier, so the OMP_WAIT_POLICY=active and GOMP_SPINCOUNT=200000 that the launcher
sets for this engine do nothing on Windows. This PR makes the clang build a supported, tested build
(make deepseek-v4 CC=clang), builds the release's Windows deepseek_v4.exe that way (libomp linked in
statically, no new DLL), and on Windows adds KMP_BLOCKTIME=200 to the launcher's defaults when the wait policy
is active, so the libomp team sleeps when the engine is idle instead of spinning forever.

Root cause

libgomp on MinGW is configured with config_path="mingw32 posix". Its barriers are config/posix/bar.c: they wait
on a semaphore with no spin loop. Only the Linux port (config/linux/wait.h) spins GOMP_SPINCOUNT times first.
A DeepSeek V4 decode step runs hundreds of small parallel regions, and with libgomp each one ends in a semaphore
wake-up of every worker.

Microbenchmark (a scratch C program, not part of the PR): #pragma omp barrier in a loop inside one parallel
region, and a parallel for over a 1024 x 512 f32 matrix-vector product (the shape of a decode region), 20000
iterations each. i7-7700HQ (4 cores, 8 threads), Windows 11, MSYS2 UCRT64 gcc 16.2.0 and clang 23.1.5 with LLVM
OpenMP 23.1.5, both -O2 -march=x86-64-v3 -fopenmp -static, OMP_DYNAMIC=FALSE, 3 interleaved rounds (each
value below is the range of the 3):

runtime, settings barrier, 4 threads barrier, 8 threads matvec region, 4 / 8 threads CPU while idle 1 s after the loop (4 threads)
gcc / libgomp, no settings 21.9-22.4 us 38.3-38.6 us 170-172 / 118-119 us 0.00 s
gcc / libgomp, OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 22.5-23.5 us 38.0-38.6 us 170-173 / 117-120 us 0.00 s
gcc / libgomp, GOMP_SPINCOUNT=infinite 21.6-22.5 us 38.4-40.6 us 169-173 / 117-133 us 0.00 s
clang / libomp, no settings 0.28-0.30 us 0.55-2.58 us 150-155 / 83-92 us 0.61 s
clang / libomp, OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 (the launcher today) 0.25-0.30 us 0.46-0.51 us 150-156 / 82-92 us 3.02-3.05 s
clang / libomp, the same plus KMP_BLOCKTIME=200 (this PR) 0.28-0.30 us 0.50-0.55 us 151-154 / 85-87 us 0.61-0.62 s
clang / libomp, OMP_WAIT_POLICY=passive 13.5-13.9 us 21.4-22.1 us 163-164 / 107-118 us 0.00 s

The libgomp rows do not move with any setting, and libgomp never burns CPU while idle: it never spins. libomp's
barrier is 70x cheaper. The last column is why KMP_BLOCKTIME is part of this PR: with OMP_WAIT_POLICY=active
and no KMP_BLOCKTIME, libomp's blocktime is infinite and the three idle workers spin at 100% (#341). With 200 ms
they spin for 200 ms after the last region (0.6 s of the 1 s) and then sleep.

A second round with the same program (clang / libomp, 4 threads, OMP_DYNAMIC=FALSE, 3 interleaved rounds,
ranges of the 3) checked the passive case, which this PR leaves without KMP_BLOCKTIME:

clang / libomp, settings barrier matvec region CPU while idle 1 s after the loop
OMP_WAIT_POLICY=passive 15.2-16.6 us 170-186 us 0.00 s
OMP_WAIT_POLICY=passive KMP_BLOCKTIME=200 0.27-0.35 us 161-167 us 0.59-0.61 s
OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 KMP_BLOCKTIME=200 (this PR, Windows) 0.29-0.31 us 160-166 us 0.56-0.59 s
OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 (the launcher today) 0.25-0.30 us 160-164 us 2.94-3.00 s

With KMP_BLOCKTIME=200, passive behaves like active: the team spins for 200 ms after each region and then
sleeps. That is why the launcher adds it only when the policy is active.

A pure-gcc fix was considered and not taken. libgomp's barrier cannot be made to spin from outside, and LLVM's
libomp.a for MinGW has no GNU ABI entry points (no GOMP_* symbols), so gcc-compiled code cannot be linked
against libomp either. What is left is replacing the engine's ~46 #pragma omp regions with a hand-written thread
pool and barrier, which is a much larger change than building with the compiler whose runtime already spins.

What changes

  • c/Makefile
    • clang names the MinGW target x86_64-w64-windows-gnu (MSYS2's clang and llvm-mingw), gcc
      x86_64-w64-mingw32. Only the second was recognised, so with CC=clang the Makefile treated Windows as
      an unknown platform: make deepseek-v4 CC=clang answered "supported only on x86-64/aarch64 Linux and
      Windows/MSYS2". MINGW now matches both.
    • The static OpenMP archive check ([Bug]: Static OpenMP library missing from the UCRT64 toolchain #2612) looks for libomp.a and names mingw-w64-ucrt-x86_64-llvm-openmp
      and docs/deepseek-v4.md when the compiler is clang, instead of warning that libgomp.a is missing. gcc's
      message is unchanged.
  • c/Makefile.deepseek-v4, Windows block, only when CC is clang:
    • LDFLAGS = -lm -fopenmp -static: libomp is linked in from MSYS2's libomp.a, so the engine still imports
      only system DLLs. No -pthread: clang's MinGW driver adds -lpthread itself, and the flag only drew
      "argument unused during compilation".
    • LTO ?= 0: with GNU ld, clang's LTO objects need LLVM's linker plugin (LLVMgold.dll), which MSYS2's clang
      package does not install; the default build stopped at the link with "error loading plugin". LTO=1 still
      works for anyone who has the plugin or lld.
    • The gcc line is unchanged (pinned by the new test).
  • c/deepseek_v4.c: v4_moe_bank_hash_layer was written in two places and never read, and clang 23 warns about
    it (-Wunused-but-set-global). Removed with its #define/#undef. With this, make deepseek-v4 CC=clang has 0 warnings
    (the release's VK=1 adds 3 in backend_vulkan.c, which this PR does not touch; see Correctness).
  • c/coli (env_for_engine) and c/openai_server.py (tune_child_env): KMP_BLOCKTIME=200 beside
    GOMP_SPINCOUNT=200000 for deepseek_v4, setdefault like its neighbours, inside the same COLI_NO_OMP_TUNE
    guard, on Windows only and only when the resulting OMP_WAIT_POLICY is active (any case). libgomp ignores
    it; it is the value omp_tune.h and colibri.c already set for libomp builds on Linux and FreeBSD.
    • Not under passive: libomp gives passive its blocktime of 0 only while KMP_BLOCKTIME is unset, so a
      launcher default would turn a user's OMP_WAIT_POLICY=passive into 200 ms of spin-yield after every region
      (measured above: 0.6 s of CPU in the idle second instead of 0).
    • Not on macOS: its V4 links Homebrew's libomp too, and there the change from an infinite to a 200 ms
      blocktime has not been measured ([Bug]: OpenMP spin-wait tuning is inert on macOS — and applying it costs 2.2x decode on a 32 GB host #707 measured both knobs slower on Apple Silicon). Linux builds V4 with gcc
      and libgomp; a Linux clang build keeps what it had.
    • c/coli's env_for (the GLM path), which sets the same active policy on Windows, gets the same line: with
      this PR make colibri CC=clang builds with the Windows flags, and such a build would otherwise spin all its
      workers forever while idle.
  • .github/workflows/release.yml (Windows row):
    • installs mingw-w64-ucrt-x86_64-clang, and LLVM's OpenMP package in a step of its own: MSYS2 renamed it
      from openmp to llvm-openmp, and the runner's package list (update: false) may predate the rename. A name
      pacman does not know makes the whole install fail (6b09123), so the step asks pacman -Si which name the
      list has.
    • builds make deepseek-v4 XDNA=1 VK=1 CC=clang (new matrix value v4_args); the other engines stay gcc.
    • the archive check fails if the packaged deepseek_v4.exe is not a libomp build (no KMP_BLOCKTIME string in
      it) or imports any lib*.dll the archive does not carry.
  • .github/workflows/ci.yml (engines-all-platforms, Windows row): the loop still builds the default gcc
    deepseek-v4 (the build make deepseek-v4 and source installs produce), and after it the job builds the
    release's make deepseek-v4 XDNA=1 CC=clang VK=1 (new matrix value v4_args; the flags stamp rebuilds the
    units). It installs what that needs: clang, the OpenMP package (same step as release.yml),
    vulkan-headers and shaderc. So every PR compiles backend_vulkan.c, vk_tier.c and vk_chain.c with
    clang and links V4 the release's way (clang, VK=1, no LTO); before, no job did.
  • Tests:
    • c/tests/test_makefile_platform.py: test_clang_mingw_triple_builds_for_windows (the windows-gnu triple
      gives colibri.exe and the -static -lpsapi link) and test_deepseek_v4_windows_link_per_compiler (dry run
      of Makefile.deepseek-v4 with OS=Windows_NT: clang's link line ends -lm -fopenmp -static, gcc's still
      ends -static -pthread -flto). Both run under make check (test-python).
    • c/tests/test_cli_output.py: test_v4_libomp_team_sleeps_when_idle_on_windows,
      test_v4_blocktime_only_for_an_active_team_on_windows (an explicit Passive, macOS and Linux get no
      KMP_BLOCKTIME); KMP_BLOCKTIME added to the kill-switch test.
    • c/tests/test_openai_server.py: test_direct_v4_server_libomp_team_sleeps_when_idle (Windows default, an
      explicit value wins, PASSIVE and macOS get none); KMP_BLOCKTIME added to the kill-switch test.
    • c/tests/test_env_defaults.py (the GLM path): the Windows default includes KMP_BLOCKTIME=200,
      test_passive_keeps_libomp_blocktime_zero, and the kill-switch and non-Windows tests also check
      KMP_BLOCKTIME.
  • docs/deepseek-v4.md (Build: the clang build, why, the two packages, the numbers), docs/tuning.md (MinGW's
    libgomp ignores the wait policy and the spin count), CHANGELOG.md (Unreleased / Performance).

New env flags: none. Defaults: the gcc build is unchanged; on Windows, when the wait policy is active, the
launcher adds KMP_BLOCKTIME=200 (deepseek_v4 in coli and openai_server.py, and coli's GLM path),
which libgomp ignores. Linux and macOS: unchanged.

Correctness

All on the same i7-7700HQ laptop, Windows 11, MSYS2 UCRT64 (gcc 16.2.0, clang 23.1.5, LLVM OpenMP 23.1.5),
Python 3.14, with the generated tiny V4 fixture (tools/make_deepseek_v4_tiny.py, generator version 2, the
ref.json committed on dev).

  • tests/test_deepseek_v4_tiny.py, tests/test_deepseek_v4_prefix.py and tests/test_deepseek_v4_brio.py
    (--binary <exe> --fixture <copy>) pass with both the gcc build and the clang build of this branch: 12, 7 and 5
    PASS lines, "all checks completed". That covers teacher forcing and greedy against the oracle, sessions, the
    served engine, KV prefix reuse and the numeric channel.
  • The benchmark below produced the same token hash in all 42 runs of each workload (and in the 56 of an earlier
    round of it), for every build and every OpenMP setting.
  • tests/test_deepseek_v4 (the C unit test) passes with the gcc build. With clang it does not link, and that is
    not new: on Windows this test only links with LTO, which drops the GPU unit's references to units the test
    does not link. gcc with LTO=0 fails the same way (25 undefined references to coli_v4_layer_gpu and
    friends).
  • New tests: tests.test_makefile_platform 7 tests OK; on dev the 2 new ones fail (the clang triple is not
    Windows; clang's link line has -pthread -flto). tests.test_cli_output -k v4 -k kill_switch -k explicit_setting 6 OK, tests.test_openai_server -k direct_v4 4 OK; on dev the new test in each fails
    (no KMP_BLOCKTIME). tests.test_env_defaults OK.
  • Builds: make deepseek-v4 CC=clang 0 warnings (29 s); make deepseek-v4 (gcc) prints the same 11
    -Wcast-function-type warnings in backend_loader_dsv4.c and the lto-wrapper note as dev (gcc 16.2). The
    clang deepseek_v4.exe imports KERNEL32.dll and the api-ms-win-crt-* sets only, like the gcc one.
  • Second round (this version, e3f9ba5), same laptop, a fresh git archive of the branch:
    • tests.test_env_defaults 32 tests OK, tests.test_makefile_platform 7 OK, tests.test_family_registry 63
      OK, tests.test_cli_output tests.test_openai_server 523 OK (1 skipped). The 9 tests selected by -k blocktime -k passive -k libomp -k kill_switch -k non_windows -k measured_defaults (the launcher's Windows default, an
      explicit Passive, macOS and Linux, the kill switches) all pass.
    • The Makefile warning with stand-in compilers that have no OpenMP archive: fake-clang gets "fake-clang has
      no libomp.a, so the -static -fopenmp link will fail with "cannot find -lomp". ... pacman -S --needed
      mingw-w64-ucrt-x86_64-llvm-openmp (docs/deepseek-v4.md)", fake-gcc still gets libgomp.a,
      mingw-w64-ucrt-x86_64-libgomp and docs/windows.md. None with the real clang or gcc.
    • The release configuration, make deepseek-v4 XDNA=1 VK=1 CC=clang (LTO off; Vulkan-Headers 1.4.368 through
      CPATH; GLSLC=true, so the committed .spv files are used), from a clean tree: builds and links in 39 s.
      3 warnings, all in backend_vulkan.c, which this PR does not touch: -Wc23-extensions (a label followed by
      a declaration, line 5724) and -Wunused-but-set-global for g_vk_gemm_calls and g_vk_coop_calls. The
      binary imports KERNEL32.dll and the api-ms-win-crt-* sets only, and passes both checks release.yml adds
      (a KMP_BLOCKTIME string, no lib*.dll import). test_deepseek_v4_tiny, _prefix and _brio pass with it
      on the CPU path: 12, 7 and 5 PASS lines, "all checks completed".
    • ci.yml's order in one tree: make deepseek-v4 XDNA=1 (gcc, 87 s, 12 warning: lines: the 11 and the note
      above), then make deepseek-v4 XDNA=1 CC=clang VK=1: "build flags changed, rebuilding the units", 32 objects
      recompiled by clang, linked in 34 s, the same 3 warnings, and that binary passes test_deepseek_v4_tiny (12
      PASS lines).
  • make -C c check (gcc, the default build), same laptop: portable and test-c pass. test-python ran 3473
    tests with 11 errors, and the same 11 fail on dev on this machine: 6 read a UTF-8 file with Python's Windows
    default encoding (UnicodeDecodeError: 'charmap' codec: test_v4_shared_expert_bindings reading
    deepseek_v4.c, test_glm53_cap_launch_source, test_glm53_metal_clamped_source, test_openai_tools_v41_e2e
    x2, test_qwenimage_gguf_import), and 5 in test_qwen38_int2_convert/test_qwen38_int4_convert cannot delete a
    temporary file Windows still holds open (WinError 32). The new and changed tests are not among them.

Performance

The tiny fixture, i7-7700HQ, engine team of 4 (the engine's own sizing: 4 physical cores of 8), every run from a
fresh copy of the fixture (so .coli_usage is the committed one), 7 interleaved rounds per configuration. Decode
tok/s = (generated - 1) / after_first; the engine prints after_first in ms, so the clang values are
quantized (99/0.064 = 1547, 99/0.063 = 1571). Median [min-max]:

build, OpenMP settings 8-token prompt, 100 tokens: decode tok/s 72-token prompt, 40 tokens: decode tok/s 72-token prompt: time to first token
dev, gcc (LTO), the launcher's settings today 500 [448-521] 507 [419-520] 67 ms [61-70]
this branch, gcc (LTO), the launcher's new settings 508 [440-521] 507 [429-520] 62 ms [59-80]
this branch, gcc LTO=0 488 [421-513] 507 [459-513] 63 ms [62-68]
this branch, clang + libomp, the launcher's new settings 1547 [1523-1597] 1560 [1393-1560] 20 ms [19-23]
this branch, clang + libomp, without KMP_BLOCKTIME 1571 [1500-1650] 1500 [1500-1560] 20 ms [18-25]
this branch, clang + libomp, OMP_WAIT_POLICY=passive 853 [773-868] 848 [722-867] 39 ms [37-43]

"The launcher's settings" are OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 OMP_DYNAMIC=FALSE, plus
KMP_BLOCKTIME=200 for "new". The tiny model's matrices are small, so OpenMP's per-region cost is most of a token
here and the 3.1x overstates what a real checkpoint gains. The issue measured 1.5x on DeepSeek V4 Flash (Ryzen AI
Max+ 392, 12 threads, llvm-mingw clang 23 with libomp.dll) with an LTO build (-flto, llvm-mingw links with
lld). This PR's clang build has no LTO; that LTO makes no difference is shown only on the tiny fixture, where
OpenMP's overhead dominates. KMP_BLOCKTIME=200 costs nothing measurable in decode (rows 4 and 5 overlap).

Idle served engine: openai_server.Engine started on the fixture (so with tune_child_env's settings), one
32-token request, then 5 s with no request, CPU time of the engine process over those 5 s, 2 rounds:

engine launcher CPU while idle
clang build dev (no KMP_BLOCKTIME) 3.00 cores (both rounds)
clang build this branch (KMP_BLOCKTIME=200) 0.00 cores
gcc build (dev) either 0.00 cores

So the launcher change is required for the clang build: without it a coli serve with this engine keeps all but
one of its compute threads at 100% between requests.

No experiment manifest is attached: there is no real DeepSeek V4 checkpoint on the test machine. The full-model
figure is the issue's.

Validation (upstream template)

  • make -C c check (Windows, MSYS2 gcc): test-c passes; test-python has 11 errors, all of which fail
    the same way on dev on this machine (see Correctness). Not run on Linux or macOS.
  • make -C c cuda-test / dsv4-cuda-test: not run. No CUDA code changes; the clang engine loads
    coli_cuda_dsv4*.dll through the same backend_loader_dsv4.c (compiled by clang, 0 warnings), but it was not
    run with a GPU (see Not tested).
  • Performance claims include a validated experiment manifest with raw evidence: not attached (tiny fixture
    and microbenchmark only, above).
  • The default CPU build remains dependency-free (gcc unchanged; the clang build links libomp statically)
  • No model files, generated binaries, or benchmark artifacts are included

Compatibility and risks

  • Linux and macOS: unchanged. The KMP_BLOCKTIME default is Windows only, so macOS's V4 (Homebrew libomp, active
    policy from the launcher) keeps its infinite blocktime; whether 200 ms would help it there is not measured.
  • The gcc build on Windows: unchanged, except that the launcher now also sets KMP_BLOCKTIME=200 with the
    active policy, which libgomp ignores. A user's own KMP_BLOCKTIME still wins, OMP_WAIT_POLICY=passive
    gets no KMP_BLOCKTIME (so libomp's blocktime 0), and COLI_NO_OMP_TUNE still removes all of them.
  • Windows release: deepseek_v4.exe changes compiler and OpenMP runtime. Same tokens on the tiny fixture's
    oracle, which is all that was checked: gcc contracts floating-point expressions into FMAs more freely than
    clang by default (-ffp-contract=fast against on) and the clang build has no LTO, so the rounding differs
    and a real checkpoint can pick another token where two are nearly tied. The archive check fails the release if
    the binary is not a libomp build or imports a MinGW DLL.
  • The release build of V4 on Windows has no LTO any more (clang, no linker plugin). On this laptop the gcc build
    was the same speed with and without LTO (table above), and clang without LTO is 3x faster than gcc with it, but
    only on the tiny fixture; the issue's 1.5x was an LTO clang build.
  • c/Makefile now treats any *-windows-gnu compiler as MinGW. Before, such a compiler fell through every
    platform block; now make <engine> CC=clang builds with the Windows flags. Only deepseek-v4 was built and
    run with clang for this PR (coli's GLM path gets the same KMP_BLOCKTIME default, so a clang colibri.exe
    would not spin forever when idle, but none was built).
  • Windows users of V4's CUDA tier: the team now really spins (active, then 200 ms of blocktime) where libgomp
    slept. This is a behaviour change that was not measured: there is no GPU run here. docs/tuning.md records
    severe regressions from active teams contending with CUDA dispatch (GLM), and colibri.c and omp_tune.h
    skip active waiting when a GPU is on. The precedent for V4 is Linux, where the launcher's active policy already
    makes the team spin (bounded by GOMP_SPINCOUNT) with the CUDA tier on.
  • Spinning is not a win everywhere ([Bug]: OpenMP spin-wait tuning is inert on macOS — and applying it costs 2.2x decode on a 32 GB host #707 measured it 2.2x slower on an M1 Max, a host where the team waited on the
    disk). This PR does not change the policy: the launcher already asked for active waiting on Windows, and only
    now does the runtime honour it. OMP_WAIT_POLICY=passive remains the per-host override, also through the
    launcher, and COLI_NO_OMP_TUNE=1 drops all of the launcher's OpenMP settings.

Not tested:

  • The decode and idle-server tables were not rerun with this version's launcher. They are from the first round,
    where the launcher set KMP_BLOCKTIME=200 for V4 on every host and policy; on Windows with the active policy,
    the case measured there, this version sets the same environment, and the passive rows were measured without
    KMP_BLOCKTIME, which is what it now leaves passive. The second round ran the unit tests, the builds and the
    ompbar rounds above.
  • A Vulkan run of the clang engine. The release configuration (VK=1, no LTO) was built, linked and run on the
    tiny fixture's CPU path (Correctness), but no Vulkan device was used, and the shaders were the committed
    .spv files (GLSLC=true): the runner's glslc build of them was not run. The gate is this PR's own Release
    run (release.yml runs on pull requests that change it) and, from now on, ci.yml's Windows job: this PR
    should not be merged before both are green.
  • The workflows themselves (ci.yml, release.yml) were not run, so neither was the OpenMP package step on the
    runner. Its fallback should rarely matter: MSYS2 renamed openmp to llvm-openmp with LLVM 20.1.0 (March
    2025), and llvm-openmp provides and replaces openmp, while the libgomp split that broke the runners'
    older package list (6b09123) came with gcc 16.2.0-4 in September 2026. Either package links statically: the
    last openmp on the MSYS2 mirror (18.1.2) and the first llvm-openmp (20.1.0) both ship a libomp.a of 32
    kmp_*.cpp.obj objects that defines __kmpc_fork_call, beside the libomp.dll.a import library, as 23.1.5
    does.
  • The CUDA tier with the clang engine (coli_cuda_dsv4*.dll loaded at run time). The loader is plain
    LoadLibrary/GetProcAddress C and compiles warning-free with clang, but no GPU run was made.
  • A real DeepSeek V4 checkpoint, other CPUs, and more than 8 threads.
  • The other point of the issue (the release targets x86-64-v3, so the AVX-512 kernels never run on Windows) is
    not addressed here.

Refs #2286

🤖 Generated with Claude Code

KairaCode and others added 2 commits November 12, 2026 19:57
…rs spin (JustVugg#2286)

MinGW's libgomp is configured "mingw32 posix": its barriers wait on a
semaphore with no spin loop, so OMP_WAIT_POLICY=active and GOMP_SPINCOUNT,
which the launcher sets for deepseek_v4, do nothing on Windows, and each of
a token's many small parallel regions pays a thread wake-up. On an
i7-7700HQ (4 threads) an OpenMP barrier costs 22 us with libgomp whatever
those variables say, 0.3 us with LLVM's libomp.

- c/Makefile: clang's MinGW triple (x86_64-w64-windows-gnu) is Windows too,
  so `make deepseek-v4 CC=clang` reaches the V4 build; the static OpenMP
  archive check looks for libomp.a when the compiler is clang.
- c/Makefile.deepseek-v4: with CC=clang on Windows, libomp.a is linked in
  (-static as before, no new DLL), no -pthread (clang's driver links
  winpthreads; the flag only warns), LTO off unless asked for (GNU ld has no
  LLVM plugin in MSYS2's clang package). The gcc line is unchanged.
- c/deepseek_v4.c: drop v4_moe_bank_hash_layer, written and never read
  (clang's -Wunused-but-set-global), so the clang build has 0 warnings.
- c/coli, c/openai_server.py: KMP_BLOCKTIME=200 beside GOMP_SPINCOUNT for
  deepseek_v4. With OMP_WAIT_POLICY=active alone libomp's team spins
  forever: a served clang engine burned 3 cores while idle (JustVugg#341).
- release.yml, ci.yml: the Windows deepseek_v4.exe is built with CC=clang;
  the OpenMP package is installed under either of its MSYS2 names; the
  release checks the packaged engine is a libomp build importing no
  MinGW DLL.
- docs and CHANGELOG.

Tiny V4 fixture on the i7-7700HQ, 7 interleaved rounds, same tokens in
every run: decode 500 -> 1547 tok/s, a 72-token prompt 67 -> 20 ms.
test_deepseek_v4_tiny/_prefix/_brio pass with both builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JustVugg#2286)

The launcher's KMP_BLOCKTIME=200 default for deepseek_v4 is now set on
Windows only, and only when the resulting OMP_WAIT_POLICY is active.
libomp gives passive its blocktime of 0 only while KMP_BLOCKTIME is
unset, so the default turned a user's passive into 200 ms of spinning
after every region. macOS's V4 also links libomp (Homebrew), where the
change was not measured (JustVugg#707), and Linux builds V4 with libgomp. coli's
GLM path, which sets the same active policy on Windows, gets the same
line: a clang colibri.exe now builds with the Windows flags.

ci.yml's Windows job builds the default gcc deepseek-v4 again, and the
release's XDNA=1 VK=1 CC=clang one after it, with the Vulkan headers and
shaderc installed, so the clang compile of the Vulkan units and the
LTO-less link are built on every PR. The Makefile's missing libomp.a
warning points to docs/deepseek-v4.md. Docs and CHANGELOG: same tokens
on the tiny fixture's oracle, the issue's 1.5x came from an LTO build,
the Windows CUDA tier now spins (not measured).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JustVugg

Copy link
Copy Markdown
Owner

Thanks, the analysis in #2286 and here is convincing, and the clang build is the right direction.

We took it into batch #2805, and CI's "All engines (windows)" job failed at the link of the clang build:

ucrt64/lib\libomp.a(kmp_ftn_extra.cpp.obj): undefined reference to `omp_get_num_devices'
ucrt64/lib\libomp.a(kmp_alloc.cpp.obj): undefined reference to `omp_get_num_devices'
clang: error: linker command failed with exit code 1
FAILED: deepseek-v4 CC=clang VK=1

The runner installed mingw-w64-ucrt-x86_64-clang-22.1.7-1 and mingw-w64-ucrt-x86_64-llvm-openmp-22.1.7-1; your measurements used 23.1.5. In 22.1.7 the static libomp.a calls omp_get_num_devices, which it does not define. The release job installs the same packages, so its deepseek_v4.exe would fail the same way.

So #2800 is out of the batch for now, and the other seven PRs went ahead. Could you make the static link work with what MSYS2 ships today (22.1.7), or check for that version up front, and run the Windows job on your branch? Once it passes we will take it in the next batch.

…2286)

libomp.a defines omp_get_num_devices() only as a COFF weak symbol, and
GNU ld leaves it unresolved with MSYS2's llvm-openmp 22.1.7 or binutils
2.46, which is what GitHub's windows runner installs: `make deepseek-v4
CC=clang` failed at the link in JustVugg#2805's CI, on libomp's own kmp_alloc
and kmp_ftn_extra. Only llvm-openmp 23.x with binutils 2.47 links it.

Makefile.deepseek-v4 now links one call to it the same way; only when
that fails on this symbol does backend_loader_dsv4.c define it,
returning 0, what libomp returns when no offload runtime is linked.
Where it links, nothing is added, so there is no second definition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KairaCode

Copy link
Copy Markdown
Author

Thanks for running it. The symbol isn't really missing: libomp.a defines omp_get_num_devices only as a COFF weak symbol, and GNU ld resolves it only with llvm-openmp 23.x plus binutils 2.47. The runner (update: false) has 22.1.7 and binutils 2.46, and 23.1.5 with binutils 2.46 fails too, so a version check alone wouldn't be reliable.

The fix (000475a) is a link probe in Makefile.deepseek-v4: for a clang build on MinGW it links one call to omp_get_num_devices, and only when that fails does backend_loader_dsv4.c define it, returning 0 (no offload devices; colibri uses no target offload). Where it already links, nothing is added.

I ran the Windows job on a fork branch with the same setup-msys2 settings and packages, in two configurations:

  • 22.1.7 / binutils 2.46 (stub on)
  • 23.1.5 / binutils 2.47 (stub off)

In both, make deepseek-v4 XDNA=1 CC=clang VK=1 and the release line link, the exe imports no MinGW DLL, and the tiny DeepSeek V4 oracle is token-exact: https://github.com/KairaCode/colibri/actions/runs/43445682779. The commit is now on this PR's branch.

Not checked there: upstream's full "All engines" loop; only the V4 builds and the tests ran.

Side note: if ci.yml or release.yml ever switch to update: true, today's MSYS2 ships libgomp.a in its own mingw-w64-ucrt-x86_64-libgomp package, so the gcc builds would need it added.

@JustVugg
JustVugg merged commit 3ba7992 into JustVugg:dev Nov 14, 2026
0 of 122 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants