Skip to content

Commit a82fabc

Browse files
committed
update readme
1 parent 6ed5335 commit a82fabc

File tree

2 files changed

+10
-2
lines changed

2 files changed

+10
-2
lines changed

README.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,15 @@
22
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
33
![version](https://img.shields.io/badge/version-1.0-blue)
44

5-
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).
5+
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support **fast** and **lossless** inference of 1.58-bit models on CPU (with NPU and GPU support coming next).
6+
7+
The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of **1.37x** to **5.07x** on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by **55.4%** to **70.0%**, further boosting overall efficiency. On x86 CPUs, speedups range from **2.37x** to **6.17x** with energy reductions between **71.9%** to **82.2%**. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. More details will be provided soon.
8+
9+
<img src="./assets/m2_performance.jpg" alt="m2_performance" width="800"/>
10+
11+
>The tested models are dummy setups used in a research context to demonstrate the inference performance of bitnet.cpp.
12+
13+
We hope the release of bitnet.cpp can inspire more 1-bit LLMs trained in large-scale settings.
614

715
## Demo
816

@@ -18,7 +26,7 @@ https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
1826

1927
## Supported Models
2028

21-
bitnet.cpp supports a list of 1-bit models available on [Hugging Face](https://huggingface.co/)
29+
bitnet.cpp supports a list of 1-bit models available on [Hugging Face](https://huggingface.co/), which are trained with research settings.
2230

2331

2432
<table>

assets/m2_performance.jpg

238 KB
Loading

0 commit comments

Comments
 (0)