bitseek - Flash Multi Head Latent Attention inspired by deepseek.

[chetanreddyv]

Features

• 1.58-bit weight quantization (ternary values)
• 8-bit activation quantization
• Flash Multi-head Latent Attention with causal masking
• Rotary Position Embeddings (RoPE)
• SwiGLU activation in FFN layers
• SubLN normalization
• No bias terms in linear or normalization layers
• Llama 2 tokenizer integration

Architecture Overview

1. Sequence Length: 2048
2. Model Size: ~300M parameters
3. Key Components:
   • BitLinear layers for efficient weight and activation quantization
   • Rotary Position Embeddings (RoPE) for positional encoding
   • SwiGLU activation in Feed-Forward Network (FFN) layers
   • SubLN (Sub-layer Normalization) normalization
   • No bias terms in linear or normalization layers
   • Flash Multi-head latent attention inspired by DeepSeek

I have the code for this implementation but no compute, I want to contribute this code here as open source.

[BradKML]

Would this also apply to models like Qwen/Dots with open weights? Also, why not do "smart" PTQ with bitnet? Then again there needs to be a sharable library for this https://huggingface.co/nisten/Biggie-SmoLlm-0.15B-Base

[sprappcom]

8b version when out? i think it'll be great with 8b active parameters too. any etas on expectation?
i really hope for an bitnet 1.58 8b with 4096 context and i'll be really happy to use it for conversation ai and general filtering of spam content

[BradKML]

MS BitNet is likely just proof-of-concept, not something they will just remake, since its likely too valuable