itselouardi/ZAYA1-8B-bnb-4bit

ZAYA1‑8B – 4‑bit Quantized (bnb‑nf4)

This repo contains a 4‑bit quantized version of Zyphra/ZAYA1‑8B using bitsandbytes (load_in_4bit=True, nf4 quantization type).

It reduces VRAM usage to ~5.1 GB while keeping high generation quality – ideal for local deployment, low‑resource environments, and quick experimentation.

⚡ Part of the ZAYA1 Community Deployment Stack
GitHub: zaya1‑vllm‑docker (CUDA/ROCm Docker images & benchmarks coming soon)

---

📦 Model Details

• Architecture: Mamba‑Transformer hybrid (Zyphra’s ZAYA1 design)
• Original model: Zyphra/ZAYA1‑8B
• Quantization method: BitsAndBytes 4‑bit NF4 with double quantization
• Precision: bfloat16 compute dtype
• VRAM after loading: ~5.1 GB (vs. ~16 GB for FP16)

---

🚀 Quick Start

Install dependencies:

pip install transformers bitsandbytes accelerate torch

Load and generate:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "ayoubelouardi/ZAYA1-8B-bnb-4bit"

# The model loads directly in 4‑bit – no extra config needed
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

prompt = "Once upon a time,"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

⚠️ Note: trust_remote_code=True is required because the model uses a custom architecture.

---

🧪 How it was created

The quantization was performed using the following script (included in the community repo):

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "Zyphra/ZAYA1-8B",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("Zyphra/ZAYA1-8B", trust_remote_code=True)

model.save_pretrained("./ZAYA1-8B-bnb-4bit")
tokenizer.save_pretrained("./ZAYA1-8B-bnb-4bit")

The saved checkpoint automatically includes the quantization config – so you don’t need to set load_in_4bit again when loading from this repo.

---

📊 Benchmarks

GPU	Quantization	Tokens/sec (batch=1)	Notes
NVIDIA A100 80GB	FP16 (original)	coming soon	via vLLM
NVIDIA A100 80GB	4‑bit (this repo)	coming soon	via transformers
AMD MI300X 192GB	FP16 (original)	coming soon	via vLLM + ROCm

Full benchmarks will be published in the GitHub repository. Stay tuned!

---

🔗 Links

• Original model: Zyphra/ZAYA1‑8B
• Zyphra’s vLLM fork: Zyphra/vllm
• Community deployment repo: zaya1‑vllm‑docker (Docker images, GGUF progress, benchmarks)
• Author: Ayoub El Ouardi

---

📜 License

This model is subject to the Zyphra Community License Agreement.
Please review the original license before using.

---

🙏 Acknowledgements

Thanks to Zyphra for releasing ZAYA1‑8B and maintaining the open‑source forks.
This quantized version is a community contribution and is not affiliated with Zyphra.