AraFusion/arafusion-mdlm-641m-charlevel-scratch-v2

AraFusion — Arabic Masked Diffusion Language Model

Trained with MDLM on FineWeb-2 Arabic subsets. Supports dialect-conditioned generation via Classifier-Free Guidance (CFG).

Model details

Architecture	DiT (BERT-base scale: 12L / 12H / 768d)
Vocab	96 000 (MorphBPE, see AraFusion/arafusion-morphBPE)
Sequence length	512 tokens
Diffusion	Linear noise schedule, 1 000 sampling steps
Dialects	MSA ([MSA]), Najdi ([NAJDI]), Egyptian ([EGYPT])
CFG p_uncond	0.10
Training steps	200 000 (pre-train)
Precision	BF16

Data

Trained on three FineWeb-2 Arabic subsets:

Subset	Dialect	Split
arb_Arab	Modern Standard Arabic	train 10 %
ars_Arab	Najdi / Saudi Arabic	train (full)
arz_Arab	Egyptian Arabic	train (full)

See AraFusion/arafusion-arabic-raw for the raw text and AraFusion/arafusion-arabic-packed for the packed training sequences.

Checkpoints

Training job: charlevel-641m-51502

Path in repo
checkpoints/0-1000-v1.ckpt
checkpoints/0-1000-v2.ckpt
checkpoints/0-1000.ckpt
checkpoints/0-100000.ckpt
checkpoints/0-102000.ckpt
checkpoints/0-104000.ckpt
checkpoints/0-106000.ckpt
checkpoints/0-108000.ckpt
checkpoints/0-110000.ckpt
checkpoints/0-112000.ckpt
checkpoints/0-114000.ckpt
checkpoints/0-116000.ckpt
checkpoints/0-118000.ckpt
checkpoints/0-120000.ckpt
checkpoints/0-122000.ckpt
checkpoints/0-124000.ckpt
checkpoints/0-126000.ckpt
checkpoints/0-128000.ckpt
checkpoints/0-130000.ckpt
checkpoints/0-132000.ckpt
checkpoints/0-134000.ckpt
checkpoints/0-136000.ckpt
checkpoints/0-138000.ckpt
checkpoints/0-140000.ckpt
checkpoints/0-142000.ckpt
checkpoints/0-144000.ckpt
checkpoints/0-146000.ckpt
checkpoints/0-148000.ckpt
checkpoints/0-150000.ckpt
checkpoints/0-152000.ckpt
checkpoints/0-154000.ckpt
checkpoints/0-156000.ckpt
checkpoints/0-158000.ckpt
checkpoints/0-160000.ckpt
checkpoints/0-162000.ckpt
checkpoints/0-164000.ckpt
checkpoints/0-166000.ckpt
checkpoints/0-168000.ckpt
checkpoints/0-170000.ckpt
checkpoints/0-172000.ckpt
checkpoints/0-174000.ckpt
checkpoints/0-176000.ckpt
checkpoints/0-178000.ckpt
checkpoints/0-180000.ckpt
checkpoints/0-182000.ckpt
checkpoints/0-184000.ckpt
checkpoints/0-186000.ckpt
checkpoints/0-188000.ckpt
checkpoints/0-190000.ckpt
checkpoints/0-192000.ckpt
checkpoints/0-194000.ckpt
checkpoints/0-196000.ckpt
checkpoints/0-198000.ckpt
checkpoints/0-200000.ckpt
checkpoints/0-41500-v1.ckpt
checkpoints/0-42000.ckpt
checkpoints/0-42500.ckpt
checkpoints/0-43000.ckpt
checkpoints/0-43500.ckpt
checkpoints/0-44000.ckpt
checkpoints/0-44500.ckpt
checkpoints/0-45000.ckpt
checkpoints/0-45500.ckpt
checkpoints/0-46000.ckpt
checkpoints/0-46500.ckpt
checkpoints/0-47000.ckpt
checkpoints/0-47500.ckpt
checkpoints/0-48000.ckpt
checkpoints/0-48500.ckpt
checkpoints/0-49000.ckpt
checkpoints/0-49500.ckpt
checkpoints/0-50000-v1.ckpt
checkpoints/0-50000.ckpt
checkpoints/0-50500.ckpt
checkpoints/0-51000.ckpt
checkpoints/0-51500.ckpt
checkpoints/0-52000.ckpt
checkpoints/0-52500.ckpt
checkpoints/0-53000.ckpt
checkpoints/0-53500.ckpt
checkpoints/0-54000.ckpt
checkpoints/0-54500.ckpt
checkpoints/0-55000.ckpt
checkpoints/0-55500.ckpt
checkpoints/0-56000.ckpt
checkpoints/0-56500.ckpt
checkpoints/0-57000.ckpt
checkpoints/0-57500.ckpt
checkpoints/0-58000.ckpt
checkpoints/0-58500.ckpt
checkpoints/0-59000.ckpt
checkpoints/0-59500.ckpt
checkpoints/0-60000.ckpt
checkpoints/0-60500.ckpt
checkpoints/0-61000.ckpt
checkpoints/0-61500.ckpt
checkpoints/0-62000.ckpt
checkpoints/0-62500.ckpt
checkpoints/0-63000.ckpt
checkpoints/0-63500.ckpt
checkpoints/0-64000.ckpt
checkpoints/0-64500.ckpt
checkpoints/0-65000.ckpt
checkpoints/0-65500.ckpt
checkpoints/0-66000.ckpt
checkpoints/0-66500.ckpt
checkpoints/0-67000.ckpt
checkpoints/0-67500.ckpt
checkpoints/0-68000.ckpt
checkpoints/0-68500.ckpt
checkpoints/0-69000.ckpt
checkpoints/0-69500.ckpt
checkpoints/0-70000.ckpt
checkpoints/0-70500.ckpt
checkpoints/0-71000.ckpt
checkpoints/0-71500.ckpt
checkpoints/0-72000.ckpt
checkpoints/0-72500.ckpt
checkpoints/0-73000.ckpt
checkpoints/0-73500.ckpt
checkpoints/0-74000.ckpt
checkpoints/0-74500.ckpt
checkpoints/0-75000.ckpt
checkpoints/0-76000.ckpt
checkpoints/0-78000.ckpt
checkpoints/0-80000.ckpt
checkpoints/0-82000.ckpt
checkpoints/0-84000.ckpt
checkpoints/0-86000.ckpt
checkpoints/0-88000.ckpt
checkpoints/0-90000.ckpt
checkpoints/0-92000.ckpt
checkpoints/0-94000.ckpt
checkpoints/0-96000.ckpt
checkpoints/0-98000.ckpt
checkpoints/best.ckpt
checkpoints/last-v1.ckpt
checkpoints/last.ckpt

Usage

# Requires the MDLM library from https://github.com/kuleshov-group/mdlm
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("AraFusion/arafusion-morphBPE")

# Load checkpoint and run a sampling step — see MDLM docs for full API.

Citation

@misc{arafusion2026,
  title   = {AraFusion: Dialect-Conditioned Arabic Masked Diffusion Language Model},
  year    = {2026},
  url     = {https://huggingface.co/AraFusion}
}