AraSSM: A Bidirectional State-Space Encoder for Arabic Masked Language Modeling

Abstract

Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT are the standard backbone for Arabic natural language understanding, but their self-attention scales quadratically with sequence length. AraSSM is, to our knowledge, the first bidirectional Mamba/state-space encoder pretrained specifically for Arabic via masked language modeling, trained end-to-end on four consumer-grade RTX 2080Ti GPUs (~960 GPU-hours) on a cleaned and deduplicated ~80GB corpus of Arabic Wikipedia and CulturaX text.

Method

Each encoder block runs a forward and a backward selective-scan mixer over the same normalized hidden state and merges their outputs through a learned projection, giving full bidirectional context with O(L) complexity in sequence length instead of attention’s O(L²). The model uses a hidden size of 512 across 12 layers, the AraBERTv02 tokenizer, and the standard BERT-style MLM objective, with fp16 mixed precision, gradient checkpointing, and gradient accumulation to fit the 11GB VRAM budget.

Contribution

Evaluated on four Arabic NLU benchmarks under the AraBERT fine-tuning protocol with three seeds, AraSSM matches or exceeds base-sized Transformer baselines on sentiment classification (HARD: 96.37% accuracy), is competitive on extractive QA (ARCD) and named entity recognition (ANERcorp), and trails only on natural language inference (XNLI-ar) — despite being pretrained entirely on consumer hardware rather than accelerator clusters. An ablation isolating MLM pretraining shows it is close to a prerequisite for data-scarce, semantically demanding tasks.