We evaluate the expressiveness and performance of our proposed architecture using standard au-
toregressive language modeling benchmarks, comparing its results to those of existing open-source
LLMs, including RNN-based methods (Peng et al., 2023; Gu & Dao, 2023), in Table 10. Evaluations
are conducted using both pre-training metrics, specifically perplexity, and zero-shot performance
measures. For training, we used the Pile dataset (Gao et al., 2020) over a single epoch, adhering
to the training strategy used by Biderman et al. (2023). The Adam optimizer is applied with a
6 × 10−4 learning rate, and each batch contains 1024 samples with a sequence length of 2048 tokens,
mirroring the setup of Mamba. The training process consists of 14,300 steps, with a 1430-step
warmup, equivalent to 1% of total training steps. Detailed model specifications are in Table 9.
