Kyoseong Koo  Hyeong Jin Shin  Jae Sung Lee

10.5626/JOK.2025.52.11.961

Bi-directionalization encoder language model RWKV model lightweight model

In recent years, Transformer-based language models have been extensively developed and have shown substantial effectiveness across various natural language processing tasks. However, these models often incur significant computational costs in terms of time and memory complexity and they are predominantly designed as unidirectional autoregressive architectures. To address these limitations, research has increasingly focused on developing lightweight and bidirectional alternatives. This paper proposes Bi-RWKV, a bidirectional extension of the lightweight RWKV model, specifically designed for encoder-based language tasks. By examining eight configurations of bidirectional integration for RWKV’s time-mixing and channel-mixing modules, we identify the optimal architecture. To ensure a fair comparison of different model architectures, we maintained consistent hyperparameter values and comparable numbers of model parameters, deliberately omitting pretraining. Experimental results on named entity recognition, chunking, and Korean morphological and part-of-speech tagging demonstrate that Bi-RWKV achieves comparable or superior accuracy to Transformer-based encoders while reducing inference time by a factor of 2.7 to 4.