Abstract: Due to the increasing adoption of transfer learning approaches, Natural Language Processing (NLP) has undergone a significant transformation particularly in the field of text classification. Through the application of transfer learning, many pre-trained transformer models, such as BERT, distilBERT, and RoBERTa, have emerged and shifted attention towards NLP. These transformer models are already trained on huge corpora of unlabeled text, helping them to deeply recognize semantics, grammar, and contextual relationships. Unlike other deep learning models, transformers are not dependent on large task-specific datasets. Instead, they support pre-training information to generalize effectively across several NLP tasks like text classification. In this study, we utilized two datasets: the IMDB dataset for binary text classification and AG News dataset for multi-class classification. To achieve our objective, we have implemented five transformer models - BERT, distilBERT, RoBERTa, ALBERT and XLNET - on both datasets to train our model. Among these models, RoBERTa achieved the highest accuracy of 95.4% on the IMDB dataset and 94.2% on the AG news dataset. Although with such state-of-the-art performance, the RoBERTa model emerged as the most effective and robust technique for text classification technique as the confusion matrix also confirmed its superior precision and balanced classification abilities.

Key words: text classification, transfer learning, pre-trained models, BERT, distilBERT, RoBERTa, ALBERT, XLNet