Breaking Neural Barriers with Transformers Replacing RNN and CNN
DOI:
https://doi.org/10.21467/proceedings.7.6.51Keywords:
Transformers, RNNs, CNNsAbstract
Transformers have emerged as a groundbreaking advancement in deep learning, addressing the inherent limitations of traditional architectures like Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs). This paper presents a comprehensive comparative analysis of these architectures, highlighting the structural superiority, computational efficiency, and scalability of transformers. The novel contributions of this study include: (1) a controlled experimental framework for unbiased evaluation across RNNs, CNNs, and transformers using synthetic multiclass time-series datasets; (2) optimization techniques for transformers, such as sparse attention and hybrid models combining CNNs and transformers, which enhance computational efficiency while maintaining high performance; and (3) development of lightweight transformer models tailored for edge computing applications through pruning, quantization, and knowledge distillation. Empirical results demonstrate that transformers outperform RNNs and CNNs in capturing long-range dependencies, global context, and complex patterns across diverse tasks in natural language processing (NLP), computer vision, and multimodal learning. Furthermore, the study explores real-world applications in healthcare, finance, and autonomous systems to validate the practical utility of optimized transformer models. These findings position transformers as pivotal drivers of future advancements in artificial intelligence.
References
[1] A. Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, vol. 30, pp. 5998–6008, 2017.
[2] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[3] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv:1810.04805, 2019.
[5] A. Dosovitskiy et al., “An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929, 2021.
[6] Z. Dai et al., “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context,” arXiv preprint arXiv:1901.02860, 2019.
[7] Z. Liu et al., “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows,” in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2021, pp. 10012–10022.
[8] W. Wang et al., “PVT v2: Improved Baselines with Pyramid Vision Transformer,” Computer Vision and Image Understanding, vol. 215, p. 103327, 2022.
[9] S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” in Proc. ICLR, 2016.
[10] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” in Proc. NIPS Deep Learning Workshop, 2015.
[11] Z. Sun et al., “MobileBERT: A Compact Task-Agnostic BERT for Resource-Limited Devices,” arXiv preprint arXiv:2004.02984, 2020.
[12] V. Sanh et al., “DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter,” arXiv preprint arXiv:1910.01108, 2019.
[13] S. Nerella et al., “Transformers and Large Language Models in Healthcare: A Review,” Artificial Intelligence in Medicine, vol. 154, p. 102900, 2024.
[14] S. Lundberg and S.-I. Lee, “A Unified Approach to Interpreting Model Predictions,” in Proc. NeurIPS, 2017, pp. 4765–4774.
[15] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’ Explaining the Predictions of Any Classifier,” in Proc. ACM KDD, 2016, pp. 1135–1144.
[16] IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems, “Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems,” IEEE Standards Association, 2020.
[17] A. Radford et al., “Learning Transferable Visual Models from Natural Language Supervision,” in Proc. ICML, vol. 139, 2021, pp. 8748–8763.
[18] European Union, “General Data Protection Regulation (GDPR),” Official Journal of the European Union, 2018.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
