Author(s): Xu, XT (Xu, Xintao); Liu, Y (Liu, Yi); Chen, G (Chen, Gang); Ye, JB (Ye, Junbin); Li, ZG (Li, Zhigang); Lu, HX (Lu, Huaxiang)

Source: COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE Volume: 2022 Article Number: 4398839 DOI: 10.1155/2022/4398839 Published: MAY 28 2022

Abstract: In the field of natural language processing (NLP), machine translation algorithm based on Transformer is challenging to deploy on hardware due to a large number of parameters and low parametric sparsity of the network weights. Meanwhile, the accuracy of lightweight machine translation networks also needs to be improved. To solve this problem, we first design a new activation function, Sparse-ReLU, to improve the parametric sparsity of weights and feature maps, which facilitates hardware deployment. Secondly, we design a novel cooperative processing scheme with CNN and Transformer and use Sparse-ReLU to improve the accuracy of the translation algorithm. Experimental results show that our method, which combines Transformer and CNN with the Sparse-ReLU, achieves a 2.32% BLEU improvement in prediction accuracy and reduces the number of parameters of the model by 23%, and the sparsity of the inference model increases by more than 50%.

Accession Number: WOS:000819224500011

PubMed ID: 35669640

Author Identifiers:

Author        Web of Science ResearcherID        ORCID Number

Liu, Yi                  0000-0003-3056-7713

Xu, Xintao                  0000-0002-3389-7518

ISSN: 1687-5265

eISSN: 1687-5273

Full Text: https://www.hindawi.com/journals/cin/2022/4398839/