Cross-language dissemination of Chinese classical literature using multimodal deep learning and artificial intelligence

Abstract Against the backdrop of rapid advancements in artificial intelligence (AI), multimodal deep learning (DL) technologies offer new possibilities for cross-language translation. This work proposes a multimodal DL-based translation model, the Transformer-Multimodal Neural Machine Translation (TMNMT), to promote the cross-language dissemination and comprehension of Chinese classical literature. The proposed model innovatively integrates visual features generated by conditional diffusion models and leverages knowledge distillation techniques to achieve efficient transfer learning, fully exploiting the latent information in multilingual corpora. The work designs a gated neural unit-based multimodal feature fusion mechanism and a decoder-based visual feature attention module to enhance translation performance, thus dynamically combining textual and visual information. Experimental results demonstrate that TMNMT significantly outperforms baseline models in multimodal and text-only translation tasks. It achieves a BLEU score of 39.2 on the Chinese literature dataset, a minimum improvement of 1.55% over other models, and a METEOR score of 64.8, with a minimum improvement of 8.14%. Moreover, incorporating the decoder’s visual module notably boosts performance, with BLEU and METEOR scores on the En-Ge Test2017 task improving by 2.55% and 2.33%, respectively. This work provides technical support for the multilingual dissemination of Chinese classical literature and broadens the application prospects of AI in cultural domains.