Spectral-spatial feature fusion for real-time facial expression recognition

Abstract Facial expression recognition (FER), as a critical task in computer vision and affective computing, has gained considerable attention in recent years. However, current methods often suffer from high computational costs and limited capability in extracting key discriminative features. To address these issues, this paper proposes SPAYOLO (Spectral-aware Perception and Aggregation YOLOv8), a novel FER network based on the YOLOv8 architecture. We introduce a new Spectral-aware Perception and Aggregation Module (SPAM), designed to enhance expression recognition performance by systematically modeling spatial and frequency features. SPAM comprises three components: a Hierarchical Receptive Modeling (HRM) path that uses multi-scale convolutional branches to capture fine-grained and mid-level spatial variations; a Frequency Enhancement Path (FEP) that leverages Fast Fourier Transform (FFT) to extract high-frequency texture and micro-expression features; and a Gated Attention Mechanism (GAM) that adaptively fuses spatial and frequency features to mitigate feature distribution inconsistency and improve discriminative stability. Experimental results show that the proposed model achieves an accuracy of 70.74% on the FER2013 dataset and 67.88% on the AffectNet dataset, while maintaining high computational efficiency. These results highlight its suitability for real-time facial expression recognition tasks.Our findings validate the effectiveness of hierarchical feature fusion and frequency-domain enhancement in FER tasks, offering valuable insights for future research in computer vision.The custom code for this study is available at GitHub repository: https://github.com/YociLam/Spectral-Spatial-Feature-Fusion-for-Real-Time-Facial-Expression-Recognition .