SFAMNet: A scene flow attention-based micro-expression network
Document Type
Article
Publication Date
1-1-2024
Abstract
Tremendous progress has been made in facial Micro-Expression (ME) spotting and recognition; however, most works have focused on either spotting or recognition tasks on the 2D videos. Until recently, the estimation of the 3D motion field (a.k.a scene flow) for the ME has only become possible after the release of the multi-modal ME dataset. In this paper, we propose the first Scene Flow Attention-based Micro-expression Network, namely SFAMNet. It takes the scene flow computed using the RGB-D flow algorithm as the input and predicts the spotting confidence score and emotion labels. Specifically, SFAMNet is an attention-based end-to-end multi-stream multi-task network devised to spot and recognize the ME. Besides that, we present a data augmentation strategy to alleviate the small sample size problem during network learning. Extensive experiments are performed on three tasks: (i) ME spotting; (ii) ME recognition; and (iii) ME analysis on the multi-modal CAS(ME)3 dataset. Empirical results indicate that depth is vital in capturing the ME information and the effectiveness of the proposed approach. Our source code is publicly available at https://github.com/genbing99/SFAMNet. © 2023 Elsevier B.V.
Keywords
Analysis, Attention, Facial micro-expression, Recognition, Scene flow, Spotting
Divisions
ai
Funders
Ministry of Higher Education, Malaysia [Grant no. FRGS/1/2022/ICT02/HWUM/02/1, JWS 2022/01],Ministry of Science and Technology, Taiwan [Grant no. 111-2221-E-035-059-MY3]
Publication Title
Neurocomputing
Volume
566
Publisher
Elsevier B.V.