VQAPT: A New visual question answering model for personality traits in social media images
Document Type
Article
Publication Date
11-1-2023
Abstract
Visual Question Answering (VQA) for personality trait images on social media is challenging because of multiple emotions and actions with complex backgrounds in social media images. This work aims at developing a new VQA model for different personality traits (VQAPT) identification in a single image. This work considers the Big Five Factors (BFF) for personality traits namely, Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism. VQA is proposed based on the observation that multiple personality traits can be seen in a single image. We propose a model integrating text recognition and person/face recognition to derive the unique relationship between the text and the person's action in the image. Furthermore, a dynamic text-object graph for personality traits identification is constructed according to the query. For understanding a query, we explore the Contrastive Language-Image Pre-trained (CLIP) transformer encoder in this work. Since it is the first work of its kind, we have created a new dataset under this work for evaluation and the dataset is available publicly as mentioned in Section 4. The effectiveness of the proposed method is also evaluated on two benchmark datasets, namely TextVQA for VQA and PTI for personality traits identification.
Keywords
Personality trait images, Multimodal concept, Text recognition, Social media images, Natural language processing, Visual question answering
Divisions
fsktm
Funders
Ministry of Education, Malaysia (FRGS/1/2020/ICT02/UM/02/4),University Grants Commission, India
Publication Title
Pattern Recognition Letters
Volume
175
Publisher
Elsevier
Publisher Location
RADARWEG 29, 1043 NX AMSTERDAM, NETHERLANDS