F²ACT-VQA: Fusion-Driven Fact Retrieval with Generative Reranking for Knowledge-Intensive Visual Question Answering

Abstract

Knowledge-intensive visual question answering requires identifying the subject in an image and retrieving question-relevant external knowledge. Existing approaches often make a hard entity decision before evidence retrieval or retrieve coarse document-level evidence, making them vulnerable to irreversible entity errors and irrelevant context. In this paper, we propose F²ACT-VQA, a novel fusion-driven fact-centric multimodal framework that performs global retrieval over atomic knowledge-base FACTs represented as Subject–Predicate–Object (SPO) triples. Instead of selecting an entity first, F²ACT-VQA converts the image–question pair into a structured multimodal retrieval intent and directly matches it against the entire FACT store. Weighted fusion combines the generated query FACT with the image; multimodal reranking and a category-aware inspector then produce a compact, diverse evidence set for grounded answer generation. Experiments on InfoSeek and Encyclopedic VQA demonstrate the effectiveness of F²ACT-VQA in both fact retrieval and downstream question answering.