CELLO: Causal Evaluation of Large Vision-Language Models

Dataser Construction Pipeline

First, we extract causal graphs from scene graphs that include relationships and regions within an image. Then, we select corresponding causal tasks based on the ladder of causation. Finally, causal questions are constructed by employing templates with an LLM. We consider four types of causal graphs and twelve different causal tasks in total.

Specifically,we divide the pipeline into 3 aspects.

Causal Graph Extraction

Preprocess Visual Genome Dataset(it has comprehensive suite of images along with corresponding scene graphs): Specifically, we first catalog and analyze every relationship type present in Visual Genome, with a focus on those signifying arrangement, positioning, and other significant interactions, such as those labeled “support”, “fixed to”, and “hold”. Then, we compile a set of graph templates drawn from multiple sources in the literature,including direct, confounding, collision, and chain.These templates illustrate various toy problems in causal reasoning using well-defined graph structures. Finally, we perform isomorphic subgraph matching against these predefined templates to determine the type of causal graph extracted.

Causal Task Selection

Select 12 tasks in 4 rungs in ladder of causation.

Discovery (Rung 0). We include causal tasks such as causality identification (CaI, e.g., “Which of the following elements is crucial for the girl’s safety?”), causal attribution (CA, e.g., “What indirectly causes the balloon’s stability?”), and abstract reasoning (AR, e.g., “What is indirectly influenced by the wave’s force?”).

Association (Rung 1). We consider collider bias (CB, e.g., “Why don’t the balloons fly away?”).

Intervention (Rung 2). We inquire about confounder identification (CoI, e.g., “Why are the books placed steadily?”), backdoor adjustment set (BAS, e.g., “To assess the relationship between the solidity of shelves and the stability of books, which of the following variables should we control for? ”), and controlled direct effect (CDE, e.g., “If the state of the wall is not changed and the shelves become unstable, will the books drop?”).

Counterfactual (Rung 3). We explore counterfactual scenarios such as counterfactual reasoning (CR, e.g., “If the shelf has fallen down, would the books still be placed steadily?”), natural direct effect (NDE, e.g., “If the retainer of the shelf has been removed, would the books drop?”), natural indirect effect (NIE, e.g., “If the shelf has been fixed to a unstable wall, would the books stay steady?”), sufficient cause (SC, e.g., “If the wall has fallen down, would the books drop?”), and necessary cause (NC, e.g., “If the balloons has flown away, would the woman let go?”).

Causal Question Construction

Question Construction : Prompt LLM to generate causal questions bt applying in-context learning.

Follow 3 instructions:

The demonstration provides:

(1) Relevant descriptions, which are extracted from the dataset descriptions that are associated with the core entities. For instance, “books are on the shelf ”

(2) Causal graph, which is constructed through the process of Section 4.1. Each edge of the graph is expressed in textual form, such as “shelf supports books”.

(3) Constraints, which ensure the validity of the question and prevent information leakage, such as “do not include ‘shelf’ or ‘wall’ in your generated question”.

Answer Construction:

Two Settings: (1) Multiple-Choice Format,3 distractors and 1 correct answer. The three distractors are constructed using the entities based on the following constraints:

(1) Irrelevant entities (Image Distractor): These entities are present in the image but absent from the causal graph, such as “window”.

(2) Partially correct entities (Graph Distractor): These entities are present in the causal graph but only represent part of the correct answer, such as “shelf”.

(3) Induced entities (Text Distractor): These entities are neither in the image nor in the causal graph but introduced solely from the question text, such as “bookends”. This distractor can also be seen as a object hallucination

(2) Binary Answer : Yes or No.