Logo PhysBench

Benchmarking and Enhancing VLMs for
Physical World Understanding

Wei Chow1*, Jiageng Mao1*, Boyi Li2, Daniel Seita1, Vitor Guizilini3, Yue Wang1
1University of Southern California, 2UC Berkeley,3Toyota Research Institute
*Equal Contribution
arXiv Code

🤗

 Dataset

🏆

 Leaderboard

🔺

 Eval.AI

Introduction

Vision-Language Models (VLMs) have emerged as promising tools for building embodied agents, whereas their lack of physical world understanding hampers their effectiveness in real-world applications. To address this challenge, we present Logo PhysBench, a comprehensive benchmark designed to evaluate and enhance VLMs' understanding of the physical world across diverse and complex tasks.

PhysBench comprises 100,000 entries of interleaved video-image-text data, and the data is categorized into four major classes: physical object properties, physical object relationships, physical scene understanding, and physics-driven dynamics, covering 19 subclasses and 10 distinct capability dimensions.

Our extensive experiments on 39 representative VLMs reveal significant gaps in physical world understanding, likely due to the absence of physical knowledge in their training data. To improve VLMs' physical understanding, we propose an agent-based method called PhysAgent, which leverages prior physical knowledge and expert model assistance to enhance physical world understanding capabilities.

Furthermore, we demonstrate that improving VLMs’ understanding of the physical world can significantly facilitate the deployment of embodied agents in real-world scenarios, moving towards bridging the gap between human and machine intelligence in comprehending the physical world.

Leaderboard on PhysBench

Accuracy scores for General VLM on the test subset (10,002 entries) of Logo PhysBench.

Accuracy scores for Image VLM and Video VLM on the test subset without interleaved data entries (8,099 entries) of Logo PhysBench.


Method types: Seq ⏩: Sequential input of images after frame selection from videos,, Merge 🖼️: merging video frames into a single image

🚨 To submit your results to the leaderboard, please send to this email with your result json files.

🚨 For more submission details, please refer to this link and this link.

Logo PhysBench Dataset

Overview

We propose Logo PhysBench, which comprehensively evaluates VLMs' perception of the physical world across four major task categories:

  • (1) Physical Object Properties: Assessment of physical attributes of objects such as mass, size, density, tension, friction, bending stiffness, elasticity, and plasticity.
  • (2) Physical Object Relationships: Evaluation of spatial relationships involving object movement, speed, and position.
  • (3) Physical Scene Understanding: Interpretation of environmental factors, including light sources, viewpoints, temperature, and so on.
  • (4) Physics-based Dynamics: Understanding of physical events like collisions, throwing, fluid dynamics, explosions, and similar phenomena.
algebraic reasoning

Sampled PhysBench examples from four major dimensions


arithmetic reasoning
A comparison between Logo PhysBench. and other physical understanding question-answering benchmarks reveals that
Logo PhysBench is a comprehensive dataset, covering a wide range of tasks related to physical world understanding.

The complete Logo PhysBench dataset consists of 100,000 entries, organized into 19 subclasses and 10 distinct capability dimensions. For convenience, we selected a subset of 10,002 entries, which are more challenging and diverse, as the test set, and 200 entries as the validation set for parameter choosing.

  • val: 200 examples used for model development, validation, or for those with limited computing resources.
  • test: 10,002 examples for standard evaluation (include val). Notably, the answer labels for test will NOT be publicly released.
  • train: The remaining 89,998 examples.
You can download the dataset on Hugging Face Dataset.

data-overview

Key statistics of Logo PhysBench.

Statistics

Can VLMs Understand the Physical World?

To assess whether VLMs can understand the physical world, we evaluated 39 representative VLMs on Logo PhysBench and found that:

  • VLMs exhibit limited understanding of the physical world.
  • Closed-source models generally perform better.

grade-lv

The visualization of model performance across 19 sub-tasks is presented, where different colors represent the respective categories.
The four colors, from left to right, represent physical object properties, physical object relationships, physical scenes, and physical-based dynamics.

Why Do VLMs Struggle with Physical World Understanding?

To assess whether VLMs can understand the physical world, we evaluated 39 representative VLMs on Logo PhysBench and found that:

  • Physical world understanding differs significantly from common VQA tasks. We established a correlation map with the common VQA benchmark. Our analysis identifies a notable distinction between PhysBench and traditional VLM benchmarks, with PhysBench demonstrating a closer alignment with MMMU, which necessitates complex reasoning and diverging from the majority of other benchmarks.
  • VLMs's physical world understanding ability does not scale with model size, data, or frames.While keeping the data size constant, increasing the model size or, alternatively, increasing the data size while keeping the model size unchanged, led to inconsistent results. Similarly, increasing the number of frames also yielded unstable outcomes.

We were perplexed by the fact that increasing the amount of training data did not improve the VLM's understanding of the physical world. To investigate further, we examined the training datasets of LLaVA-1.5, VILA-1.5, and PLLaVA-1.5 and identified a lack of physical world knowledge in these datasets. Additionally, keywords frequently encountered in Logo PhysBench are notably rare in the training data of these model. This deficiency in relevant data likely contributes to the VLM's poor comprehension of physical world concepts. We further support this hypothesis by analyzing the error distribution and fine-tuning the VLM in subsequent experiments.

To investigate the poor performance of VLMs on Logo PhysBench, we randomly selected 500 questions and obtained explanations from three models—GPT-4o, Phi-3V, and Gemini-1.5-flash. Expert annotators classified the root causes of the mispredictions into six categories: perception errors, reasoning errors, lack of knowledge, refusal to answer, failure to follow instructions, and annotation errors in the dataset. We find that perceptual and knowledge gaps constitute the majority of errors.


grade-lv

Distribution of error types for GPT-4V, Gemini-1.5-flash, Phi-3V.

Our error analysis revealed that inadequate physical world knowledge and reasoning capabilities were key contributors to the models’ poor performance. To investigate whether introducing additional examples could enhance performance, we conducted tests on 200 entries of Logo PhysBench, pairing each with a similar example. These additional examples were incorporated through fine-tuning or in-context learning. As shown in the below figure, the performance improvements after adding physical world knowledge examples indicate that VLMs can transfer physical knowledge to some extent. This suggests that the original data’s lack of physical world knowledge was a significant factor in the models’ suboptimal performance.

grade-lv

Physics knowledge transfer study.

How to enhance VLMs for Physical World Understanding?

Recognizing perceptual inaccuracies and knowledge gaps as key sources of error, we introduce PhysAgent to improve VLMs' understanding of the physical world by integrating expert models for enhanced perception and incorporating memory for physical knowledge.


grade-lv

Architecture of PhysAgent.

The results lead to the following conclusions:
(1) Prompting methods is unstable, and using pure language yields catastrophic results. As observed, the CoT strategy has minimal impact, while both Desp-CoT and PLR show a decline in performance. This suggests that descriptive prompts are not particularly effective for addressing the questions, implying that our dataset requires a deeper understanding of the videos or images to answer accurately.
(2) ContPhy even worsens performance. In three out of four tasks, ContPhy underperforms compared to its base model, GPT-4o, due to suboptimal module invocation and limited flexibility in its logical templates, which struggle to adapt to diverse scenarios. Additionally, ContPhy relies on models like RCNN to process visual information instead of directly leveraging GPT-4o, leading to potential information loss and subsequent performance degradation.
(3) PhysAgent consistently improves zero-shot performance, notably achieving a 36.5% improvement for GPT-4o in spatial tasks. Compared to the CoT, Desp-CoT, and PLR prompting strategies, our method demonstrates clear advantages.


grade-lv

Performance of different methods.

Can Physical World Understanding Help Embodied Application?

we conducted five embodied agent tasks to verify enhancing VLMs' physical understanding facilitates the deployment of embodied agents.

we observe consistent improvements after fine-tuning with a subset of Logo PhysBench, indicating that the benchmark's data is of high quality and suitable for use as demonstration data in open-world robotics tasks. Additionally, PhysAgent consistently yields stable zero-shot gains across all five tasks, with especially significant progress observed in the force task. While the improvements are less pronounced compared to direct fine-tuning.


grade-lv

Performance on 5 embodied tasks shown in the former figure.
The color blocks from left to right represent success, VLM reasoning error, and execution error, respectively.

BibTeX


      @inproceedings{
          TODO
      }