HoloCount: A Holistic Visual Counting Benchmark for MLLMs

The evaluation of Multimodal Large Language Models (MLLMs) has arrived at a sobering reality: while these systems excel at qualitative scene description, their quantitative precision remains a critical fragility. In the paper “HoloCount: A Holistic Visual Counting Benchmark for MLLMs,” Deng et al. identify a persistent bottleneck of “numerical hallucinations,” where models produce confident but mathematically untethered responses. The HoloCount benchmark is positioned not as another “flat” accuracy test, but as a diagnostic red-teaming tool designed to shift model assessment from simple perceptual enumeration toward the more rigorous demands of cognitive reasoning.

To move beyond the limitations of previous evaluations, the authors developed HoloCount based on two core principles: Ambiguity Elimination, ensuring every query has a singular well-defined answer, and Category/Scenario Diversity, covering 1,481 unique visual concepts. Current counting benchmarks frequently fail to provide strategic utility due to several systemic limitations:

• Narrow Object Diversity: Most evaluations focus on a limited set of fixed categories, failing to test true generalization.
• Lack of Diagnostic Depth: Single-score accuracy metrics provide no insight into whether a failure is visual, logical, or arithmetic in nature.
• Idealized Environments: Benchmarks often overlook the complex failure modes triggered by high-density scenes, logical constraints, or deceptive linguistic priors.

Decoding HoloCount: The Three-Level Taxonomy

To provide a more granular failure analysis, Deng et al. transitioned away from unstructured mixtures of queries toward a decoupled evaluation framework. This hierarchical structure allows researchers to isolate whether a model’s collapse occurs during initial visual perception or during the subsequent stage of symbolic reasoning.

Taxonomy Pillar	Core Focus	Example Sub-task
I. Semantic Counting	Foundational skills; atomic enumeration and attribute-based filtering (chromatic, material, scale).	Counting based on material (e.g., “How many metal dog statues are visible?”).
II. Analytical Counting	The integration of logic and vision; spatial-based reasoning and set-based operations.	Differential comparison (e.g., “How many more red blocks than blue blocks?”).
III. Robustness Testing	Systemic limits under adverse perceptual domains and resilience against grounded counter-priors.	Linguistic prior conflict (e.g., counting the legs of a three-legged cat).

The “Catastrophic Failure” of High Density

The findings from the Robustness Testing section, specifically the “Dense” subset, reveal a universal weakness across all tested architectures. This is characterized by the authors as an out-of-distribution (OOD) failure, where current vision encoders prove incapable of maintaining fine-grained spatial distinctions when objects heavily overlap. To ensure the results were not skewed by semantic ambiguity, the authors utilized synthetic symbols on a blank canvas to mimic dense distributions.

The performance drop-off is severe. For example, Qwen3.5-27B, which maintains a macro average above 75% on the benchmark, sees its accuracy collapse to 20% in high-density scenarios. The authors hypothesize that current architectural paradigms cannot handle the crowding of the visual canvas, leading to systematic under-counting.

High-Density Performance Statistics (From Source Table 4)

• Gemini-3.1-Pro-Preview: 49.0% Accuracy | 10.46 Mean Absolute Error (MAE)
• Qwen3-VL-32B-Instruct: 3.0% Accuracy | 32.94 MAE
• Gemma3-27B: 0.0% Accuracy | 50.08 MAE

Contextualizing the MAE: Given that the mean count in HoloCount is approximately 8.06 (Source Appendix A.1), an MAE of 50.08 represents a catastrophic predictive failure, with errors deviating significantly from the true quantity.

The Inverse Paradox: When Smaller Models Outperform the Giants

HoloCount identifies a striking “Inverse Performance Paradox” within the “Null-Target Prompting” subset. This task evaluates whether a model hallucinates when asked to count objects absent from the image. Paradoxically, smaller, less complex models frequently outperformed the most advanced proprietary engines. Lightweight models like Qwen3-VL-8B-Instruct and Qwen2.5-VL-7B-Instruct achieved near-perfect scores (96.4% and 95.6% respectively), while Gemini-3.1-Pro-Preview collapsed to 55.2%.

The authors suggest this is a symptom of “over-confidence.” Larger models, optimized for exhaustive scene retrieval, become reluctant to output a count of zero, succumbing to the “Siren Call” of linguistic priors—the tendency to rely on textual training data (e.g., “cats have four legs”) rather than the grounded visual evidence of the specific image.

Pro-Tip: The Gemini Advantage While proprietary giants struggled with null targets, the Gemini series demonstrated a unique resilience in the “Linguistic Prior Conflict” subset. Gemini-3.1-Pro-Preview achieved 73.6% accuracy, suggesting the presence of a robust “vision-first” arbitration mechanism capable of suppressing textual biases in favor of observed pixels.

Performance Insights and the “Thinking Mode” Advantage

Following a systematic evaluation of over 20 MLLMs, the authors identified three primary performance trends that define the current landscape of multimodal intelligence:

1. The Narrowing Gap: Advanced open-source models now rival proprietary leaders. The Qwen3.5-397B-A17B model achieved the highest macro average across 20 subsets at 76.9%, narrowly surpassing Gemini-3.1-Pro-Preview (74.7%).
2. Logic-Perception Dissociation: There is a sharp performance cliff between perception and reasoning. Most models exceed 85% accuracy in basic atomic counting but drop by 40 points or more when tasked with set-based logic like “Differential Comparison.”
3. System 2 Reasoning Benefits: The application of “thinking mode” (extended step-by-step reasoning) consistently improved accuracy. This move from “System 1” pattern matching to “System 2” deliberate calculation resulted in gains of 10.6 to 15.4 absolute percentage points. Notably, smaller models (4B/9B) saw higher relative gains than the 397B scale, indicating that counting is a deliberate process that can be effectively prompted.

Conclusion: A Roadmap for Reliable Multimodal Systems

The HoloCount benchmark demonstrates that current MLLMs are far from reliable in count-sensitive environments such as logistics, retail analytics, or safety monitoring. The systematic failures in high-density scenes and the fragility of models when faced with deceptive priors suggest that visual grounding remains a critical performance gap.

The authors provide a clear roadmap for future development: the industry must move beyond superficial perception. Developing truly grounded systems will require new architectures that prioritize cognitive reasoning and arithmetic composition, ensuring that multimodal systems can resist the pull of linguistic hallucinations in favor of verifiable visual truth.

Key Takeaways

• Quantitative Fragility: High-performance models remain surprisingly unreliable when asked to perform exact enumeration under logical constraints.
• Architecture vs. Scale: Scaling model parameters does not automatically solve counting hallucinations; failures in dense and null-target scenes are systemic architectural issues.
• Reasoning Over Perception: The primary bottleneck is rarely the model’s ability to “see” the object, but rather the symbolic and arithmetic reasoning required after perception.
• Red-Teaming the Future: Benchmarks like HoloCount are essential for identifying the boundaries of visual reasoning reliability before these models are deployed in high-stakes environments.

Read the full paper on arXiv · PDF