Thanks to the extensive coverage of tasks in the MMTBench, we can evaluate the multimodal performance of LVLMs on a task map. In this way, the roles of different tasks in multimodal capability can be systematically interpreted by analyzing relationships between tasks in the map. Based on the task map, we find that LVLMs obtain a more consistent performance ranking on tasks closer to each other. Besides, our task map can also be used to discover Out-of-Domain (OoD) tasks and In-domain tasks.

The OoD tasks mean tasks that the current model struggles to handle. Discovering OoD tasks can provide insights for future evaluation efforts and the development of stronger LVLMs. To this end, we conduct hierarchical clustering on the task map to find OoD tasks. We use two criteria to identify clusters containing OoD tasks. First, LVLMs would achieve poor performance on OoD tasks. Second, the performance of LVLMs on OoD tasks would be inconsistent with the overall multimodal score in the leaderboard because LVLMs with competitive overall scores would even fail to solve OoD tasks. In this way, we can see that clusters 8, 9, and 11 achieve low multimodal accuracy and ranking correlation. From these clusters, we find that current multimodal large models lack the ability to perform fine-grained visual cognition and understanding of positional and spatial relationships, such as localization and detection tasks. Moreover, they exhibit poor performance in tasks related to new data structures or types of images, showing a lack of proficiency in handling tasks related to GUI and special data structures like tables.

In-domain tasks are tasks that most current multimodal large models can handle correctly. Discovering in-domain tasks guides the commercial application of LVLMs in specific scenarios. Different from OoD tasks, we identify in-domain tasks by looking for clusters with large ranking correlation and high multimodal accuracy. In this way, we can see that clusters 2, 3, and 10 achieve relatively high accuracy and large ranking correlation. We observe that current multimodal large models possess strong high-level visual comprehension capabilities, enabling them to effectively handle visual recognition tasks, even when dealing with specialized images such as medical images. Moreover, they benefit from the powerful LLMs to accurately describe images.

To analyze the error distribution of LVLMs on the MMTBench, we examined three LVLMs: GPT-4V, GeminiProVision, and InternVL-Chat-V1.2 (InternVL). In results, perception error stands out as the most common type of error across all models, with GPT-4V exhibiting a significantly lower perception error rate (51%) compared to GeminiProVision (76.9%) and InternVL (67.2%), indicating its superior performance in perception tasks. Reasoning error emerges as the second most prevalent error type, with InternVL having the highest reasoning error rate (14.8%), followed by GeminiProVision (10.4%) and GPT-4V (9.94%), highlighting the challenges all models face in complex reasoning tasks.