Task Ability Decomposition and Difficulty Quantification of Visual Tasks for AGI Evaluation

With the rapid development of multi-modal foundation models and the pursuit of artificial general intelligence (AGD),there is a growing need for corresponding evaluation systems. Systematic AGI evaluation requires tasks that encompass a widerange of ability dimensions and difficulty levels. However, although many benchmarks exist, the field still lacks a quantificationsystem to assess ability decompositions or dificulty levels. Here, we took the visual domain as a starting point and proposed anexplainable system for task ability decomposition and dificulty level quantification of vision (TADDL-V). Using large languagemodels, TADDL-V decomposed the visual abilities required for a given task and leveraged statistical data to map between abilitysets and task difficulty levels. The estimated ability masses align with human intuition, and TADDL-V’s task difficulty estimatesare empirically validated against aggregated human comparisons of task difficulty. Furthermore, we proposed an AGI visualevaluation task set, AGI-V70, comprising 70 composite visual tasks that incorporate visual abilities across a broad spectrum oftask difficulties. Together, TADDL-V serves as a prototype for ability decomposition and task difficulty level quantification, whichare essential for future AGI evaluations.

Science China Technological Sciences (JCR Q1, in press)