Test Bed for Abstraction and Reasoning

Subin Kim, Phunyaphibarn Prin, Donghyun Ahn, Sundong Kim

http://doi.org/10.5626/JOK.2024.51.1.59

Abstraction and Reasoning Corpus (ARC) proposed by François Chollet is a benchmark designed to develop generalizable intelligence, suitable for measuring cognitive abilities of both humans and computers. While most problems can be solved by humans, a computing-based ARC-Solver that can solve more than 30% of the problems is not yet known. In this study, a benchmark dataset, Mini-ARC, was introduced to simplify the model complexity while maintaining the difficulty level of the original ARC. To collect Mini-ARC, O2ARC was designed. It is an interface that can track the human solution process. A total of 3,000 solutions were collected from 25 people. This study proposed a new approach to developing a computing-based ARCSolver by constructing a system that could massively secure the simplified cognitive solution process. The Mini-ARC dataset can be found at https://github.com/ksb21ST/Mini-ARC.