ARC VS ARC-Game

It's great for generating synthetic data for training LLMs for solving Abstraction & Reasoning Corpus (ARC) by François Chollet. Game of life helps the LLMs with a 2D understanding of the world.

https://github.com/fchollet/ARC

It's quite hard. You can download the dataset here [1] and it comes with a little webpage so that you can try it yourself.

It's worth noting that you are allowed to make three guesses.

[1]: https://github.com/fchollet/ARC

https://github.com/fchollet/ARC

The ARC 2 dataset is crowd sourced. If you can come up with a challenging task, then please contribute it.

In that case, per your your definition, there will always be edgecases. Take a look at ARC for an example of something that is easy for humans to do, but not yet doable by any AI at anywhere close to a human level. In respect to forecasting, I couldn't honestly say with any degree of confidence. Scaling larger transformer models may hit a roadblock, or it may work for everything, or there may be another development which changes the game. The soonest I think AGI that meets your definition will happen is 10 years from now, but I'm not confident in that prediction.

Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022.

https://twitter.com/fchollet/status/1631699463524986880

https://github.com/fchollet/ARC

Secondly, there are a few benchmarks that might actually be a good way to gauge the intelligence of AI. The Abstract and Reasoning Corpus attempts to measure actual intelligence. Turns out that LLMs have not actually improved their score on this test since 2019! Whether GPT-4 will be able to do a better job, specially when images can be used, remains to be seen. However, initial results are not very promising.

Source

neuro-symbolic systems where the neural network is tasked to _invent the system_. take for instance the ARC task ( https://github.com/fchollet/ARC ), when humans do these tasks (it appears to be the case that) we first invent a set of symbolic rules appropriate for the task at hand, then apply these rules

Current models are bad at using abstraction and reasoning to address new problems. They require training for each task, and performs poorly outside of the tasks they were trained on. Researchers are working on this but it's a hard problem - possibly the core of what it means to be "intelligent".

I made a web-game named "The ARC Game" that aims to provide easy to use UI for ARC dataset from "The Measure of Intelligence" paper by François Chollet. The game consist of 800 tasks split into several groups by its difficulty. Each level consist of several grid pairs describing transitions from input to output grids. The aim of the player is to find some logic behind it and apply it to test grids. The game stores progress locally within your browser. It made mostly for kids, but anyone can play it nevertheless. I think that game like this might be a useful tool for improving and measuring children's abstraction and reasoning skills. Link: https://github.com/volotat/ARC-Game

Here is the game: https://volotat.github.io/ARC-Game/And source code: https://github.com/volotat/ARC-Game