metagol VS ARC

In addition to what u/Logtalking answered you can also start out from the original Metagol implementation that Cropper and Muggleton provided, available here on github.

It's great for generating synthetic data for training LLMs for solving Abstraction & Reasoning Corpus (ARC) by François Chollet. Game of life helps the LLMs with a 2D understanding of the world.

https://github.com/fchollet/ARC

It's quite hard. You can download the dataset here [1] and it comes with a little webpage so that you can try it yourself.

It's worth noting that you are allowed to make three guesses.

[1]: https://github.com/fchollet/ARC

https://github.com/fchollet/ARC

The ARC 2 dataset is crowd sourced. If you can come up with a challenging task, then please contribute it.

In that case, per your your definition, there will always be edgecases. Take a look at ARC for an example of something that is easy for humans to do, but not yet doable by any AI at anywhere close to a human level. In respect to forecasting, I couldn't honestly say with any degree of confidence. Scaling larger transformer models may hit a roadblock, or it may work for everything, or there may be another development which changes the game. The soonest I think AGI that meets your definition will happen is 10 years from now, but I'm not confident in that prediction.

Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022.

https://twitter.com/fchollet/status/1631699463524986880

https://github.com/fchollet/ARC

Secondly, there are a few benchmarks that might actually be a good way to gauge the intelligence of AI. The Abstract and Reasoning Corpus attempts to measure actual intelligence. Turns out that LLMs have not actually improved their score on this test since 2019! Whether GPT-4 will be able to do a better job, specially when images can be used, remains to be seen. However, initial results are not very promising.

Source

neuro-symbolic systems where the neural network is tasked to _invent the system_. take for instance the ARC task ( https://github.com/fchollet/ARC ), when humans do these tasks (it appears to be the case that) we first invent a set of symbolic rules appropriate for the task at hand, then apply these rules

Current models are bad at using abstraction and reasoning to address new problems. They require training for each task, and performs poorly outside of the tasks they were trained on. Researchers are working on this but it's a hard problem - possibly the core of what it means to be "intelligent".