ARC VS langchain

It's great for generating synthetic data for training LLMs for solving Abstraction & Reasoning Corpus (ARC) by François Chollet. Game of life helps the LLMs with a 2D understanding of the world.

https://github.com/fchollet/ARC

It's quite hard. You can download the dataset here [1] and it comes with a little webpage so that you can try it yourself.

It's worth noting that you are allowed to make three guesses.

[1]: https://github.com/fchollet/ARC

https://github.com/fchollet/ARC

The ARC 2 dataset is crowd sourced. If you can come up with a challenging task, then please contribute it.

In that case, per your your definition, there will always be edgecases. Take a look at ARC for an example of something that is easy for humans to do, but not yet doable by any AI at anywhere close to a human level. In respect to forecasting, I couldn't honestly say with any degree of confidence. Scaling larger transformer models may hit a roadblock, or it may work for everything, or there may be another development which changes the game. The soonest I think AGI that meets your definition will happen is 10 years from now, but I'm not confident in that prediction.

Yes, François Chollet released ARC(Abstraction and Reasoning Corpus) benchmark for this in 2019, and the benchmark can be scored automatically. Humans solve 100% of tests and GPTs solve 0% of tests and GPTs made exactly zero progress from 2019 to 2022.

https://twitter.com/fchollet/status/1631699463524986880

https://github.com/fchollet/ARC

Secondly, there are a few benchmarks that might actually be a good way to gauge the intelligence of AI. The Abstract and Reasoning Corpus attempts to measure actual intelligence. Turns out that LLMs have not actually improved their score on this test since 2019! Whether GPT-4 will be able to do a better job, specially when images can be used, remains to be seen. However, initial results are not very promising.

Source

neuro-symbolic systems where the neural network is tasked to _invent the system_. take for instance the ARC task ( https://github.com/fchollet/ARC ), when humans do these tasks (it appears to be the case that) we first invent a set of symbolic rules appropriate for the task at hand, then apply these rules

Current models are bad at using abstraction and reasoning to address new problems. They require training for each task, and performs poorly outside of the tasks they were trained on. Researchers are working on this but it's a hard problem - possibly the core of what it means to be "intelligent".

Langchain and the implementation of Custom Tools also is a great (and very efficient) way to setup a dedicated Q&A (for example for chat purpose) agent.

We're using it via langchain talking to Amazon Bedrock which is hosting Claude 1.x. It's comparable to GPT3.x, not bad. The integration doesn't seem to be fully there though, I think langchain is expecting "Human:" and "AI:", but Claude uses "Assistant:".

https://github.com/hwchase17/langchain/issues/2638

Depending on how much work you want to put into it, you can get started at HuggingFace with their models and datasets, but you'd need compute power, multiple MLOps, etc. I was introduced to the concept in this video, since Google has their Vertex AI tools on Google Cloud, and there's always LangChain but I'm not sure about anything recent.

a documentation chatbot proof of concept using GPT3.5 and LangChain

I encourage everyone to pay attention to the Langchain open-source project and leverage it to achieve tasks that ChatGPT cannot handle.

Yeah I never know where memory goes exactly in langchain, it's not exactly clear all the time. But sure, the main insight I remember is this, take a look at their MULTI_PROMPT_ROUTER_TEMPLATE: https://github.com/hwchase17/langchain/blob/560c4dfc98287da1...

It's a lot of instructions for an LLM, they seem to forget an LLM is an auto-completion machine, and which data it is trained on. Using <<>> for sections is not a normal thing, it's not markdown, which probably the thing read way more often on the internet, instead of open json comments, why not type signatures, instead of so many rules, why not give it examples? It is an autocomplete machine!

They are relying too much on the LLM being smart because they probably only test stuff in GPT-4 and 3.5, but with GPT4All models this prompt was not working at all, so I had to rewrite it, for simple routing, we don't even need json, carying the `next_inputs` here is weird if you don't need it.

So this is my version of it: https://gist.github.com/rogeriochaves/b67676977eebb1936b9b5c...

It's so basic it's dumb, yet it is more powerful, as it does not rely on GPT-4 level intelligence, it's just what I needed