reflexion VS monitors4codegen

In terms of defining whether it is a lot better than gpt 3.5 (and it’s different models) I recommend reading the technical report: https://arxiv.org/pdf/2303.08774.pdf or the paper on “self reflection” as it’s quite interesting and allowed it to perform quite well on some parameters https://arxiv.org/abs/2303.11366 or the code for the human-eval test https://github.com/GammaTauAI/reflexion-human-eval

One of the things I've found is that at least with GPT-4, you can automate a lot of infrastructure. This can make it much easier to get started with new project where you need to set up various servers, containers, VMs etc. I haven't fully automated it yet, but I've done it enough manually through copy/paste between ChatGPT and the terminal to see it that would work if it was hooked up to langchain. You can create prompts to make it set up VMs, docker containers and then install and configure all kinds of software.

To make it work in most cases it should follow the same stuff as this implementation of a Reflexion agent for SOTA Human-Eval Python results, but use it for infrastructure. GPT-4 can generally figure out how to create tests from the documentation that will make it know when the task it finished correctly.

https://github.com/GammaTauAI/reflexion-human-eval

Spin off project based on reflection, apparently GPT-4 gets 20% improvement in coding tasks: https://github.com/GammaTauAI/reflexion-human-eval

You might also be interested in the "Reflexion" paper that's been doing the rounds in the past few days: https://github.com/GammaTauAI/reflexion-human-eval