phasellm VS cuttlefish

If you're looking for an open source solution you can customize, check out the ResearchLLM demo: https://phasellm.com/researchllm

Code: https://github.com/wgryc/phasellm/tree/main/demos-and-produc...

Using Claude, Cohere, GPT-4, OpenAssistant. Formally swapping between them using PhaseLLM (open source library similar to LangChain).

Large language model evaluation and workflow framework from Phase AI. (https://github.com/wgryc/phasellm)

Fantastic questions! Re: working/not working at times -- this is still an issue. It's why I'm building PhaseLLM more broadly (https://github.com/wgryc/phasellm) -- need a robust pipeline that can also "reset" parts of itself if an LLM makes errors or mistakes.

You can see my prompts in this file: https://github.com/wgryc/phasellm/blob/main/demos-and-produc... I autogenerate a fairly big starting prompt and keep resubmitting it. It describes the data set extensively, which helps quite a bit.

That being said, a lot more can be done here around prompt optimization + making this more robust.

That being said, you might want to actually run head-to-head tests between models. PhaseLLM (free, open source) allows you to build a workflow and plug and play various models (including Dolly 2.0 and GPT-4). Then you can run tests to see how much worse/better the various LLMs are and if that's acceptable for your use case.

Large language model evaluation and workflow framework from Phase AI. (https://github.com/wgryc/phasellm)

I made something vaguely similar for your local terminal[0].

The idea is to give you a chat with an assistant that can use local tooling. Here it's Python for data analysis, I went more in the direction of "give it access to your terminal, so it can answer questions / do tasks on your local machine" which is something web-based options can't do right now.

I.e. ask it about your system details (processes, wifi) or to do things (configure something). Have it run the relevant commands, analyze the output, and respond either in natural language or i.e. a chart.

It's a really cool area of putting AI in a feedback loop (langchain-like), which I think is where the magic happens, and where we'll see much more happening in the future.

[0]:https://github.com/cube2222/cuttlefish