phasellm VS supercharger

If you're looking for an open source solution you can customize, check out the ResearchLLM demo: https://phasellm.com/researchllm

Code: https://github.com/wgryc/phasellm/tree/main/demos-and-produc...

Using Claude, Cohere, GPT-4, OpenAssistant. Formally swapping between them using PhaseLLM (open source library similar to LangChain).

Large language model evaluation and workflow framework from Phase AI. (https://github.com/wgryc/phasellm)

Fantastic questions! Re: working/not working at times -- this is still an issue. It's why I'm building PhaseLLM more broadly (https://github.com/wgryc/phasellm) -- need a robust pipeline that can also "reset" parts of itself if an LLM makes errors or mistakes.

You can see my prompts in this file: https://github.com/wgryc/phasellm/blob/main/demos-and-produc... I autogenerate a fairly big starting prompt and keep resubmitting it. It describes the data set extensively, which helps quite a bit.

That being said, a lot more can be done here around prompt optimization + making this more robust.

That being said, you might want to actually run head-to-head tests between models. PhaseLLM (free, open source) allows you to build a workflow and plug and play various models (including Dolly 2.0 and GPT-4). Then you can run tests to see how much worse/better the various LLMs are and if that's acceptable for your use case.

Large language model evaluation and workflow framework from Phase AI. (https://github.com/wgryc/phasellm)

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). It isn't specified whether this is pass@1 or pass@10.

For comparison:

* GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons

* WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0

* The best open model I know of atm is replit-code-instruct-glaive, a replit-code-3b fine tune, which scores a 63.5% pass@1. An independent developer abacaj has reproduced that announcement as part of code-eval, a repo for getting human-eval results: https://github.com/abacaj/code-eval

Those interested in this area may also want to take a look at this repo https://github.com/my-other-github-account/llm-humaneval-ben... that also ranks with Eval+, the CanAiCode Leaderboard https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... and airate https://github.com/catid/supercharger/tree/main/airate

Also, as with all LLM evals, to be taken with a grain of salt...

Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv, June 12, 2023. https://doi.org/10.48550/arXiv.2305.01210.

Leverage locally-hosted Large Language Models to write software + unit tests (https://github.com/catid/supercharger)

supercharger Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism

Added support for 65B LLaMa model to https://github.com/catid/supercharger tonight. It runs faster than Baize 30B (maybe due to lack of adapter) and only slightly slower than Galpaca 30B. Benchmarks here: https://docs.google.com/spreadsheets/d/1TYBNr_UPJ7wCzJThuk5ysje7K1x-_62JhBeXDbmrjA8/edit?usp=sharing

Here's the code that loads it: https://github.com/catid/supercharger/blob/main/server/model_koala.py