supercharger VS vocode-python

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). It isn't specified whether this is pass@1 or pass@10.

For comparison:

* GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons

* WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0

* The best open model I know of atm is replit-code-instruct-glaive, a replit-code-3b fine tune, which scores a 63.5% pass@1. An independent developer abacaj has reproduced that announcement as part of code-eval, a repo for getting human-eval results: https://github.com/abacaj/code-eval

Those interested in this area may also want to take a look at this repo https://github.com/my-other-github-account/llm-humaneval-ben... that also ranks with Eval+, the CanAiCode Leaderboard https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... and airate https://github.com/catid/supercharger/tree/main/airate

Also, as with all LLM evals, to be taken with a grain of salt...

Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv, June 12, 2023. https://doi.org/10.48550/arXiv.2305.01210.

Leverage locally-hosted Large Language Models to write software + unit tests (https://github.com/catid/supercharger)

supercharger Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism

Added support for 65B LLaMa model to https://github.com/catid/supercharger tonight. It runs faster than Baize 30B (maybe due to lack of adapter) and only slightly slower than Galpaca 30B. Benchmarks here: https://docs.google.com/spreadsheets/d/1TYBNr_UPJ7wCzJThuk5ysje7K1x-_62JhBeXDbmrjA8/edit?usp=sharing

Here's the code that loads it: https://github.com/catid/supercharger/blob/main/server/model_koala.py

Vocode || Engineering (multiple roles) || SF/Remote || Full-time/Contract || https://vocode.dev

Vocode–an open source library for building LLM applications you can talk to. (https://github.com/vocodedev/vocode-python)

Coqui also looks interesting.

https://github.com/coqui-ai/TTS

Support for it was recently added to vocode:

https://github.com/vocodedev/vocode-python/pull/56

Direct link to the code: https://github.com/vocodedev/vocode-python

Hey everyone! We are so grateful for the warm reception from our launch this week.

We're back with PrankGPT (origin story of Vocode), rebuilt using our library https://github.com/vocodedev/vocode-python

Source code for the backend is public and available on replit to check it out

Given that Vocode (realtime audio, llm, etc) came out a few days ago, could you speak to how yours compares to it?

https://github.com/vocodedev/vocode-python

It's a demo for their new open source library integrating several AI tools: https://github.com/vocodedev/vocode-python