streamdeck-ohs VS supercharger

I asked it to browse this repository and tell me what I had to change to turn an ugly line into a pretty graph for example. I had spend 30 minutes looking back and forth (I don't know the plugin template nor do I touch js). If that's not being able to code then I don't know what is.

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). It isn't specified whether this is pass@1 or pass@10.

For comparison:

* GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons

* WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0

* The best open model I know of atm is replit-code-instruct-glaive, a replit-code-3b fine tune, which scores a 63.5% pass@1. An independent developer abacaj has reproduced that announcement as part of code-eval, a repo for getting human-eval results: https://github.com/abacaj/code-eval

Those interested in this area may also want to take a look at this repo https://github.com/my-other-github-account/llm-humaneval-ben... that also ranks with Eval+, the CanAiCode Leaderboard https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... and airate https://github.com/catid/supercharger/tree/main/airate

Also, as with all LLM evals, to be taken with a grain of salt...

Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv, June 12, 2023. https://doi.org/10.48550/arXiv.2305.01210.

Leverage locally-hosted Large Language Models to write software + unit tests (https://github.com/catid/supercharger)

supercharger Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism

Added support for 65B LLaMa model to https://github.com/catid/supercharger tonight. It runs faster than Baize 30B (maybe due to lack of adapter) and only slightly slower than Galpaca 30B. Benchmarks here: https://docs.google.com/spreadsheets/d/1TYBNr_UPJ7wCzJThuk5ysje7K1x-_62JhBeXDbmrjA8/edit?usp=sharing

Here's the code that loads it: https://github.com/catid/supercharger/blob/main/server/model_koala.py