Selefra VS supercharger

[Feature]Modules support filtering, while labels support customization of any format. by @FelixsJiang in #30

Selefra Project Repository: github.com/selefra/selefra

Analyze cloud resources using GPT (https://github.com/selefra/selefra)

Selefra: Selefra is an open-source Policy as Code tool that can use natural language to write rules for security compliance checks, cost configuration checks, and architecture rationality checks on current cloud services.

GitHub: https://github.com/selefra/selefra

Website: https://www.selefra.io/

Here, we strongly encourage you to try Selefra and enjoy a faster and more efficient cloud security analysis and resolution experience. You can find more information about Selefra on our official website (https://www.selefra.io/) or GitHub (https://github.com/selefra/selefra), or follow our Twitter account (https://twitter.com/SelefraCorp) for more real-time updates.

Check out and star GitHub: https://github.com/selefra/selefra

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). It isn't specified whether this is pass@1 or pass@10.

For comparison:

* GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons

* WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0

* The best open model I know of atm is replit-code-instruct-glaive, a replit-code-3b fine tune, which scores a 63.5% pass@1. An independent developer abacaj has reproduced that announcement as part of code-eval, a repo for getting human-eval results: https://github.com/abacaj/code-eval

Those interested in this area may also want to take a look at this repo https://github.com/my-other-github-account/llm-humaneval-ben... that also ranks with Eval+, the CanAiCode Leaderboard https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul... and airate https://github.com/catid/supercharger/tree/main/airate

Also, as with all LLM evals, to be taken with a grain of salt...

Liu, Jiawei, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. “Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.” arXiv, June 12, 2023. https://doi.org/10.48550/arXiv.2305.01210.

Leverage locally-hosted Large Language Models to write software + unit tests (https://github.com/catid/supercharger)

supercharger Write Software + unit tests for you, based on Baize-30B 8bit, using model parallelism

Added support for 65B LLaMa model to https://github.com/catid/supercharger tonight. It runs faster than Baize 30B (maybe due to lack of adapter) and only slightly slower than Galpaca 30B. Benchmarks here: https://docs.google.com/spreadsheets/d/1TYBNr_UPJ7wCzJThuk5ysje7K1x-_62JhBeXDbmrjA8/edit?usp=sharing

Here's the code that loads it: https://github.com/catid/supercharger/blob/main/server/model_koala.py