human-eval
evalplus
| human-eval | evalplus | |
|---|---|---|
| 6 | 3 | |
| 2,081 | 978 | |
| 4.8% | 11.6% | |
| 0.0 | 9.3 | |
| 4 months ago | 12 days ago | |
| Python | Python | |
| MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
human-eval
-
Nathan Lambert review of LLAMA 2: Open-Source LLM from Meta
Code / math / reasoning: Not much discussion of code data in the paper and RLHF process. For instance, StarCoder at 15 billion parameters beats the best model at 40.8 for HumanEval and 49.5 MBPP (Python).
-
New ChatGPT rival, Claude 2, launches for open beta testing
In terms of coding capabilities, Claude 2 demonstrated a reported increase in proficiency. Its score on the Codex HumanEval, a Python programming test, rose from 56 percent to 71.2 percent. Similarly, on GSM8k, a test comprising grade-school math problems, it improved from 85.2 to 88 percent.
-
Hot Take: ChatGPT is not getting dumber, you are.
there are for code https://github.com/openai/human-eval
-
Official WizardCoder-15B-V1.0 Released! Can Achieve 59.8% Pass@1 on HumanEval!
❗Note: The above table conducts a comprehensive comparison of our WizardCoder with other models on the HumanEval and MBPP benchmarks. We adhere to the approach outlined in previous studies by generating 20 samples for each problem to estimate the pass@1 score and evaluate it with the same code. The scores of GPT4 and GPT3.5 reported by OpenAI are 67.0 and 48.1 (maybe these are the early version of GPT4&3.5).
- A new way to predict when software jobs will become automated
- OpenAI Codex - The Model behind GitHub Copilot
evalplus
-
The AI Reproducibility Crisis in GPT-3.5/GPT-4 Research
*Further Reading*:
- [GPT-4's decline over time (HackerNews)](https://news.ycombinator.com/item?id=36786407)
- [GPT-4 downgrade discussions (OpenAI Forums)](https://community.openai.com/t/gpt-4-has-been-severely-downg...)
- [Behavioral changes in ChatGPT (arXiv)](https://arxiv.org/abs/2307.09009)
- [Zero-Shot Replication Effort (Github)](https://github.com/emrgnt-cmplxty/zero-shot-replication)
- [Inconsistencies in GPT-4 HumanEval (Github)](https://github.com/evalplus/evalplus/issues/15)
- [Early experiments with GPT-4 (arXiv)](https://arxiv.org/abs/2303.12712)
- [GPT-4 Technical Report (arXiv)](https://arxiv.org/abs/2303.08774)
-
Official WizardCoder-15B-V1.0 Released! Can Achieve 59.8% Pass@1 on HumanEval!
❗Note: In this study, we copy the scores for HumanEval and HumanEval+ from the LLM-Humaneval-Benchmarks. Notably, all the mentioned models generate code solutions for each problem utilizing a single attempt, and the resulting pass rate percentage is reported. Our WizardCoder generates answers using greedy decoding and tests with the same code.
What are some alternatives?
chat-with-gpt - An open-source ChatGPT app with a voice
gpt_academic - 为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss等。
llm-humaneval-benchmarks
llm_oracle - LLM Oracle is a GPT-4 powered tool for predicting future events. It's like a Magic 8 Ball that is able to perform basic research, calculations, and reasoning.
ggml - Tensor library for machine learning
zero-shot-replication
WizardLM - Family of instruction-following LLMs powered by Evol-Instruct: WizardLM, WizardCoder and WizardMath
Baichuan-13B - A 13B large language model developed by Baichuan Intelligent Technology
llama - Inference code for Llama models
chatgpt_academic - 为GPT/GLM提供图形交互界面，特别优化论文阅读润色体验，模块化设计支持自定义快捷按钮&函数插件，支持代码块表格显示，Tex公式双显示，新增Python和C++项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持清华chatglm等本地模型 [Moved to: https://github.com/binary-husky/gpt_academic]