refactor-benchmark
openai-python
refactor-benchmark | openai-python | |
---|---|---|
2 | 61 | |
21 | 20,314 | |
- | 5.0% | |
5.9 | 9.6 | |
3 months ago | 7 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
refactor-benchmark
-
GPT-4 Turbo with Vision is a step backwards for coding
FWIW, I agree with you that each model has its own personality and that models may do better or worse on different kinds of coding tasks. Aider leans into both of these concepts.
The GPT-4 Turbo models have a lazy coding personality, and I spent months of effort figuring out how to both measure and reduce that laziness. This resulted in aider supporting "unified diffs" as a code editing format to reduce such laziness by 3X [0] and the aider refactoring benchmark as a way to quantify these benefits [1].
The benchmark results I just shared about GPT-4 Turbo with Vision cover both smaller, toy coding problems [2] as well as larger edits to larger source files [3]. The new model slightly underperforms on smaller coding tasks, and significantly underperforms on the larger edits where laziness is often a culprit.
[0] https://aider.chat/2023/12/21/unified-diffs.html
[1] https://github.com/paul-gauthier/refactor-benchmark
[2] https://aider.chat/2024/04/09/gpt-4-turbo.html#code-editing-...
[3] https://aider.chat/2024/04/09/gpt-4-turbo.html#lazy-coding
-
OpenAI: Memory and New Controls for ChatGPT
1-2 sentences: Rather than writing code, GPT-4 Turbo often inserts comments like "... finish implementing function here ...". I made a benchmark that provokes and quantifies that behavior.
1-2 paragraphs:
I found that I could provoke lazy coding by giving GPT-4 Turbo refactoring tasks, where I ask it to refactor a large method out of a large class. I analyzed 9 popular open source python repos and found 89 such methods that were conceptually easy to refactor, and built them into a benchmark [0].
GPT succeeds on a task if it can remove the method from its original class and add it to the top level of the file with appropriate changes to the SIZE of the abstract syntax tree. By measuring the size of the AST, we infer that GPT didn't replace a bunch of code with a comment like "... insert original method here...". I also gathered other laziness metrics like counting the number of new comments that contained "...", which correlated well with the AST size test.
[0] https://github.com/paul-gauthier/refactor-benchmark
openai-python
- The Stainless SDK Generator
-
GPT-4 Turbo with Vision is a step backwards for coding
The ongoing model anchoring/grounding issue likely affects all GPT-4 checkpoints/variants, but is most prominent with the latest "gpt-4-turbo-2024-04-09" variant due to its most recent cutoff date, might imply deeper issues with the current model architecture, or at least how it's been updated:
https://github.com/openai/openai-python/issues/1310
See also the original thread on OpenAI's developer forums (linked on the GitHub issue) with confirmations from others.
A test code snippet is included in the GitHub issue to A/B test the problem yourself with your own questions if need be.
-
Show HN: GritQL, a Rust CLI for rewriting source code
- Iterate on large codebases quickly: we use Rust for maximum performance
GritQL has already been used on thousands of repositories for complex migrations[1] but we're excited to collaborate more with the open source community.
[1] Ex. https://github.com/openai/openai-python/discussions/742
-
Integrating AI chatbotπ© into your application in 2024 π§π»ββοΈπ is how easy? π€
Explore OpenAI and learn how to easily integrate it into your existing application with the OpenAI Python module.
-
Need Help with OpenAI Upgrade: Seeking Guidance on 'openai.Completion' Deprecation Error
You tried to access openai.Completion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API. You can run `openai migrate` to automatically upgrade your codebase to use the 1.0.0 interface. Alternatively, you can pin your installation to the old version, e.g. `pip install openai==0.28` A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742
-
Anthropic announces Claude 2.1 β 200k context, less refusals
Relatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo.
This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e...
-
Web scraping experiment with AI (Parsing HTML with GPT-4)
Make sure to install the OpenAI library first. Since I'm using Python, I need
-
OpenAI: "We are dealing with periodic outages due to an abnormal traffic pattern reflective of a DDoS attack"
Source: https://github.com/openai/openai-python/discussions/742
- GitHub - openai/openai-python: The official Python library for the OpenAI API
- OpenAI Python v1 Released
What are some alternatives?
llama-cpp-python - Python bindings for llama.cpp
whisper.cpp - Port of OpenAI's Whisper model in C/C++
langchain - β‘ Building applications with LLMs through composability β‘ [Moved to: https://github.com/langchain-ai/langchain]
langchain - π¦π Build context-aware reasoning applications
openai-node - The official Node.js / Typescript library for the OpenAI API
ChatGPT-AutoExpert - ππ§ π¬ Supercharged Custom Instructions for ChatGPT (non-coding) and ChatGPT Advanced Data Analysis (coding).
sharegpt - Easily share permanent links to ChatGPT conversations with your friends
AutoGPT - AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
BetterChatGPT - An amazing UI for OpenAI's ChatGPT (Website + Windows + MacOS + Linux)
Awesome-LLMOps - An awesome & curated list of best LLMOps tools for developers
maelstrom - A workbench for writing toy implementations of distributed systems.
openai-cookbook - Examples and guides for using the OpenAI API