Our great sponsors
-
simpleaichat
Python package for easily interfacing with chat apps, with robust features and minimal code complexity.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
I've been spending a lot of time figuring out how to make these GPT models successful at *editing* existing code. This is more difficult than having them write brand new code.
So far, GPT-4 has been significantly better at editing code than GPT-3.5 Turbo. This is for two reasons [1]:
1. GPT-4's bigger context window. The new 16k window for 3.5 solves this problem.
2. GPT-4 is much better at following instructions about how to format the edits into a diff-like output format. I previously shared notes on a large number of code edit output formats that I tried with GPT-3.5 Turbo [2].
Essentially, 3.5 wasn't capable of outputting any sort of diff-based edit format. The only thing it could reliably do was send back *all* of the code (say an entire source file) with the changes included. On the other hand, GPT-4 is capable of reliably outputting (some) diff-like formats.
So I'm very curious to see if the new fine-tuned 3.5 is more capable at editing code. I'll be running some benchmarks to find out!
[1] https://github.com/paul-gauthier/aider#gpt-4-vs-gpt-35
[2] https://news.ycombinator.com/item?id=35441666
The big feature here is the function calls, as this is effectively a replacement for the "Tools" feature of Agents popularized by LangChain, except in theory much more efficient since it may not require an extra call to the API. In the case of LangChain which selects Tools and their outputs through trying to hack out Markdown (which often fails and causes ParsingErrors), this variant of ChatGPT appears to be finetuned for it so perhaps it'll be more reliable.
While developing a more-simple LangChain competitor (https://github.com/minimaxir/simpleaichat) I discovered a neat trick for allowing ChatGPT to select tools from a list reliably: put the list of tools into a numbered list, and force the model to return only a single number by using the logit_bias parameter: https://github.com/minimaxir/simpleaichat/blob/main/PROMPTS....
The slight price drop for ChatGPT inputs is of course welcome, since inputs are the bulk of the costs for longer conversations. A 4x context window at 2x the price is a good value too.
As they are accepting a JSON schema for the function calls, it is likely they are using token biasing based on the schema (using some kind of state machine that follows along with the tokens and only allows the next token to be a valid one given the grammar/schema). I have successfully implemented this for JSON Schema (limited subset) on llama.cpp. See also e.g. this implementation: https://github.com/1rgs/jsonformer
- The API is extremely cheap
- There are plenty of open source tools built on top if it (example list: https://github.com/heartly/awesome-writing-tools)
While I wish this work was open, they are both the best and cheapest option out there... by a mile.
You can get GPT 4 access by submitting an eval if gets merged (https://github.com/openai/evals). Here's the one that got me access[1]
Although from the blog post it looks like they're planning to open up to everyone soon, so that may happen before you get through the evals backlog.
1: https://github.com/openai/evals/pull/778
Thank you for sharing this. I’ve been working on something for editing exiting code by using GitHub Issues and Actions for prompting, with the response from GPT as a Pull Request/Comment on the Issue[1]. I will definitely try out your project!
[1] https://github.com/busse/kodumisto
