openrouter-runner VS openai-cookbook

I've bumped into a few of these. I use https://openrouter.ai as a model abstraction, but not as a router. https://withmartian.com does the same thing but with a more enterprise feel. Also https://www.braintrustdata.com/ though it's less clear how committed they are to that feature.

That said, while I've really enjoyed the LLM abstraction (making it easy for me to test different models without changing my code), I haven't felt any desire for a router. I _do_ have some prompts that I send to gpt-3.5-turbo, and could potentially use other models, but it's kind of niche.

In part this is because I try to do as much in a single prompt as I can, meaning I want to use a model that's able to handle the hardest parts of the prompt and then the easy parts come along with. As a result there's not many "easy" prompts. The easy prompts are usually text fixup and routing.

My "routing" prompts are at a different level of abstraction, usually routing some input or activity to one of several prompts (each of which has its own context, and the sum of all contexts across those prompts is too large, hence the routing). I don't know if there's some meaningful crossover between these two routing concepts.

Another issue I have with LLM portability is the use of tools/functions/structured output. Opus and Gemini Pro 1.5 have kind of implemented this OK, but until recently GPT was the only halfway decent implementation of this. This seems to be an "advanced" feature, yet it's also a feature I use even more with smaller prompts, as those small prompts are often inside some larger algorithm and I don't want the fuss of text parsing and exceptions from ad hoc output.

But in the end I'm not price sensitive in my work, so I always come back to the newest GPT model. If I make a switch to Opus it definitely won't be to save money! And I'm probably not going to want to fiddle, but instead make a thoughtful choice and switch the default model in my code.

Why not use something like http://openrouter.ai? Pay as you go and you can select any model you want. Heaven!

teknium / Nous released Mistral finetunes (Hermes) that are quite great, and even published the datasets used for training.

But for the worldsim I think they are really using Claude (probably Haiku or Sonnet) via openrouter (https://openrouter.ai/).

Not affiliated with the project but you could use something like OpenRouter to give users a massive list of models to choose from with fairly minimal effort

https://openrouter.ai/

> I hate that they require a phone number

https://openrouter.ai/ lets you make one account and get API access to a bunch of different models. They also provide access to hosted versions of a bunch of open models.

Useful if you want to compare 15 different models without bothering to create 15 different accounts or download 15 x 20GB of models :)

You sure can! NeuroEngine[1] hosts some nice free demos of what are basically the state of the art in unfiltered models, and if you need API access, OpenRouter[2] has dozens of unfiltered models to choose from.

[1] https://www.neuroengine.ai/

[2] https://openrouter.ai/

However, this needs to be changed as other providers like OpenRouter can also start supporting this feature in the future.

Great resources for me are - The LangChain Blog: Very technical but great graphics of complex topics. Gives you a good understanding of what is currently possible and what the hot topics are - Product Hunt: Great resource to see what others are building with AI - Replicate and OpenRouter for custom made / fine tuned models - AI Twitter

One of my personal favorite OpenAI resources these days is the OpenAI Cookbook: this is an excellent way to start learning how their different models work, how to start taking advantage of the many cutting edge techniques in the AI space, and how to start integrating your data with AI workloads.

A pretrained LLM is a closed-book system: It can only access information that it was trained on. With domain fine-tuning, the system manifests additional material. An early prototype of this technique was shown in this OpenAi cookbook: For the target domain, text was embedded using an API, and then when using the LLM, embeddings were retrieved using semantic similarity search to formulate an answer. Although this approach evolved to retrieval-augmented generation, its still a technique to adapt a Gen2 (2020) or Gen3 (2022) LLM into a question-answering system.

https://github.com/openai/openai-cookbook/blob/main/examples...

It sounds like you are trying to tweak embeddings for similarity search. Rather than fine-tune the model's layers, you may want to try training a linear transformation the existing model's output embedding. Openai has a cookbook on how to do that. You will need some data though - but I think you can try it with ~20 pieces of synthetically generated data.

tutorial from OpenAI https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb

OpenAI Cookbook