synquid VS outlines

https://www.youtube.com/watch?v=HnOix9TFy1A

Links to more projects and papers by Prof. Polikarpova: https://cseweb.ucsd.edu/~npolikarpova/

I think this is one of the main ones she discusses in the talk: https://github.com/nadia-polikarpova/synquid

My first post was poorly written. I didn't mean to imply that taking pride in a job well done was bad. I don't think it is. I think programmers have plenty of reason to be happy when they do good work[1]--what I think is foolish is having visions of grandeur when it comes to programming, which in my opinion is what _why seemed to have based on his reasoning for quitting. I could be way off mark, but that's my take.

I mentioned this in another comment, but I think it also has to do with a confusion of categories. _why seemed to want recognition akin to that received by, e.g. Thomas Bernhard, Kafka, for something like shoes or his other software/libraries or general contributions to computing. But the issue is these things will always be utilities for specialists, and any aesthetic properties they have (elegant design, expression, etc.) are secondary to their functioning and they'll always be relegated to the dusty realm of specialists since the code is not the product--the software is. One can write code to create an aesthetic object that is enjoyed and revered by the masses, but I have a hard time envisioning a future in which the masses will ever enjoy and revere code or engineering for its own sake.

Pride was the wrong word to use and one I lazily reached for. After reading your comment, you've helped me realize that what I advise against is misapplication of expectations to different categories of things. _why seemed to want an aesthetic reception and legacy on a general, popular scale for work that is ultimately only a utility to the vast majority of the population and indeed, not even accessible to the population, and even if it were, I don't think many people would admire programming libs for fun--such a hobby will remain the lot of only enthusiasts. There is no pop coding like there is pop music.

[1]: Though I'd also argue that much of what you state taking pride in is not programming--which is just expressing ideas in programming languages--what you are talking about is engineering/design, which can be done perfectly well and separately from the programming part. we just happen to solve a lot of problems with computers these days so we need to express solutions for computers to consume and we tend to blend those responsibilities (we'll one day get to a point where the computers do most of the programming and we just design https://github.com/nadia-polikarpova/synquid)

Synquid synthesizes programs from refinement types, which are very similar in that you express a type-level predicate on the output using an expression which involves the input.

[4] https://github.com/outlines-dev/outlines/tree/main

The next step is defining how to guide generation. For this step, we'll use the Outlines library. Outlines is a library for controlling how tokens are generated. It applies logic to enforce schemas, regular expressions and/or specific output formats such as JSON.

No benchmarks, just my anecdotal experience trying to get local LLM's to respond with JSON. The method above works for my use case nearly 100% of the time. Other things I've tried (e.g. `outlines`[0]) are really slow or don't work at all. Would love to hear what others have tried!

0 - https://github.com/outlines-dev/outlines

As I was playing with the Outlines library (https://outlines-dev.github.io/outlines/), I discussed with my friend Maxime how funny it would be if we set up a way to pair LLMs in chess matches till one wins. The first time I tried it, it required substantial prompt engineering to get some of those LLMs to propose valid moves. Large language models can mostly stay focused and even play rather well; see https://news.ycombinator.com/item?id=37616170 for example. However small language models aren't as easy to convince.

Some of those LLMs have seen very little chess notation and so after the first few opening moves there aren't any valid tactics, let alone strategy, so they would end up either repeating the same move, or hallucinate moves that are not valid (Kxe5, but there would be a queen on e5!)

Then Outlines came along and we could force them to pick valid moves with little cost! Maxime worked super fast and got a first version of this idea as a gradio space.

I think it is pretty fun to see the (mostly terrible, but otherwise valid) chess that those LLMs play. Maybe it will even be instructive to how we can create small LLMs that can play much better than the ones on the leaderboard.

Anyway, you can check it out here:

https://huggingface.co/spaces/mlabonne/chessllm

What is interactive about it: you can pick the LLMs from available models on HuggingFace (within reason, small LLMs are preferable so that the space does not crash) or push one of your own small models to HF and have it fight with others. At the end of the game the leaderboard is updated.

Hope you find it fun!

> The most obvious usage of this is forcing a model to output valid JSON

Isn't this something that Outlines [0], Guidance [1] and others [2] already solve much more elegantly?

0. https://github.com/outlines-dev/outlines

1. https://github.com/guidance-ai/guidance

2. https://github.com/sgl-project/sglang

And I think “just” Markdown doesn’t quite cut it for safe guidance. For example: directly generating content for your components. But I’m really excited about tooling like outlines appearing, with a greater focus on guided generation for structured data. Because this is often what we actually need!

It's pretty easy to force a locally running model to always output valid JSON: when it gives you probabilities for the next tokens, discard all tokens that would result in invalid JSON at that point (basically reverse parsing), and then apply the usual techniques to pick the completion only from the remaining tokens. You can even validate against a JSON schema that way, so long as it is simple enough.

There are a bunch of libraries for this already, e.g.: https://github.com/outlines-dev/outlines

https://github.com/outlines-dev/outlines/blob/7fae436345e621... squares with my experience using LLMs for anything real

  sequence = generator("Alice had 4 apples and Bob ate 2. Write an expression for Alice's apples:")

There are "smaller" models, for example tinyllama 1.1B (tiny seems like an exaggeration). PHI2 is 2.7B parameters. I can't name a 500M parameter model but there is probably one.

The problem is they are all still broadly trained and so they end up being Jack of all trades master of none. You'd have to fine tune them if you want them good at some narrow task and other than code completion I don't know that anyone has done that.

If you want to generate json or other structured output, there is Outlines https://github.com/outlines-dev/outlines that constrains the output to match a regex so it guarantees e.g. the model will generate a valid API call, although it could still be nonsense if the model doesn't understand, it will just match the regex. There are other similar tools around.