Lark VS pydantic

Lark supports, and recommends, writing and storing the grammar in a .lark file. We have syntax highlighting support in all major IDEs, and even in github itself. For example, here is Lark's built-in grammar for Python: https://github.com/lark-parser/lark/blob/master/lark/grammar...

You can also test grammars "live" in our online IDE: https://www.lark-parser.org/ide/

The rationale is that it's more terse and has less visual clutter than a DSL over Python, which makes it easier to read and write.

First, let me introduce myself. My name is Erez. You may know some of the Python libraries I wrote in the past: Lark, Preql and Data-diff.

It's not super maintained but you might enjoy building something with ppci, Pure Python Compiler Infrastructure. It has some front-ends and some back-ends. There's also PeachPy for an assembler. People like using Lark for parsing, I hear.

lark, a parsing library where I am somewhat involved has a really nice solution to this: Rules starting with _ are inlined in a post processing step.

Lark is a good library to assist with this.

If I may ask, did you consider Lark, and if so, why wasn't it fit for your purposes?

A grammar provides the high level constructs you need to define the "shape" of your data, and it largely takes care of the rest. Grammar libraries exist in other language (eg. lark or Parsimonius in Python) and they weren't created just to make XML parsing easier.

I made a solid attempt at an Earley parser framework of my own, but apparently to get the most reliable performance from Earley parsing you need to implement Joop Leo's improvement for right-recursive grammars, which nobody has been able to adequately explain to me. I've read Kegler's open letter to Vaillant, I've tried to read other implementations, I've even tried to beat my head against the original academic paper, but I don't have the background knowledge to make sense of it all.

Pydantic is bundled with FastAPI and is excellent for modelling, validating, and serialising API responses.

Pydantic Settings is a python library that extends 🚀 Pydantic for dealing with settings management.

And here is where FastAPI ✨ shines: It leverages Pydantic powerful data validation to offer out of the box JSON Schema and OpenAPI specs via Swagger and Redoc.

If you carefully think about it, the system that extracts raw text from the PDF needs to both detect and render PDF form elements like checkboxes and radiobuttons in a way that LLMs can understand. In this example, we’ll use LLMWhisperer to extract PDF raw text representing checkboxes and radiobuttons. You can use LLMWhisperer completely free for processing up to 100 pages per day. As for structuring the output from LLMWhisperer, we’ll use GPT3.5-Turbo and we’ll use Langchain and Pydantic to help make our job easy.

Pydantic: Use Pydantic to declare your data model. This output parser allows users to specify an arbitrary Pydantic Model and query LLMs for outputs that conform to that schema.

1. Pydantic: Ultimate Data Validation

I'm really surprised ijl got angry that his mail was quoted, it looks innocent enough to me.

For reference it's been edited out here: https://github.com/pydantic/pydantic/issues/589

But github shows edits, so the edit is meaningless for privacy. Here's the original mail (yes, I'm blatantly ignoring his request to not publish this, I'm just this evil.)

    I've looked into replacing ujson in pydantic with orjson

First, note the method prefix_allowed_tokens_fn. This method applies a Pydantic model to constrain/guide how the LLM generates tokens. Next, see how that constrain can be applied to txtai's LLM pipeline.

utype is a concise alternative of pydantic with simplified parameters and usages, supporting both sync/async functions and generators parsing, and capable of using native logic operators to define logical types like AND/OR/NOT, also provides custom type parsing by register mechanism that supports libraries like pydantic, attrs and dataclasses