docquery
sqlparse
docquery | sqlparse | |
---|---|---|
4 | 7 | |
1,645 | 3,598 | |
0.5% | - | |
0.0 | 8.3 | |
about 1 year ago | 5 days ago | |
Python | Python | |
MIT License | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
docquery
-
Understanding HTML with Large Language Models
There is a visual demo here: https://sites.google.com/view/llm4html/home.
This work is very exciting for a few reasons:
* HTML is an incredibly rich source of visually structured information, with a semi-structured representation. This is as opposed to PDFs, which are usually fed into models with a "flat" representation (words + bounding boxes). Intuitively, this offers the model a more direct way to learn about nested structure, over an almost unlimited source of unsupervised pre-training data.
* Many projects (e.g. Pix2Struct https://arxiv.org/pdf/2210.03347.pdf, also from Google) operate on pixels, which are expensive (both to render and process in the transformer). Operating on HTML directly means smaller, faster, more efficient models.
* (If open sourced) it will be the first (AFAIK) open foundation model for the RPA/automation space (there are several closed projects). They claim they plan to open source the dataset at least, which is very exciting.
I'm particularly excited to extend this and similar (https://arxiv.org/abs/2110.08518) for HTML question answering and web scraping.
Disclaimer: I'm the CEO of Impira, which creates OSS (https://github.com/impira/docquery) and proprietary (http://impira.com/) tools for analyzing business documents. I am not affiliated with this project.
-
Pdfgrep – a commandline utility to search text in PDF files
DocQuery (https://github.com/impira/docquery), a project I work on, allows you to do something similar, but search over semantic information in the PDF files (using a large language model that is pre-trained to query business documents).
For example:
$ docquery scan "What is the due date?" /my/invoices/
-
This Week In Python
docquery – An easy way to extract information from documents
- DocQuery: Document Query Engine Powered by Natural Language Processing
sqlparse
-
Show HN: Databasediagram.com – Private, Text to Entity-Relationship Diagram Tool
Suggest checking out the sqlparse library for a way to do the different flavours without needing to address each case directly: https://github.com/andialbrecht/sqlparse
-
Data Load Diagram
Gotcha, since we haven't actually written all of this yet I don't have any useful code snippets to share but we've discussed tackling the problem internally using something like sqlparse. You'd need to identify the relevant sql chunks, parse them for table dependency information and then create the relevant entities in whichever data lineage tool you were using.
-
This Week In Python
sqlparse – A non-validating SQL parser module for Python
-
Open Source SQL Parsers
Regular expressions is a popular approach to extract information from SQL statements. However, regular expressions quickly become too complex to handle common features like WITH, sub-queries, windows clauses, aliases and quotes. sqlparse is a popular python package that uses regular expressions to parse SQL.
-
Automated SQL formatting checks
This one is not bad: https://github.com/andialbrecht/sqlparse.
- Let's write a compiler, part 5: A code generator
-
BigQuery Lineage
We used this repo for this: https://github.com/andialbrecht/sqlparse. I may have miscommunicated. We didn't write the parser from scratch, we created a way for the parser to detect downstream and upstream dependencies of the resource.
What are some alternatives?
pdfgrep
zetasql - ZetaSQL - Analyzer Framework for SQL
pdf-keywords-extractor
pyparsing - Python library for creating PEG parsers [Moved to: https://github.com/pyparsing/pyparsing]
natbot - Drive a browser with GPT-3
Lark - Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.
django-htmx-patterns - Pattern repository for Django and htmx with full example code
PLY - Python Lex-Yacc
django-functest - Helpers for creating high-level functional tests in Django, with a unified API for WebTest and Selenium tests.
sqlfluff - A modular SQL linter and auto-formatter with support for multiple dialects and templated code.
pdfgrep - PDFGrep is a GNU/Emacs module providing grep comparable facilities but for PDF files
Pygments