spider VS LLMStack

Hey all - I'm currently working on a self-service, natural language BI tool that aims to go beyond the base "text to sql" of current tools. I've got the bones built, but I'm struggling to develop a suite of test questions (ideally with complex metrics like "What is our profitability" or abstract concepts like "How are our sales doing"). Does anyone know of any lists of questions (and ideally answers on the quantitative questions) for the MS adventureworks database, or any other complex (30+ table) public test databases? I've looked at Spider, but most of the datasets are too small to simulate real-world business datasets, and the questions are more "can you write fancy SQL" and less "can you answer a vague stakeholder question on unknown data".

Hi HN community. We are excited to open source Dataherald’s natural-language-to-SQL engine today (https://github.com/Dataherald/dataherald). This engine allows you to set up an API from your structured database that can answer questions in plain English.

GPT-4 class LLMs have gotten remarkably good at writing SQL. However, out-of-the-box LLMs and existing frameworks would not work with our own structured data at a necessary quality level. For example, given the question “what was the average rent in Los Angeles in May 2023?” a reasonable human would either assume the question is about Los Angeles, CA or would confirm the state with the question asker in a follow up. However, an LLM translates this to:

select price from rent_prices where city=”Los Angeles” AND month=”05” AND year=”2023”

This pulls data for Los Angeles, CA and Los Angeles, TX without getting columns to differentiate between the two. You can read more about the challenges of enterprise-level text-to-SQL in this blog post I wrote on the topic: https://medium.com/dataherald/why-enterprise-natural-languag...

Dataherald comes with “batteries-included.” It has best-in-class implementations of core components, including, but not limited to: a state of the art NL-to-SQL agent, an LLM-based SQL-accuracy evaluator. The architecture is modular, allowing these components to be easily replaced. It’s easy to set up and use with major data warehouses.

There is a “Context Store” where information (NL2SQL examples, schemas and table descriptions) is used for the LLM prompts to make the engine get better with usage. And we even made it fast!

This version allows you to easily connect to PG, Databricks, BigQuery or Snowflake and set up an API for semantic interactions with your structured data. You can then add business and data context that are used for few-shot prompting by the engine.

The NL-to-SQL agent in this open source release was developed by our own Mohammadreza Pourreza, whose DIN-SQL algorithm is currently top of the Spider (https://yale-lily.github.io/spider) and Bird (https://bird-bench.github.io/) NL 2 SQL benchmarks. This agent has outperformed the Langchain SQLAgent anywhere from 12%-250%.5x (depending on the provided context) in our own internal benchmarking while being only ~15s slower on average.

Needless to say, this is an early release and the codebase is under swift development. We would love for you to try it out and give us your feedback! And if you are interested in contributing, we’d love to hear from you!

This is an active field of research. You might want to look at the main challenge dataset for it : https://yale-lily.github.io/spider. It would be interesting to use ChatGPT's model as a pre-processor and then feed its output in to a more finetuned model like PICARD.

We tried using OpenAI/Davinci for SQL query authoring, but it quickly became obvious that we are still really far from something the business could find value in. The state of the art as described below is nowhere near where we would need it to be:

https://yale-lily.github.io/spider

https://arxiv.org/abs/2109.05093

https://github.com/ElementAI/picard

To be clear, we haven't tried this on actual source code (i.e. procedural concerns), so I feel like this is a slightly different battle.

The biggest challenge I see is that the queries we would need the most assistance with are the same ones that are the rarest to come by in terms of training data. They are also incredibly specific in the edge cases, many time requiring subjective evaluation criteria to produce an acceptable outcome (i.e. recursive query vs 5k lines of unrolled garbage).

I've been looking for databases with real-world schema and faker data (eg 10,000 entries of fake users) to test my natural langaugae to SQL generative model, as well as the efficiency of the generated queries

The cloest thing I can find is annotated dataset like Spider (https://yale-lily.github.io/spider) but after digging more into it, it's not as real-world-ish as I've hoped for.

Are there any SaSS, paid services, etc, where I can have access databases with complex real-world(-ish) schemas (populated with real-world-ish data)?

Thanks!

There are projects out there that do this.

Possibly relevant: https://yale-lily.github.io/spider

I briefly worked on a startup to commercialize this tech, but we decided it wasn't accurate enough to be useful. It was very cool when it actually worked. If you can only produce what you want half the time on simple queries, that doesn't seem very useful to me though.

There is very less description available in the Spider dataset research paper. Which says that these files are used for value specific queries. Does that means gold.sql files should only contain queries with value check (for example: SELECT * FROM table WHERE student_name = 'Student_A'). If that's the case there are many instances in gold files without actual values from the dataset (for example: SELECT COUNT(*) FROM table). Thanks.

We have recently added support to query data from SingleStore to our agent framework, LLMStack (https://github.com/trypromptly/LLMStack). Out of the box performance performance when prompting with just the table schemas is pretty good with GPT-4.

The more domain specific knowledge needed for queries, the harder it has gotten in general. We've had good success `teaching` the model different concepts in relation to the dataset and giving it example questions and queries greatly improved performance.

This will hopefully improve the startup times for FFmpeg when streaming from virtual display buffers. We use FFmpeg in LLMStack (low-code framework to build and run LLM agents) to stream browser video. We use playwright to automate browser interactions and provide that as tool to the LLM. When this tool is invoked, we stream the video of these browser interactions with FFmpeg by streaming the virtual display buffer the browser is using.

There is a noticeable delay booting up this pipeline for each tool invoke right now. We are working on putting in some optimizations but improvements in FFmpeg will definitely help. https://github.com/trypromptly/LLMStack is the project repo for the curious.

We built https://github.com/trypromptly/LLMStack to serve exactly this persona. A low-code platform to quickly build RAG pipelines and other LLM applications.

Kudos to the team for a very detailed notebook going into things like pipeline evaluation wrt performance and costs etc. Even if we ignore the framework specific bits, it is a great guide to follow when building RAG systems in production.

We have been building RAG systems in production for a few months and have been tinkering with different strategies to get the most performance out of these pipelines. As others have pointed out, vector database may not be the right strategy for every problem. Similarly there are things like lost in the middle problems (https://arxiv.org/abs/2307.03172) that one may have to deal with. We put together our learnings building and optimizing these pipelines in a post at https://llmstack.ai/blog/retrieval-augmented-generation.

https://github.com/trypromptly/LLMStack is a low-code platform we open-sourced recently that ships these RAG pipelines out of the box with some app templates if anyone wants to try them out.

Django has been my go to framework for any new web project I start for more than a decade. Its batteries-included approach meant that one could go pretty far with just Django alone. Included admin interface and the views/templating setup was what first drew me to the project.

Django project itself has kept pace with recent developments in web development. I still remember migrations being an external project, getting merged in and the transition that followed. Ecosystem is pretty powerful too with projects like drf, channels, social-auth etc., covering most things we need to run in production.

https://github.com/trypromptly/LLMStack is a recent project I built entirely with Django. It uses django channels for websockets, drf for API and reactjs for the frontend.

We recently opensourced a similar platform for building workflows by chaining LLMs visually along with LocalAI support.

Check it out at https://github.com/trypromptly/LLMStack. Like you said, it was fairly easy to integrate LocalAI and is a great project.

Would you mind expanding why it was tough to get started with Retool?

We are building https://github.com/trypromptly/LLMStack, a low-code platform to build LLM apps with a goal of making it easy for non-tech people to leverage LLMs in their workflows. Would love to learn about your experience with retool and incorporate some of that feedback into LLMStack.

We built LLMStack for our internal purposes and pulled it out into its own repo and open sourced it at https://github.com/trypromptly/LLMStack.

LLMStack (https://github.com/trypromptly/LLMStack) is a no-code platform to build LLM apps that we have been working on for a few months and open-sourced recently. It comes with everything out of the box that one needs to build LLM apps locally or in an enterprise setting.