emerging-trajectories VS CX_DB8

I think LLMs can do a lot more than people assume, but they need to be given the proper frameworks.

When was the last time a researcher, economist, etc. was given 10,000 papers and simply told "do some original work"? That's not how it works. Daniel (the author) provides some good examples where _streamlined_ work can happen, but again, this is pretty basic stuff.

To push this further, though, imagine LLMs that fill in frameworks... A few steps here: (1) do a lit review, (2) fill in the framework, (3) discuss what might be missing, and maybe even try and fill in the missing information.

I'm doing something like this with politics and economics (see: https://emergingtrajectories.com/) and it works generally well. I think with a ton more engineering, curating of knowledge bases, etc., one can get these LLMs to actually find some new "nuggets" of information.

Admittedly, it's very hard, but I think there's something there.

RAG will have a place in the LLM world, since it's a way to obtain data/facts/info for relevant queries.

Since you asked about alternatives...

(a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2].

(b) Depending on how you define RAG, knowledge graphs could be a form of RAG or alternatively an alternative to them. Companies like Elemental Cognition[3] are building distinct alternatives to RAG that use such graphs and give LLMs the ability to run queries on said graphs. Another approach here is to build "fact databases" where, you structure observations about the world into standalone concepts/ideas/observations and reference those[4]. Again, similar to RAG but not quite RAG as we know it today.

[1] https://deepmind.google/discover/blog/alphageometry-an-olymp...

[2] https://arxiv.org/abs/2306.12672

[3] https://ec.ai/

[4] https://emergingtrajectories.com/

For those interested in using search-augmented "reasoning", I implemented something similar in Emerging Trajectories[1], an open source package that forecasts geopolitical and economic events. We extract facts[2] from various websites (Google searches, news articles, RSS feeds) and have the LLM generate a hypothesis on a metric.

We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3].

[1] https://emergingtrajectories.com/

[2] Sample code: https://github.com/wgryc/emerging-trajectories/blob/main/eme...

[3] https://emergingtrajectories.com/a/statement/28

I'm working on research agents to help with economic, financial, and political research. These agents are open source (see: https://github.com/wgryc/emerging-trajectories).

The use cases are pretty straight forward and low risk:

1. Run a Google web search.

2. Query a news API.

3. Write a document based on the above, while citing sources.

Here's an example of something written yesterday, where I'm forecasting whether July 2024 will be the hottest on record: https://emergingtrajectories.com/a/forecast/74

This is working well in that the writeups are great and there are some "aha" moments, like the agent finding and referencing the The National Snow and Ice Data Center (NSIDC)... Very cool! I wouldn't have thought of it.

Then there's the part where the agent also tells me that the Oregon Department of Transportation has holidays during the summer, which doesn't matter at all.

So, YMMV, as they say... But I am more productive with these agents. I wouldn't publish anything formally without confirming and reviewing the content, though.

LLM agents to forecast geopolitical and economic events.

- Site: https://emergingtrajectories.com/

- GitHub repo: https://github.com/wgryc/emerging-trajectories

I've helped a number of companies build various sorts of LLM-powered apps (chatbots mainly) and found it interesting but not incredibly inspiring. The above is my attempt to build something no one else is working on.

It's been a lot of fun. Not sure if it'll be a "thing" ever, but I enjoy it.

I was working on this stuff before it was cool, so in the sense of the precursor to LLMs (and sometimes supporting LLMs still) I've built many things:

1. Games you can play with word2vec or related models (could be drop in replaced with sentence transformer). It's crazy that this is 5 years old now: https://github.com/Hellisotherpeople/Language-games

2. "Constrained Text Generation Studio" - A research project I wrote when I was trying to solve LLM's inability to follow syntactic, phonetic, or semantic constraints: https://github.com/Hellisotherpeople/Constrained-Text-Genera...

3. DebateKG - A bunch of "Semantic Knowledge Graphs" built on my pet debate evidence dataset (LLM backed embeddings indexes synchronized with a graphDB and a sqlDB via txtai). Can create compelling policy debate cases https://github.com/Hellisotherpeople/DebateKG

4. My failed attempt at a good extractive summarizer. My life work is dedicated to one day solving the problems I tried to fix with this project: https://github.com/Hellisotherpeople/CX_DB8

I really missed out on this thread despite being likely one of the most important folks to post on it (I turned my time in Policy Debate into an NLP career - see DebateSum: https://huggingface.co/datasets/Hellisotherpeople/DebateSum and CX_DB8: https://github.com/Hellisotherpeople/CX_DB8)

For those who are interested in the intersection of AI and Debate Evidence, there's a lot more work being done right now. We have a follow-up dataset to DebateSum on its way to a paper at some conference called OpenCaseList: https://huggingface.co/datasets/Yusuf5/OpenCaselist which is basically DebateSum but 40x better in every way. This is also likely the largest and best quality argument mining dataset ever gathered.

Fun anecdote, when I tried to introduce automatic extractive summarization tools to the debate community, I had parent/judge/teacher groups who were FLIPPING out about this. They were not happy at the idea of automatic debating or computer assisted debating systems.

This has deep connections with my attempt to implement an effective queryable word-level grammatically correct extractive text summarizer (AKA: The way most people actually summarize documents) - https://github.com/Hellisotherpeople/CX_DB8

I will try to implement this with the necessary changes to actually make this work properly, where instead of generating a new answer, it simply highlights the most likely text spans.

Is there any path forward to make Haystack do word-level extractive summarization? e.g. like this: https://github.com/Hellisotherpeople/CX_DB8

or like this: https://huggingface.co/spaces/Hellisotherpeople/Unsupervised...

I am trying to find anything better than these two for this task. I feel like Haystack could be an option - but I am not sure.