emerging-trajectories VS promptr

I think LLMs can do a lot more than people assume, but they need to be given the proper frameworks.

When was the last time a researcher, economist, etc. was given 10,000 papers and simply told "do some original work"? That's not how it works. Daniel (the author) provides some good examples where _streamlined_ work can happen, but again, this is pretty basic stuff.

To push this further, though, imagine LLMs that fill in frameworks... A few steps here: (1) do a lit review, (2) fill in the framework, (3) discuss what might be missing, and maybe even try and fill in the missing information.

I'm doing something like this with politics and economics (see: https://emergingtrajectories.com/) and it works generally well. I think with a ton more engineering, curating of knowledge bases, etc., one can get these LLMs to actually find some new "nuggets" of information.

Admittedly, it's very hard, but I think there's something there.

RAG will have a place in the LLM world, since it's a way to obtain data/facts/info for relevant queries.

Since you asked about alternatives...

(a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2].

(b) Depending on how you define RAG, knowledge graphs could be a form of RAG or alternatively an alternative to them. Companies like Elemental Cognition[3] are building distinct alternatives to RAG that use such graphs and give LLMs the ability to run queries on said graphs. Another approach here is to build "fact databases" where, you structure observations about the world into standalone concepts/ideas/observations and reference those[4]. Again, similar to RAG but not quite RAG as we know it today.

[1] https://deepmind.google/discover/blog/alphageometry-an-olymp...

[2] https://arxiv.org/abs/2306.12672

[3] https://ec.ai/

[4] https://emergingtrajectories.com/

For those interested in using search-augmented "reasoning", I implemented something similar in Emerging Trajectories[1], an open source package that forecasts geopolitical and economic events. We extract facts[2] from various websites (Google searches, news articles, RSS feeds) and have the LLM generate a hypothesis on a metric.

We're tracking the info forecasts to see how well this does for future events. For example, we're pitting the LLMs against each other to predict March 2024 CPI[3].

[1] https://emergingtrajectories.com/

[2] Sample code: https://github.com/wgryc/emerging-trajectories/blob/main/eme...

[3] https://emergingtrajectories.com/a/statement/28

I'm working on research agents to help with economic, financial, and political research. These agents are open source (see: https://github.com/wgryc/emerging-trajectories).

The use cases are pretty straight forward and low risk:

1. Run a Google web search.

2. Query a news API.

3. Write a document based on the above, while citing sources.

Here's an example of something written yesterday, where I'm forecasting whether July 2024 will be the hottest on record: https://emergingtrajectories.com/a/forecast/74

This is working well in that the writeups are great and there are some "aha" moments, like the agent finding and referencing the The National Snow and Ice Data Center (NSIDC)... Very cool! I wouldn't have thought of it.

Then there's the part where the agent also tells me that the Oregon Department of Transportation has holidays during the summer, which doesn't matter at all.

So, YMMV, as they say... But I am more productive with these agents. I wouldn't publish anything formally without confirming and reviewing the content, though.

LLM agents to forecast geopolitical and economic events.

- Site: https://emergingtrajectories.com/

- GitHub repo: https://github.com/wgryc/emerging-trajectories

I've helped a number of companies build various sorts of LLM-powered apps (chatbots mainly) and found it interesting but not incredibly inspiring. The above is my attempt to build something no one else is working on.

It's been a lot of fun. Not sure if it'll be a "thing" ever, but I enjoy it.

I taught https://github.com/KillianLucas/open-interpreter how to use https://github.com/ferrislucas/promptr

Then I asked it to add a test suite to a rails side project. It created missing factories, corrected a broken test database configuration, and wrote tests for the classes and controllers that I asked it to.

I didn't have to get involved with mundane details. I did have to intervene here and there, but not much. The tests aren't the best in the world, but IMO they're adding value by at least covering the happy path. They're not as good as an experienced person would write.

I did spend a non-trivial amount of time fiddling with the prompts I used to teach OI about Promptr as well as the prompts I used to get it to successfully create the test suite.

The total cost was around $11 using GPT4 turbo.

I think in this case it was a fun experiment. I think in the future, this type of tooling will be ubiquitous.

I made a CLI tool called Promptr that allows you to make changes to a codebase via plain English instructions:

https://github.com/ferrislucas/promptr

There’s a templating system (liquidjs) included which is useful if you have a library of prompts that you want to reference often.

You can think of it as a junior engineer that needs explicit instructions.

Here are a few example PR’s implemented by Promptr - see the commits for the prompt that was used to produce the code:

https://github.com/ferrislucas/promptr/pull/38

https://github.com/ferrislucas/promptr/pull/41

https://github.com/ferrislucas/promptr

You just prompt it directly or with a file, and it applies the changes to your file system. There's also a templating system that allows you to reference other files from your prompt file if you want to have a shared prompt file that contains project conventions etc.

This is exactly the use that Promptr is intended for https://github.com/ferrislucas/promptr

* full disclosure: I’m the author of Promptr

I’ve been experimenting with pairing a tool I wrote called Promptr [1] with another tool called Open Interpreter [2].

I start with a prompt that teaches Open Interpreter how to use Promptr, and then I discuss what I’m trying to accomplish. It’s certainly not perfect, but there’s definitely something good that happens when you can iterate using dialog with a robot that can modify your file system and execute commands locally.

[1] Promptr: https://github.com/ferrislucas/promptr

[2] Open Interpreter: https://github.com/KillianLucas/open-interpreter

I wrote a great tool for this: https://github.com/ferrislucas/promptr

It’s great for making changes to existing code because it automatically includes the relevant files for context.

For those looking for specific examples of useful code being authored by AI, you can check out this tool:

https://github.com/ferrislucas/promptr

The README links to example PR’s comprised of commits written by GPT4. The prompts used to produce the code are noted in the commit messages.

Promptr is a coding assistant tool that allows you to ask GPT to produce or modify code, and the results will be automatically applied to your file system.

https://github.com/ferrislucas/promptr

From the README:

I'm also interested in this problem. It can be theoretically solved by giving GPT long-term memory about a specific codebase through vector embedding generation (using OpenAI's embeddings API). The semantic embeddings can then be stored in a vector (or vector-supported) database such as Pinecone alongside metadata for querying. Some of the key considerations are how to compare vectors for similarity (there are many algorithms) and how to use metadata to better support your use case. The following resources can be helpful to further understand this technique: - https://platform.openai.com/docs/guides/embeddings - https://www.mlq.ai/fine-tuning-gpt-3-question-answer-bot/ - https://www.pinecone.io/learn/javascript-chatbot/ A couple of semi-related projects I've been looking into: - https://github.com/ferrislucas/promptr - https://github.com/pashpashpash/vault-ai

Promptr is a CLI tool for operating on your codebase using GPT. (https://github.com/ferrislucas/promptr)