promptfoo VS shap-e

Super interesting. We've been experimenting with [promptfoo](https://github.com/promptfoo/promptfoo) at my work, and this looks very similar.

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo.

I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks.

This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.

No. Its all generated with an AI model trained on 3D data from OpenAI (https://github.com/openai/shap-e) . I have an instance running on my machine and I query it from godot.

Finally, here is the Official Shap-E Github Repository. As Shap-E is a completely open source project, you can find everything that goes into the project here.

This could be what you’re after https://github.com/openai/shap-e

see here https://github.com/openai/shap-e, it also has the original paper

OpenAI have been releasing open source models recently, including Whisper (which was SOTA for Speech-to-text) and Shap-E. It's just the large scale LLMs they're not releasing. Also people are choosing to ignore that in Sam's statement before congress he argued only for regulation of large models on the scale of GPT-4, explicitly arguing against regulation of open source efforts.

It's not that no one paying attention to it, it's just the area hasn't had the same results as others. Personally I've tried to build a floor plan generator several years ago and it did not turn out great. I'm not sure of anything architecture specific out there but one recent success for 3d generation was Shape-E

(Source)