pipeline VS AI-Horde

if latency is not a critical requirement, you can try serverless GPU cloud like banana.dev, pipeline.ai . These platform provide an easy to use template for deploying LLM.

Assuming we choose pipeline.ai's services, we would have to pay $0.00055 per second of GPU usage. If we assume we will have 4000 users messaging 50 times a day, and every inference would take 10 seconds, we're looking at ~$33,000 every month for inference costs alone. This is a very rough estimation, as the real number of users will very likely be much higher when a website launches, and it will be greater than 50 messages per day for each user. A more realistic estimate would put us at over $100k-$150k a month.

Also the Horde for Stable Diffusion, pretty good concept: https://github.com/Haidra-Org/AI-Horde/

This is a crowdsourced distributed cluster of Image generation workers and text generation workers. If you like this service, consider joining the horde yourself!

There's AI Horde: https://github.com/db0/AI-Horde

You might be interested in the Stable Horde project : Official website: https://stablehorde.net/ GitHub page: https://github.com/db0/AI-Horde Discord server: https://discord.gg/3DxrhksKzn Related posts on Reddit: https://www.reddit.com/r/StableDiffusion/search/?q=horde&restrict_sr=1&sr_nsfw=

If its going to be AGPL that means you are allowed to use AI Hordes backend code if you wish. You can find it here : https://github.com/db0/AI-Horde/tree/KAI_DO_NOT_DELETE

The Kobold AI Horde is a community of volunteers who share their expensive computers for running AI models.

Kobold AI-Horde

I think that this is the correct github project: https://github.com/db0/AI-Horde

github: db0/AI-Horde: A crowdsourced distributed cluster for AI art and text generation (github.com)

ALSO!! If you're an ex-miner and have a lot of GPUs sitting around, consider offering them up to the Horde