runpodctl VS ollama

You can rent a GPU on https://www.runpod.io/, which also has stable diffusion templates so any time you start the GPU SD will be preinstalled for you, ready to use:)

Another option is to use a service like www.runpod.io to rent time on more powerful systems. A few times a week I’ll load up whatever the latest 13B or 20/24B (and even low bpw 70Bs with EXL2) model is on a system with an rtx 3090 for $0.44/hr, and sometimes I’ll treat myself to an A6000 system to run 4bit 70Bs for $0.79/hr. They also offer A4000 systems with 16GB VRAM which is plenty to run a 4bpw 13B EXL2 model or an 8bpw 7B model, and those systems are just $0.36/hr.

Many people use sites like runpod for this.

You can rent time on systems on www.runpod.io with a 48GB A6000 for $0.50/hr spot pricing and $0.79/hr regular pricing. 3090s can be had for $0.29/hr spot pricing and $.44/hr regular.

Keep in mind you can also use www.runpod.io to rent access to systems with a 3090 for about $.45/hr. It might be significantly cheaper ir at least more affordable to do this for a few hours a week instead of dropping $1,000 on a new laptop. This is what I personally do (I generally use it in the evening, and can get an Nvidia A6000 with 48GB VRAM for $.49/hr spot pricing). This lets me play with the latest 33B and 65B models, with really fast replies, and I spend maybe $5-7 a week if I use it a lot.

you can also connect the volume to other cloud storage (pcloud, dropbox, etc) (look for the option cloud sync in the "my pods" console) , or use python libraries to pull/push files from S3 buckets / dropbox / ftps / your NAS / etc, or use their docker tools to transfer files between your local PC and the volumehttps://github.com/runpod/runpodctl/blob/main/README.md

- Basic internet search (I start ollama CLI faster than I can start a browser - https://ollama.com)

- Formatting/changing text

- Troubleshooting code, esp. new frameworks/libs

- Recipes

- Data entry

- Organizing thoughts: High-level lists, comparison, classification, synonyms, jargon & nomenclature

- Learning esp. by analogy and example

RAG for:

- Website assistants (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Game NPCs (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Discord/Slack/forum bots (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Character-driven storytelling and creating art in a specific style for video game loading screens, background images, avatars, website art, etc. (https://github.com/bennyschmidt/ragdoll-studio/tree/master/r...)

Jan goes a step further by integrating with other local engines like LM Studio and ollama.

# install the Ollama curl -fsSL https://ollama.com/install.sh | sh # get the llama3 model ollama pull llama2 # install the MLFlow pip install mlflow

Ollama for running LLMs locally

curl -fsSL https://ollama.com/install.sh | sh

Streaming is not a problem (it's just a simple flag: https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L2...) but I've never used voice input.

The examples show image input though: https://github.com/ollama/ollama/blob/main/docs/api.md#reque...

Maybe you can file an issue here: https://github.com/ollama/ollama/issues

I’m a huge fan of open source models, especially the newly release Llama 3. Because of the performance of both the large 70B Llama 3 model as well as the smaller and self-host-able 8B Llama 3, I’ve actually cancelled my ChatGPT subscription in favor of Open WebUI, a self-hostable ChatGPT-like UI that allows you to use Ollama and other AI providers while keeping your chat history, prompts, and other data locally on any computer you control.