llama-cpp-python VS continue

There's a Python binding for llama.cpp which is actively maintained and has worked well for me: https://github.com/abetlen/llama-cpp-python

I'll share the core bit that took a while to figure out the right format, my main script is a hot mess using embeddings with SentenceTransformer, so I won't share that yet. E.g: last night I did a PR for llama-cpp-python that shows how Phi might be used with JSON only for the author to write almost exactly the same code at pretty much the same time. https://github.com/abetlen/llama-cpp-python/pull/1184

Python Bindings for llama.cpp

If the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.

https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.

LocalAI https://localai.io/ and LMStudio https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though)

I went with Python, llama-cpp-python, since my goal is just to get a small project up and running locally.

Continue (YC S23) | Founding Engineer | ONSITE | Full-time | San Francisco | $130-$170K + 1-2% Equity

At Continue, we are on a mission to make building software feel like making music. We are creating the open-source autopilot for VS Code and JetBrains——the easiest way to code with any LLM (https://github.com/continuedev/continue).

You are likely a good fit if you

- have founded or want to found your own startup one day

- have experience with frontend, backend, ML technologies

- are enthusiastic about AI/LLMs, open source, developer tools

- get excited about supporting users and helping customers

- want to work in-person in SF the majority of the time

More info: https://www.ycombinator.com/companies/continue/jobs/smcxRnM-...

Continue doesn’t support tab completion like Copilot yet.

A pull/merge request is being worked on: https://github.com/continuedev/continue/pull/758

Good job on the project, but it's unfortunately named. A privy also refers to a latrine.

Given that this project was started well after Continue.dev, I think it would be useful to include an FAQ or a comparison table on what exactly makes this project different.

https://github.com/continuedev/continue

In this Continue.dev file https://github.com/continuedev/continue/blob/preview/server/continuedev/libs/llm/llamacpp.py the request to llama.cpp is implemented.

Continue | Founding Engineer | ONSITE | Full-time | San Francisco | $130-$170K + 1-2% Equity

At Continue, we are on a mission to make building software feel like making music. We are creating the open-source autopilot for software development—an IDE extension that brings the power of ChatGPT to VS Code and JetBrains (https://github.com/continuedev/continue).

You are likely a good fit if you

- have founded or want to found your own startup one day

- have experience with frontend, backend, and ML technologies

- are enthusiastic about AI/LLMs, open source, developer tools

- get excited about supporting users and helping customers

- want to work in-person in SF the majority of the time

More info: https://www.ycombinator.com/companies/continue/jobs/smcxRnM-...

Original source: https://github.com/continuedev/continue/tree/main/docs/docs/languages/matlab.md

We are seeing developers deploy open-source LLMs for their teams to use while coding internally, which each developer then uses with Continue

Hi HN!

Since launching Continue two months ago (https://news.ycombinator.com/item?id=36882146), we've received amazing feedback, added features, and greatly improved reliability. But one of the biggest things we heard was the desire for a JetBrains extension. My co-founder Ty and I are super excited to share that we've released an extension for PyCharm, Intellij, WebStorm, and most other JetBrains IDEs - ready for alpha users at https://plugins.jetbrains.com/plugin/22707-continue.

Perhaps the most exciting part is that this effort was kickstarted and in great part developed by a community contributor! If you're curious what it took to make this happen, check out the PR here (https://github.com/continuedev/continue/pull/457). We hope to eventually support every IDE, so we made adding a new extension as easy as implementing a single class. If you're curious why this is possible, you can read more about the Continue Server and the architectural decisions we made here: https://blog.continue.dev/how-we-made-continue-ide-agnostic.