How to leverage a local LLM for a client?

• serve

  11 3,961 9.5 Java

  Serve, optimize and scale PyTorch models in production (by pytorch)

  

Looks like you are already up to speed loading LLaMa models which is great. Assuming this is Hugging Face PyTorch checkpoint, I think it should be possible to spin up a TorchServe instance which has in-built support for API access and HF Transformers. Since scale and latency aren’t a big concern for you, this should be good enough start.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project