chatdocs VS Olive

https://github.com/marella/chatdocs , this one, right? Takes close to a minute to answer

https://github.com/marella/chatdocs , this one.

See here: https://github.com/marella/chatdocs#configuration (chatdocs.yml file, context_length)

What about chatdocs? I asked about it here and the author seems to be open to the idea.

https://github.com/marella/chatdocs is a fork from privateGPT but with many added features, GPU support, chat UI. There is a reddit thread about it https://www.reddit.com/r/LocalLLaMA/comments/14174f4/chatdocs_privategpt_web_ui_gpu_support_more/

and: https://github.com/marella/chatdocs

i played a bit around with chatdocs (https://github.com/marella/chatdocs) but unfortunately are my source documents mixed languages, german and english to be specific. The result heavily depends on the language the questions is asked in, which totally makes sense to me.

https://github.com/PromtEngineer/localGPT or https://github.com/marella/chatdocs

Sounds like a great project and I really like the YouTube tutorials. I haven't been able to get it to work inside WSL. I tried another project here with UI and it works https://github.com/marella/chatdocs

I use Microsoft Olive to optimize my networks. It works rather well as I did make my inference speed like 2-3 times faster, the models smaller and VRAM usage was reduced as well, though this needs the latest nvidia drivers to happen.

First update drivers, then follow the instruction here: Olive/README.md at main · microsoft/Olive · GitHub I try SD1.5 and DreamShaper model, both hit 30it/s

Microsoft has provided a path in DirectML for vendors like AMD to enable optimizations called ‘metacommands’. In the case of Stable Diffusion with the Olive pipeline, AMD is building driver support for a metacommand implementation intended to improve performance and reduce the time it takes to generate output from the model. This feature will be validated on AMD RDNA™ 3 devices including AMD Radeon™ RX 7900 Series graphics cards and AMD Ryzen™ 7040 Series Mobile processors with Radeon™ graphics.

Olive is an easy-to-use hardware-aware model optimization tool that composes industry-leading techniques across model compression, optimization, and compilation. Given a model and targeted hardware, Olive composes the best suitable optimization techniques to output the most efficient model(s) for inferencing on cloud or edge, while taking a set of constraints such as accuracy and latency into consideration.

Repository: https://github.com/microsoft/Olive