pyllms
neural-engine
pyllms | neural-engine | |
---|---|---|
36 | 21 | |
665 | 1,866 | |
3.5% | - | |
8.5 | 5.1 | |
16 days ago | about 1 month ago | |
Python | ||
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pyllms
- The Man Who Killed Google Search
-
Ask HN: Who is hiring? (April 2024)
Kagi | Full & Part Time | Remote | http://kagi.com
Kagi is building a user-centric search engine, free from ads and tracking.
Our primary language is Crystal, and we are always interested in talking to developers who share our values. Some of our roles are listed below, but feel free to reach out even if you don't see a match.
https://help.kagi.com/kagi/company/hiring-kagi.html
You can reach me directly at [email protected]
-
Mojeek
If you're looking for alternative search, I have to mention Kagi (https://kagi.com/). Not free, but totally worth it to filter out results like geeksforgeeks, tutorialspoint, w3schools etc.
- Kagi Search Is Down
-
DuckDuckGo !Bangs
Tip: use bang searches from the browser address bar by setting your default search engine to DuckDuckGo (or https://kagi.com/)
-
What the Google overlords don't want you to see
Shout out to the Kagi search engine. There are no ads. The only incentive is to be good enough to earn your money.
- So I deployed Whoogle on my NAS....
-
Tell HN: I hate contemporary "predictive tile" UI design so much
All 3 of your examples are add-driven free platforms for finding content that the platform didn't produce. Free platforms are incentivized to overwhelm you with options, forcing you to look at everything, including ads. Like a grocery store or Ikea, they don't want you to make a quick in-and-out visit. They want you to look at everything they have in hopes that you'll be impulsive.
You want pretty much the opposite: a tool. Tools let you quickly and efficiently accomplish a task. No distractions. When you're done, you're done.
The paths to salvation that I see:
* Pay for tools that work well when they're available. I can't speak for them myself, but I know people who swear by <https://kagi.com/> for ad-free web searches.*
* Put time and effort into your own tools. For example, you can setup and run your own search engine. Although, if you're willing to put in that effort you might put effort into hijacking and cleaning up the interfaces of free platforms instead.
* Vote and campaign for interoperability laws to enable others to put effort into hijacking and cleaning up bad interfaces.
*Surprisingly, interfaces don't get much better when you pay for video and music streaming services. I think that comes down to how most people use them. Most people don't open Netflix knowing what they want to watch. They open Netflix knowing that they're going want to watch something, and hoping to find something entertaining.
-
DDG founder says Google's phone, manufacturing partnerships thwart competition
I've been using Kagi [0] since a few months and I am extremely surprised how well it works. With DDG it took a few months and then I just added !g everywhere because I never found what I was looking for. With Kagi I almost never do that, every now and then I think I am not able to find what I expect and then I add !g, so far it hasn't given me more then what Kagi gives me.
So, I am wondering how much money you would really need. Because if Kagi can do it, why can't Bing do it?
[0]: https://kagi.com
- Kagi: No ads, fast and personalised results
neural-engine
-
Apple Introduces M4 Chip
~38 TOPS at fp16 is amazing, if the quoted number if fp16 (ANE is fp16 according to this [1] but that honestly seems like a bad choice when people are going smaller and smaller even at the higher level datacenter cards so not sure why apple would use it instead of fp8 natively)
[1]: https://github.com/hollance/neural-engine/blob/master/docs/1...
-
Optimize sgemm on RISC-V platform
yep. they have a neural engine that is separate from the CPU and GPU that does really fast matmuls https://github.com/hollance/neural-engine. it's basically completely undocumented.
-
Apple is adding more and more neural engine cores to their products, is there any way to use them for local LLMs?
Looks like the ANE ("Apple Neural Engine") cores are powerful but not as flexible/programmable as the GPU cores. There is no sign that LLM inference is possible with them or ever will be unless Apple either opens up the closed ANE software framework for extensibility or they extend the ANE framework to support modern LLMs themselves. I would not hold my breath.
-
Anthropic’s $5B, 4-year plan to take on OpenAI
If Apple would wake up to what's happening with llama.cpp etc then I don't see such a big role for paying for remote access to big models via API
Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2])
OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today
If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE)
And then imagine it does that while leaving the CPU+GPU mostly free to run apps/games ... the whole experience of using a computer could change radically in that case.
I find it hard not to think this is coming within 5 years (although equally, I can imagine this is not on Apple's roadmap at all currently)
[1] https://github.com/hollance/neural-engine
- Everything we actually know about the Apple Neural Engine (ANE)
- What we know about the Apple Neural Engine
-
Everything we know about the Apple Neural Engine (ANE)
My question too. This semi-answer on the page seems to contradict itself (source: https://github.com/hollance/neural-engine/blob/master/docs/p... ):
"> Can I program the ANE directly?
Unfortunately not. You can only use the Neural Engine through Core ML at the moment.
There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks.
(Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)"
The last part links to this bunch of headers:
https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva...
So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?
What are some alternatives?
InternLM - Official release of InternLM2 7B and 20B base and chat models. 200K context support
Dual-Edge-TPU-Adapter - Dual Edge TPU Adapter to use it on a system with single PCIe port on m.2 A/B/E/M slot
chatblade - A CLI Swiss Army Knife for ChatGPT
ANECompat - A tool which checks compatibility of CoreML model with Apple Neural Engine
auth - Fully open source, End to End Encrypted alternative to Google Photos and Apple Photos [Moved to: https://github.com/ente-io/ente]
pytorch-apple-silicon-benchmarks - Performance of PyTorch on Apple Silicon
sqleton - ☠️ sqleton ☠️ is a CLI tool to execute SQL commands
tensorexperiments - Boilerplate for GPU-Accelerated TensorFlow and PyTorch code on M1 Macbook
geppetto - golang GPT3 tooling
more-ane-transformers - Run transformers (incl. LLMs) on the Apple Neural Engine.
glazed - a library to make it easy to output structured data in your command line tools. add the icing on top of your data
cnn-benchmarks - Benchmarks for popular CNN models