Safari Alternatives

safari reviews and mentions

• MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers
  1 project | news.ycombinator.com | 29 Nov 2023

  > Also, we know that transformers can scale

  Do we have strong evidence that other models don't scale or have we just put more time into transformers?

  Convolutional resnets look to scale on vision and language: (cv) https://arxiv.org/abs/2301.00808, (cv) https://arxiv.org/abs/2110.00476, (nlp) https://github.com/HazyResearch/safari

  MLPs also seem to scale: (cv) https://arxiv.org/abs/2105.01601, (cv) https://arxiv.org/abs/2105.03404

  I mean I don't see a strong reason to turn away from attention as well but I also don't think anyone's thrown a billion parameter MLP or Conv model at a problem. We've put a lot of work into attention, transformers, and scaling these. Thousands of papers each year! Definitely don't see that for other architectures. The ResNet Strikes back paper is a great paper for one reason being that it should remind us all to not get lost in the hype and that our advancements are coupled. We learned a lot of training techniques since the original ResNet days and pushing those to ResNets also makes them a lot better and really closes the gaps. At least in vision (where I research). It is easy to railroad in research where we have publish or perish and hype driven reviewing.

• Unlimiformer: Long-Range Transformers with Unlimited Length Input
  3 projects | news.ycombinator.com | 5 May 2023

  After a very quick read, that's my understanding too: It's just KNN search. So I agree on points 1-3. When something works well, I don't care much about point 4.

  I've had only mixed success with KNN search. Maybe I haven't done it right? Nothing seems to work quite as well for me as explicit token-token interactions by some form of attention, which as we all know is too costly for long sequences (O(n²)). Lately I've been playing with https://github.com/hazyresearch/safari , which uses a lot less compute and seems promising. Otherwise, for long sequences I've yet to find something better than https://github.com/HazyResearch/flash-attention for n×n interactions and https://github.com/glassroom/heinsen_routing for n×m interactions. If anyone here has other suggestions, I'd love to hear about them.

• How big a breakthrough is this "Hyena" architecture?
  1 project | /r/ArtificialInteligence | 28 Apr 2023
• Hyena: This new technology could blow away GPT-4 and everything like it
  1 project | /r/u_waynerad | 26 Apr 2023

  Code: https://github.com/HazyResearch/safari

• Scaling Transformer to 1M tokens and beyond with RMT
  6 projects | news.ycombinator.com | 23 Apr 2023

  the code is here https://github.com/hazyresearch/safari you should try it and let us know your verdict.

• A note from our sponsor - InfluxDB
  www.influxdata.com | 4 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →