Compressing high-dimensional vectors by 97%

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • SurveyJS - Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • finalfusion-rust

    finalfusion embeddings in Rust

  • Nice article that explains product quantization very well!

    PQ is really a nice compression technique. I implemented PQ and Optimized PQ [1] a while back in our word embedding package for Rust:

    https://github.com/finalfusion/finalfusion-rust/

    https://github.com/finalfusion/reductive/

    Particularly Optimized PQ was effective in reducing vector sizes ~10 times with virtually no reconstruction loss. This made it much easier to ship models (no more 3GB embedding matrix with a neural net that is just a few megabytes large).

    [1] http://kaiminghe.com/publications/pami13opq.pdf

  • reductive

    Vector quantization

  • Nice article that explains product quantization very well!

    PQ is really a nice compression technique. I implemented PQ and Optimized PQ [1] a while back in our word embedding package for Rust:

    https://github.com/finalfusion/finalfusion-rust/

    https://github.com/finalfusion/reductive/

    Particularly Optimized PQ was effective in reducing vector sizes ~10 times with virtually no reconstruction loss. This made it much easier to ship models (no more 3GB embedding matrix with a neural net that is just a few megabytes large).

    [1] http://kaiminghe.com/publications/pami13opq.pdf

  • SurveyJS

    Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.

    SurveyJS logo
  • excalidraw

    Virtual whiteboard for sketching hand-drawn like diagrams

  • They seem to be done using Excalidraw [0]. I'm not 100% sure about the animated one, probably excalidraw-animate [1]

    [0] https://excalidraw.com/

  • excalidraw-animate

    A tool to animate Excalidraw drawings

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts