Show HN: I scraped 25M Shopify products to build a search engine

• Geziyor

  2 2,475 0.6 Go

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  

As someone who has scraped millions of items myself, I had success using Geziyor (https://github.com/geziyor/geziyor) built in Go. Shopify sites are especially easy to scrape because they tend to share the same product data formatting and don't hide it behind JS rendering.

• showcase-songs-search

  2 153 0.0 JavaScript

  A site to instantly search 32M songs from the MusicBrainz songs database, using Typesense Search (an open source alternative to Algolia / ElasticSearch) ⚡ 🎵 🔍

  

I'm biased, but I'd recommend exploring Typesense for search.

It's an open source alternative to Algolia + Pinecone and e-commerce is a very common use-case.

Here's a live demo with 32M songs: https://songs-search.typesense.org/

Disclaimer: I work on Typesense.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• usearch

  20 1,629 9.8 C++

  Fast Open-Source Search & Clustering engine × for Vectors & 🔜 Strings × in C++, C, Python, JavaScript, Rust, Java, Objective-C, Swift, C#, GoLang, and Wolfram 🔍

  

As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :)

https://github.com/unum-cloud/usearch - for faster search

https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings

• uform

  8 865 8.2 Python

  Pocket-Sized Multimodal AI for content understanding and generation across multilingual texts, images, and 🔜 video, up to 5x faster than OpenAI CLIP and LLaVA 🖼️ & 🖋️

  

As you scale, you may benefit from these two projects I maintain, and the Big Tech uses :)

https://github.com/unum-cloud/usearch - for faster search

https://github.com/unum-cloud/uform - for cheaper multi-lingual multi-modal embeddings

Suggest a related project