Show HN: CLI tool for detecting non-exact code duplication with embedding models

1. slopo

   1 4 252 6.4 Python

   Embedding-based code duplication detector

   

   Slopo doesn't deduplicate. It only reports similar code units, which in most cases are not duplicates. This needs to be cleaned up by the user, which is fast and easy with coding agents.

   Embedding calculation is also outsourced externally: it's only a simple API call with a LiteLLM wrapper.

   "brute-force similarity search" and O(n^2) may sound scary, but it works fine and this is not a bottleneck. For large projects, other parts are much slower, which has room for improvement. [1] is an implementation you probably saw. It uses NumPy, spreads work across all CPU cores and there is also a split into blocks (block_size = 1000). In larger sets, all vectors are not loaded into memory at once. Where I need to be honest, I didn't measure actual memory usage. I just tested this on large repos, so I'm aware of bottlenecks.

   From my perspective, this is the easiest part. Code extraction, chunking, applying boost, clustering, generating report and designing everything as a single user-friendly tool is a real challenge. Architectural and product decisions are more difficult than implementation and solving performance issues.

   [1] https://github.com/rafal-qa/slopo/blob/v0.3.0/src/slopo/anal...

2. AppSignal

   www.appsignal.com featured

   Monitoring that respects your time & budget. APM, error tracking, and dashboards for modern web apps. Ten-minute setup, transparent flat pricing, and support from engineers who actually use the product.

   

3. refactor

   2 2 4 4.9 Python

   Code duplication detector with machine-readable output for AI agents - finds similar code blocks across C/C++/Python/Go/Java/JS for automated refactoring (by forhadahmed)

   

4. semhash

   3 5 956 6.0 Python

   Fast Multimodal Semantic Deduplication & Filtering

   

   Have you compared this to https://github.com/MinishLab/semhash (or considered using that for the deduplication backend)?

Suggest a related project