Top 23 Python Benchmarking Projects

Benchmarking

• Add a project

1. locust

   1 59 26,048 9.7 Python

   Write scalable load tests in plain Python 🚗💨

   

   Project mention: Smoke, stress, spike, soak, and recovery: 5 essential load test profiles | news.ycombinator.com | 2024-08-15

   Hi everyone!

   Long time lurker, first time poster here. I'm the maintainer of Locust (https://github.com/locustio/locust), and this is the second part of my series about load testing.

   Let me know what you think!

2. Judoscale

   judoscale.com featured

   Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

   

3. Merlion

   2 6 4,259 4.9 Python

   Merlion: A Machine Learning Framework for Time Series Intelligence

   

   Project mention: Merlion: A Machine Learning Framework for Time Series Intelligence | news.ycombinator.com | 2025-02-28

4. AutoRAG

   3 5 3,856 9.3 Python

   AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

   

   Project mention: AIM Weekly 28 Oct 2024 | dev.to | 2024-10-28

   📎 AutoRAG with Milvus 🛠️ ADO 🫶 Self Hosting LLM 🌐 Noema Declarative AI 📝 New NIM Blueprint for building AI Virtual Assistant 🚙 Zilliz Integrations 🫶 Using Milvus for Semantic Search 🤖 Contextual Retrieval 📎 Meta: Quantized Light Weight Models 🚙 https://arxiv.org/pdf/2407.01219 ✅ Cool Icons 🙌 IBM Watson AI Milvus Bot 📎 The Hacker's Browser 🛠️ Small and Mighty H2O Model 📝 Zilliz Cloud vs Qdrant 💫 Gravatino and Agents 🛠️ OSS Summit Europe 2024 Report ▶️ RAG Strategi 🤖 MS AI Data Visualizations 🌐 Graph RAG 👽 South Bay Meetup 15 Oct 2024 🦾 Influx and Milvus 👽 Multimodal Pipelines ✨ Constrained Sampling from LLM 🚕 BAML: Cheaper, Fast and More Accurate Function Calling 📊 Infinite World Generation with outlines txt 💻 Ollama Client Swift 🍔 Atomic Agents 🕶️ PYMUPDF4LLM 🚕 Milvus for AI Agents 📊 Fine Tuning LLAMA 3 with ORPO 🦾 Run NVIDIA Models 💻 LLM Training Meta Lingua ✨ 1 Bit LLM - MS BitNet 💻 Intro 🕶️ Mastering Chunk 📊 Storm Stanford Tool 🐍 DAMO NLP SG CaRing 🍔 LLM Reasoners

5. benchmark_VAE

   4 4 1,904 4.9 Python

   Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)

   

6. pytest-benchmark

   5 2 1,300 7.7 Python

   pytest fixture for benchmarking code

   

7. pyperf

   6 1 847 7.3 Python

   Toolkit to run Python benchmarks

   

8. pygraft

   7 1 683 6.0 Python

   Configurable Generation of Synthetic Schemas and Knowledge Graphs at Your Fingertips

   

9. InfluxDB

   influxdata.com featured

   InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.

   

10. truvari

    8 1 353 9.5 Python

    Structural variant toolkit for VCFs

    

11. CARLA

    9 2 286 0.0 Python

    CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms (by carla-recourse)

    

12. benchexec

    10 3 255 9.7 Python

    BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

    

    Project mention: JavaScript Benchmarking Is a Mess | news.ycombinator.com | 2024-12-24

    No interest in a more general tool?

    https://github.com/sosy-lab/benchexec

13. fastero

    11 1 237 2.3 Python

    Python timeit CLI for the 21st century! colored output, multi-line input with syntax highlighting and autocompletion and much more!

    

14. ferret

    12 2 212 7.3 Python

    A python package for benchmarking interpretability techniques on Transformers. (by g8a9)

    

15. Xpedite

    13 2 164 4.2 Python

    A non-sampling profiler purpose built to measure and optimize performance of C++ low latency/real time systems

    

16. python-benchmark-harness

    14 29 157 0.0 Python

    A micro/macro benchmark framework for the Python programming language that helps with optimizing your software.

    

17. CompanyKG

    15 1 131 5.1 Python

    A Large-Scale Company Relation Graph for Investment Industry

    

18. video-quality-metrics

    16 2 129 7.8 Python

    Uses FFmpeg to benchmark video encoders to compare VMAF/SSIM/PSNR with different encoder settings.

    

19. LLMeBench

    17 1 96 8.7 Python

    Benchmarking Large Language Models

    

    Project mention: Benchmarking LLMs Made Easy with LLMeBench | dev.to | 2025-02-28

    🔗 GitHub Repo → github.com/qcri/LLMeBench

20. less_slow.py

    18 2 90 7.9 Python

    Playing around "Less Slow" coding practices in Python, from numerical micro-kernels to coroutines, ranges, and polymorphic state machines

    

    Project mention: Show HN: Less Slow C++ | news.ycombinator.com | 2025-04-18

    Great question! This has been top of mind for me for the last 2–3 years.

    Short answer: sadly, no. I love the "usability" promise of coroutines—and even have 2–3 FOSS projects that could be rewritten entirely around C++ or Rust coroutines for better debuggability and extensibility—but my experiments show that the runtime cost of most coroutine‑like abstractions is simply too high. Frankly, I’m not even sure if a better design is possible on modern hardware.

    This leads me to conclude that, despite my passion for SIMD and superscalar execution, the highest‑impact new assembly instructions that x86 and Arm could standardize would center on async execution and lightweight context switching... yet I haven’t seen any movement in that direction.

    ⸻

    I also wrote toy examples for various range/async/stream models in C++, Rust, and Python, with measured latencies in inline comments:

      - Python: 

21. pytest-codspeed

    19 3 77 8.6 Python

    Pytest plugin to create CodSpeed benchmarks

    

    Project mention: Ask HN: Who is hiring? (March 2025) | news.ycombinator.com | 2025-03-03

    CodSpeed | Founding AI Engineer | On-site (Paris) / Remote (Europe) | Full-time | https://codspeed.io

    We're building software performance optimization tools to optimize and measure code performance before it is deployed to production. We avoid regressions that impact UX and help developers solve their performance issues faster. We're already live and trusted by top-tier open-source project teams such as Pydantic, Ruff, and Prisma.

    We’re at an exciting early stage and looking for talented engineers who share our passion for helping to enhance the performance of software used by billions, improving the software development lifecycle, and building tools we love to use ourselves.

    Apply at https://codspeed.notion.site/Founding-AI-Engineer-cd1bf4fd73...

22. ycecream

    20 0 55 7.4 Python

    Sweeter debugging and benchmarking Python programs.

    

23. toml-bench

    21 2 52 4.6 Python

    Which toml package to use in python?

    

24. api-performance-tests

    22 10 40 3.0 Python

    Benchmarking Litestar vs other ASGI API framework

    

25. cloud_benchmarker

    23 4 35 6.9 Python

    Cloud Benchmarker automates performance testing of cloud instances, offering insightful charts and tracking over time.

    

26. CodeRabbit

    coderabbit.ai featured

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    

Python Benchmarking related posts

• CatBench Vector Search Playground on Postgres

  1 project | news.ycombinator.com | 2 Mar 2025

• BenchExec: A Framework for Reliable Benchmarking and Resource Measurement

  1 project | news.ycombinator.com | 14 Jun 2024

• Pinpoint performance regressions with CI-Integrated differential profiling

  4 projects | dev.to | 23 Oct 2023

• Show HN: Cloud Benchmarker: See how fast your cloud instances are for real

  1 project | news.ycombinator.com | 11 Oct 2023

• Cloud Benchmarker: See how fast your cloud instances are for real!

  1 project | /r/programming | 29 Sep 2023

• Cloud Benchmarker: See how fast your cloud instances are for real! (Python Open-Source)

  1 project | /r/Python | 29 Sep 2023

• Show HN: 78% MNIST accuracy using GZIP in under 10 lines of code

  5 projects | news.ycombinator.com | 20 Sep 2023
• A note from our sponsor - InfluxDB
  influxdata.com | 25 Apr 2025

  Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems. Learn more →

Index

Sponsored

Save 47% on cloud hosting with autoscaling that just works

Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

judoscale.com

Do not miss the trending Python projects with our weekly report!