Python Benchmark

Open-source Python projects categorized as Benchmark | Edit details

Top 23 Python Benchmark Projects

  • fashion-mnist

    A MNIST-like fashion product database. Benchmark :point_down:

    Project mention: How to produce data visualizations like this? | | 2022-01-05
  • tianshou

    An elegant PyTorch deep reinforcement learning library.

    Project mention: Tensorflow vs PyTorch for A3C | | 2021-11-17

    Do you absolutely need A3C? A2C has become more widely used (see, e.g., the comment in, and the fact that both and have A2C implementations, but no A3C at first glance).

  • Scout APM

    Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.

  • ann-benchmarks

    Benchmarks of approximate nearest neighbor libraries in Python

    Project mention: Fastest way to calculate distance (drift) between vectors - at scale (billions) | | 2022-05-11
  • mmaction2

    OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark

    Project mention: Textbook or blogs for video understanding | | 2022-01-25

    No book or blog, but a great framework:

  • mmpose

    OpenMMLab Pose Estimation Toolbox and Benchmark.

    Project mention: [P] Object detection framework : Detectron2 VS MMDetection | | 2021-09-29

    The [MMLab key point detection]( is in a separate repo from detection.

  • asv

    Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

    Project mention: Is GitHub Actions suitable for running benchmarks? | | 2021-08-18

    scikit-image, the project that commissioned this task, uses Airspeed Velocity, or asv, for their benchmark tests.

  • smac

    SMAC: The StarCraft Multi-Agent Challenge

    Project mention: [D] Python plot package commonly used in RL papers | | 2021-08-09
  • SonarLint

    Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  • py-frameworks-bench

    Another benchmark for some python frameworks

  • pyperformance

    Python Performance Benchmark Suite

    Project mention: Hello, I created a interpreted dynamic programming language in C#. I use a bytecode compiler and a vm for interpretation. Right now I'm trying to optimise it. Any help would be great! | | 2022-03-19

    There are some standard benchmarks like fannkuch, deltablue, and so on (see a bunch for Python here) that you can port to your VM. They have adjustable values that you can raise or lower to increase or decrease the amount of time you take.

  • tape

    Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. (by songlab-cal)

    Project mention: ProteinBERT: A universal deep-learning model of protein sequence and function | | 2021-05-30

    We evaluated based on downstream tasks (multiple supervised benchmarks, including 4 from TAPE), not the LM performance.

  • automlbenchmark

    OpenML AutoML Benchmarking Framework

    Project mention: Show HN: AutoAI | | 2021-11-12

    Your list excludes most of well-known open-source AutoML tools such as auto-sklearn, AutoGluon, LightAutoML, MLJarSupervised, etc. These tools have been very extensively benchmarked by the OpenML AutoML Benchmark ( and have papers published, so they are pretty well-known to the AutoML community.

    Regarding Frankly, you don't seem to understand's AutoML offerings.

    I'm the creator of H2O AutoML, which is open source, and there's no "enterprise version" of H2O AutoML. The interface is simple -- all you need to specify is the training data and target. We have included DNNs in our set of models since the first release of the tool in 2017. Read more here: We also offer full explainability for our models: develops another AutoML tool called Driverless AI, which is proprietary. You might be conflating the two. Neither of these tools need to be used on the H2O AI Cloud. Both tools pre-date our cloud by many years and can be used on a user's own laptop/server very easily.

    Your Features & Roadmap list in the README indicates that your tool does not yet offer DNNs, so either you should update your post here or update your README if it's incorrect:

    Lastly, I thought I would mention that there's already an AutoML tool called "AutoAI" by IBM. Generally, it's not a good idea to have name collisions in a small space like the AutoML community.

  • Meta-SelfLearning

    Meta Self-learning for Multi-Source Domain Adaptation: A Benchmark

    Project mention: Domain adaptation text recognition/OCR dataset (MSDA) and benchmark: Multi-source domain adaptation dataset for text recognition | | 2021-09-27

    Codes address


    CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms (by carla-recourse)

    Project mention: [R] CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms | | 2021-09-29

    Abstract: Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favourable outcomes in the future (e.g., insurance approval). Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available open–source implementations, the decision in favour of certain models is primarily based on what is readily available. Going forward – to guarantee meaningful comparisons across explanation methods – we present CARLA (Counterfactual And Recourse Library), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods. We have open sourced CARLA and our experimental results on GitHub, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.

  • SDGym

    Benchmarking synthetic data generation methods.

    Project mention: [D] Synthetic data generation techniques for data privacy | | 2022-02-15

    I would suggest starting with "differentially private synthetic data generation". These methods utilize differential privacy and mostly protect against membership inference attacks, are very popular in the ML/DL community. I would also suggest reading up on privacy preserving ML methods in general and adversarial attacks against them (membership inference, inversion, reconstruction, property inference), but if you're keen on reading some code, check out sd-gym: The authors have collected implementations for a lot of PPSDG methods. Also I strongly suggest reading McMahan's 2016 paper:

  • FedScale

    FedScale: Benchmarking Model and System Performance of Federated Learning

    Project mention: We created the most comprehensive benchmark datasets for federated learning to date! | | 2021-11-19
  • fastero

    Python timeit CLI for the 21st century! colored output, multi-line input with syntax highlighting and autocompletion and much more!

    Project mention: Fastero – Python timeit CLI for the 21st century | | 2022-04-29
  • python-benchmark-harness

    A micro/macro benchmark framework for the Python programming language that helps with optimizing your software.

    Project mention: Sprucing up my read me for my Python project | | 2022-02-01
  • freqbench

    Comprehensive CPU frequency performance/power benchmark

  • ModelNet40-C

    Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions"

    Project mention: [R] NEW Robustness Benchmark for 3D Point Cloud Recognition, ModelNet40-C. "Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions" | | 2022-02-08

    Code for found:

  • imdbench

    IMDBench — Realistic ORM benchmarking

    Project mention: How slow are ORMs, really? | | 2022-03-30

    We spent a lot of time optimizing every benchmark. Here's what creator of sqlalchemy said about our code: (and then he proposed improvements). We'll surely take a look at prefetch_related.

  • compiler-benchmark

    Benchmarks compilation speeds of different combinations of languages and compilers.

    Project mention: Raw performance of compilers? | | 2021-10-13

    One synthetic benchmark I saw recently:

  • ORBIT-Dataset

    The ORBIT dataset is a collection of videos of objects in clean and cluttered scenes recorded by people who are blind/low-vision on a mobile phone. The dataset is presented with a teachable object recognition benchmark task which aims to drive few-shot learning on challenging real-world data.

    Project mention: Is there a dataset for personal items? | | 2022-04-27
  • LFattNet

    Attention-based View Selection Networks for Light-field Disparity Estimation

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2022-05-11.

Python Benchmark related posts


What are some of the best open-source Benchmark projects in Python? This list will help you:

Project Stars
1 fashion-mnist 10,016
2 tianshou 4,618
3 ann-benchmarks 2,888
4 mmaction2 2,013
5 mmpose 1,934
6 asv 682
7 smac 668
8 py-frameworks-bench 639
9 pyperformance 540
10 tape 435
11 automlbenchmark 282
12 Meta-SelfLearning 173
13 CARLA 166
14 SDGym 160
15 FedScale 149
16 fastero 147
17 python-benchmark-harness 134
18 freqbench 113
19 ModelNet40-C 107
20 imdbench 94
21 compiler-benchmark 89
22 ORBIT-Dataset 58
23 LFattNet 37
Find remote jobs at our new job board There are 8 new remote jobs listed recently.
Are you hiring? Post a new remote job listing for free.
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives