How to detect and fix flaky tests in Pytest

1. pytest-xdist

   1 3 1,572 8.4 Python

   pytest plugin for distributed testing and loop-on-failures testing modes.

   

   This is particularly problematic if you're using a library like pytest-xdist to distribute your tests across multiple CPUs. Testing in parallel and in random order introduces opportunities for both inconsistent system states to cause flakiness and race conditions where concurrently running tests interfere with each other.

2. Judoscale

   judoscale.com featured

   Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

   

3. pytest-rerunfailures

   2 2 410 5.5 Python

   a pytest plugin that re-runs failed tests up to -n times to eliminate flakey failures

   

   If you only have a few flaky tests, rerunning tests on failure to clear flakiness can be effective. Libraries like pytest-rerunfailures let you configure reruns easily, which reruns each test on the first failure to see if it's just flaky.

4. pytest-flakefinder

   3 1 140 - Python

   Runs tests multiple times to expose flakiness.

   

   Dropbox open-sourced a solution that helps you find flaky tests called pytest-flakefinder. This allows you to automatically rerun each test multiple times to find flaky tests. By default, the tool will run each test 50 times on the same commit and validate if the results are consistent. On the same commit, the code should be identical from run to run. If a test is not producing consistent results, it is considered flaky.

5. pytest-replay

   4 1 55 7.3 Python

   Saves runs to allow to re-execute previous pytest runs to reproduce crashes or flaky tests

   

   After finding the high-impact tests to fix first, you can use a library like pytest-replay to help reproduce tests locally. The difficult-to-reproduce tests often reveal real problems with your infrastructure or code. Pytest-replay lets you replay failures from replay records saved in CI, which can be a real-time save.

6. 0.30000000000000004

   5 250 1,467 2.0 CSS

   Floating Point Math Examples

   

   This is due to the way floating point numbers are represented in memory, certain numbers like 0.3 are stored as a number very close to the original value (0.30000000000000004), but not the exact same. This problem also exists in other languages like JavaScript and C++, but is much more likely to cause problems in common applications of Python like machine learning and data science.

7. InfluxDB

   influxdata.com featured

   InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.

   

Suggest a related project