evaluate
torch-fidelity
| evaluate | torch-fidelity | |
|---|---|---|
| 3 | 3 | |
| 1,819 | 872 | |
| 2.4% | - | |
| 6.1 | 8.1 | |
| 2 days ago | 3 months ago | |
| Python | Python | |
| Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
evaluate
- [D] The MMSegmentation library from OpenMMLab appears to return the wrong results when computing basic image segmentation metrics such as the Jaccard index (IoU - intersection-over-union). It appears to compute recall (sensitivity) instead of IoU, which artificially inflates the performance metrics.
- [P] Releasing 🤗 Evaluate - an evaluation library for ML
- HuggingFace/evaluate: A library for easily evaluating ML models and datasets
torch-fidelity
-
[D] A better way to compute the Fréchet Inception Distance (FID)
The Fréchet Inception Distance (FID) is a widespread metric to assess the quality of the distribution of a image generative model (GAN, Stable Diffusion, etc.). The metric is not trivial to implement as one needs to compute the trace of the square root of a matrix. In all PyTorch repositories I have seen that implement the FID (https://github.com/mseitzer/pytorch-fid, https://github.com/GaParmar/clean-fid, https://github.com/toshas/torch-fidelity, ...), the authors rely on SciPy's sqrtm to compute the square root of the matrix, which is unstable and slow.
-
[D] Are there any good FID and KID metrics implementations existing that are compatible with pytorch?
Try torch-fidelity : https://github.com/toshas/torch-fidelity
-
How to compute Inception score for 3d GAN
You can, but two ingredients are required: a rich pretrained feature extractor for your data to replace InceptionV3 pretrained on ImageNet, and https://github.com/toshas/torch-fidelity, which can be used with custom feature extractors (also supports FID and KID).
What are some alternatives?
datasets - 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools
rexmex - A general purpose recommender metrics library for fair evaluation.
EvalAI - :cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
pytorch-fid - Compute FID scores with PyTorch.
avalanche - Avalanche: an End-to-End Library for Continual Learning based on PyTorch.
clean-fid - PyTorch - FID calculation with proper image resizing and quantization steps [CVPR 2022]
semantic-kitti-api - SemanticKITTI API for visualizing dataset, processing data, and evaluating results.
pycm - Multi-class confusion matrix library in Python
trajectopy-core - Trajectopy - Trajectory Evaluation in Python
trajectopy - Trajectopy - Trajectory Evaluation in Python