CARLA
rliable
| CARLA | rliable | |
|---|---|---|
| 2 | 15 | |
| 263 | 698 | |
| 0.4% | 3.3% | |
| 0.0 | 2.5 | |
| 7 months ago | about 1 month ago | |
| Python | Jupyter Notebook | |
| MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
CARLA
-
[R] CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms
Abstract: Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favourable outcomes in the future (e.g., insurance approval). Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available open–source implementations, the decision in favour of certain models is primarily based on what is readily available. Going forward – to guarantee meaningful comparisons across explanation methods – we present CARLA (Counterfactual And Recourse Library), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods. We have open sourced CARLA and our experimental results on GitHub, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.
-
University of Tübingen Researchers Open-Source ‘CARLA’, A Python Library for Benchmarking Counterfactual Explanation Methods Across Data Sets and Machine Learning Models
4 Min Read| Paper | Github
rliable
-
[D] What is standard practice in RL when reporting average returns across multiple seeds in a table or a plot?
You can also look up https://github.com/google-research/rliable https://arxiv.org/pdf/2108.13264.pdf (Neurips 21 outstanding paper) IMHO the field would benefit if its moves in that direction.
-
What is the next booming topic in Deep RL?
You might like the best paper at NeurIPS last year: https://agarwl.github.io/rliable/
-
"Human-level Atari 200x faster", DeepMind 2022 (200x reduction in dataset scale required by Agent57 for human performance)
Deep RL at the Edge of the Statistical Precipice https://agarwl.github.io/rliable/
- How Hugging Face 🤗 can contribute to the Deep Reinforcement Learning Ecosystem?
-
Deep RL at the Edge of Statistical Precipice (NeurIPS Outstanding Paper)
You can find the paper, slides and poster at agarwl.github.io/rliable. The OP already put the poster here.
-
Google Highlights How Statistical Uncertainty Of Outcomes Must Be Considered To Evaluate Deep RL Reliably and Propose A Python Library Called ‘RLiable’
A recent Google study highlights how statistical uncertainty of outcomes must be considered for deep RL evaluation to be reliable, especially when only a few training runs are used. Google has also released an easy-to-use Python library called RLiable to help researchers incorporate these tools.
-
[R] Rliable: Better Evaluation for Reinforcement Learning—A Visual Explanation
Website: https://agarwl.github.io/rliable/
-
Towards creating better reward functions in a custom environment| Sensitivity analysis [Question]
First off, "performance" is highly speculative. So make sure you nail down what you mean and ensure reliability of those measurements. Check out https://github.com/google-research/rliable.
- Deep Reinforcement Learning at the Edge of the Statistical Precipice
-
Best RL papers from the past year or two?
Our NeurIPS'21 oral on Deep RL at the Edge of the Statistical Precipice would make for a fun read :)
What are some alternatives?
carla - Open-source simulator for autonomous driving research.
iris - Transformers are Sample-Efficient World Models. ICLR 2023, notable top 5%.
shapash - 🔅 Shapash: User-friendly Explainability and Interpretability to Develop Reliable and Transparent Machine Learning Models
cleverhans - An adversarial example library for constructing attacks, building defenses, and benchmarking both
alibi - Algorithms for explaining machine learning models
pytorch-lightning - Build high-performance AI models with PyTorch Lightning (organized PyTorch). Deploy models with Lightning Apps (organized Python to build end-to-end ML systems). [Moved to: https://github.com/Lightning-AI/lightning]
DiCE - Generate Diverse Counterfactual Explanations for any machine learning model.
sagemaker-explaining-credit-decisions - Amazon SageMaker Solution for explaining credit decisions.
causallift - CausalLift: Python package for causality-based Uplift Modeling in real-world business
responsible-ai-toolbox - Responsible AI Toolbox is a suite of tools providing model and data exploration and assessment user interfaces and libraries that enable a better understanding of AI systems. These interfaces and libraries empower developers and stakeholders of AI systems to develop and monitor AI more responsibly, and take better data-driven actions.
Stanza - Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages
pybench - Python benchmark tool inspired by Geekbench.