Top 10 ai-safety Open-Source Projects
-
-
safe-rlhf
Safe RLHF: Constrained Value Alignment via Safe Reinforcement Learning from Human Feedback
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
Thought-Cloning
[NeurIPS '23 Spotlight] Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
ToolEmu
A language model (LM)-based emulation framework for identifying the risks of LM agents with tool use
-
adversarial-reinforcement-learning
Reading list for adversarial perspective and robustness in deep reinforcement learning.
Project mention: Show HN: Evaluate LLM-based RAG Applications with automated test set generation | news.ycombinator.com | 2024-04-11
Project mention: [R] Meet Beaver-7B: a Constrained Value-Aligned LLM via Safe RLHF Technique | /r/MachineLearning | 2023-05-16
Project mention: AI Agents Can Learn to Think While Acting: A New AI Research Introduces A Novel Imitation Learning Framework Called Thought Cloning | /r/machinelearningnews | 2023-06-07
Giskard - Testing framework for ML models| Multiple roles | Full-time | France | https://giskard.ai/
We are building the first collaborative & open-source Quality Assurance platform for all ML models - including Large Language Models.
Founded in 2021 in Paris by ex-Dataiku engineers, we are an emerging player in the fast-growing market of AI Quality & Safety.
Giskard helps Data Scientists & ML Engineering teams collaborate to evaluate, test & monitor AI models. We help organizations increase the efficiency of their AI development workflow, eliminate risks of AI biases and ensure robust, reliable & ethical AI models. Our open-source platform is used by dozens of ML teams across industries, both at enterprise companies & startups.
In 2022, we raised our first round of 1.5 million euros, led by Elaia, with participation from Bessemer and notable angel investors including the CTO of Hugging Face. To read more about this fundraising and how it will accelerate our growth, you can read this announcement: https://www.giskard.ai/knowledge/news-fundraising-2022
In 2023, we received a strategic investment from the European Commission to build a SaaS platform to automate compliance with the upcoming EU AI regulation. You can read more here: https://www.giskard.ai/knowledge/1-000-github-stars-3meu-and...
We are assembling a team of champions: Software Engineers, Machine Learning researchers, and Data Scientists ; to build our AI Quality platform and expand it to new types of AI models and industries. We have a culture of continuous learning & quality, and we help each other achieve high standards & goals!
We aim to grow from 15 to 25 people in the next 12 months. We're hiring the following roles:
Project mention: [R] Identifying the Risks of LM Agents with an LM-Emulated Sandbox - University of Toronto 2023 - Benchmark consisting of 36 high-stakes tools and 144 test cases! | /r/MachineLearning | 2023-10-11Website: https://toolemu.com/
ai-safety related posts
-
[R] Identifying the Risks of LM Agents with an LM-Emulated Sandbox - University of Toronto 2023 - Benchmark consisting of 36 high-stakes tools and 144 test cases!
-
ToolEmu: Identifying the Risks of LM Agents with an LM-Emulated Sandbox
-
[R] Awesome AI Safety – A curated list of papers & technical articles on AI Quality & Safety
Index
What are some of the best open-source ai-safety projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | giskard | 3,142 |
| 2 | safe-rlhf | 1,160 |
| 3 | guardrail | 311 |
| 4 | Thought-Cloning | 232 |
| 5 | ethics | 207 |
| 6 | make-safe-ai | 166 |
| 7 | awesome-ai-safety | 135 |
| 8 | ToolEmu | 86 |
| 9 | adversarial-reinforcement-learning | 75 |
| 10 | ai-safety-cheatsheet | 9 |
Sponsored