lightwood
benchmarks
Our great sponsors
lightwood | benchmarks | |
---|---|---|
2 | 2 | |
420 | 4 | |
3.8% | - | |
9.2 | 1.8 | |
8 days ago | over 2 years ago | |
Python | Python | |
GNU General Public License v3.0 only | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
lightwood
-
[D] What would a good ML take home test look like for you?
Create a very detailed issue about this (bonus points, you can use the same thing for all candidates to have a fair evaluation). Here's an example.
-
Launch HN: MindsDB (YC W20) – Machine Learning Inside Your Database
3. A decoder that is trained to generate images takes that representation and generates an image1.
Note: above is a good illustrative example, in practice, we're good with outputting dates, numerical, categories, tags and time-series (i.e. predicting 20 steps ahead). We haven't put much work into image/text/audio/video outputs
You should be able to find more details about how we do this in the docs and most of the heavy lifting happens in the lightwood repo, the code for that is fairly readable I hope: https://github.com/mindsdb/lightwood
benchmarks
-
Forecast Metro Traffic using MindsDB Cloud and MongoDB Atlas
We will be using the Metro traffic dataset 🚇 that can be downloaded from here. You are also free to use your own dataset and follow along the tutorial.
-
Launch HN: MindsDB (YC W20) – Machine Learning Inside Your Database
Regrading benchmarks, we have three main dataset collections we focus on currently:
1. Datasets from customers, but obviously those can’t be made public.
2. The OpenML benchmark, which is fairly limited because it’s mainly binary categories, but which is good because it’s a 3rd party, so unbiased. We have some intermediary results here (https://docs.google.com/spreadsheets/d/1oAgzzDyBqgmSNC6g9CFO...) , they are middle-of-the-road. However I think the benchmark is pretty limited, i.e. it doesn’t cover most of the kinds of inputs and almost none of the output we support
3. An internal benchmark suite which currently has 59 datasets, mainly focused around classification and regression tasks with many inputs, timeseries problems and text. Some part of it is public but opening that up is a bit difficult due to licensing issues. I’m hoping that in the next year it will grow and 90%+ of it can be made public. We benchmarkagainst older versions of mindsdb, against hand made models we try to adapt to the task, against the state of the art accuracy for the dataset (if we can find it) and a few other auto ML frameworks (well, 1, but I hope to extend that list) [see this repo for the ones we made public: https://github.com/mindsdb/benchmarks, but I'm afraid it's a bit outdated]
That being said benchmarking for us is still WIP, since as far as I can tell nobody is trying to build open source models that are as broad as what we're currently doing (for better or worst), and the closed source services offered by various IaaS providers don't really come with public benchmark results outside of marketing.
What are some alternatives?
MindsDB - The platform for customizing AI from enterprise data
nni - An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
PheKnowLator - PheKnowLator: Heterogeneous Biomedical Knowledge Graphs and Benchmarks Constructed Under Alternative Semantic Models
nitroml - NitroML is a modular, portable, and scalable model-quality benchmarking framework for Machine Learning and Automated Machine Learning (AutoML) pipelines.
kraken - OCR engine for all the languages
pyprobml - Python code for "Probabilistic Machine learning" book by Kevin Murphy
probability - Probabilistic reasoning and statistical analysis in TensorFlow
Projects-Archive - This hacktober fest, the only stop you’ll need to make for ML, Web Dev and App Dev - see you there!
funsor - Functional tensors for probabilistic programming