guildai VS tune-sklearn

Experiment tracking in DvC is implemented using git to store snapshots of a project and related artifacts. You might take a look at Guild AI's support for DvC, which is tightly integrated with DvC stages. You can run any of the stages defined for a project and you get a properly isolated run (each run is a project copy to ensure that you're not corrupting the run if you modify files while it's running - as well as properly supporting concurrent runs). Once you have runs in Guild, you can use any number of tools to study, compare, export, etc.

I built an experiment tracking tool (Guild AI) that focuses on code/model reuse and so this question is dear to my heart :) Best of luck!

I'm not aware of experiment tracking in Jupyter notebooks themselves. Guild AI is able to run notebooks as experiments however.

Disclosure - I'm the author of Guild AI so take this for the biased opinion that it is.

I'm the author of Guild AI (open source experiment tracking). For some time now Guild users have asked for DvC support. This is now available as a pre-release.

Guild AI now has support for running DvC stages as experiments. DvC uses git under the covers to manage project state for each experiment, along with the experiment results. Guild doesn't touch your git repo and instead copies your project source to a new run directory. This ensures that you have a correct record of your experiment without churning your project state.

Guild.ai - https://guild.ai/

Use guild and pytorch-lightning. Make it easy for new contributors to get your data by using dvc as a data access tool.

Of course not! :)

The Ray ecosystem is actually chalk full of integrations, from XGBoost Ray (https://docs.ray.io/en/master/xgboost-ray.html), to PyTorch on Ray (https://docs.ray.io/en/master/using-ray-with-pytorch.html), and of course hyperparameter search with Ray Tune for a variety of libraries, including Sklearn (https://github.com/ray-project/tune-sklearn).

If you mainly use scikit-learn, you should consider using tune-sklearn.

Just wanted to share a not widely known feature of PyCaret. By default, PyCaret's tune_model uses the tried and tested RandomizedSearchCV from scikit-learn. However, not everyone knows about the various advanced options tune_model() currently allows you to use such as cutting edge hyperparameter tuning techniques like Bayesian Optimization through libraries such as tune-sklearn, Hyperopt, and Optuna.

You might want to try out tune-sklearn as it seems like it works for catboost as well. I am trying it use tune-sklearn to speed up my scikit-learn hyperparameter tuning.