[D] Why doesn’t your team use an experiment tracking tool?

• dvc

  109 13,139 9.6 Python

  🦉 ML Experiments and Data Management with Git

  

Unfortunately, there are some issues with `dvc exp` --- the set of experiment tracking subcommands. In particular, I rely heavily on git submodules to partition the code that instantiates a model from the code that runs an experiment. But `dvc exp` doesn't work with submodules ATM. (Bug filed here.) This is unfortunate because, if `dvc exp` worked, it would make experiment tracking a little more convenient for us. It's not a deal breaker though. I use git branches to organize individual experiments and tags to organize stages of the same experiment. I use a shared dvc cache so that I can run multiple experiments at a time without using up too much workspace storage.

• labml

  23 1,871 9.7 Jupyter Notebook

  🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• ploomber

  121 3,380 7.4 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

I find the value proposition of experiment trackers a bit off, at least in my domain (classic ML, DL is a different story). When developing a model, what gives you the biggest performance improvements is better data cleaning, rarely, hyperparameters have an important effect. So I do not use experiment trackers because I mostly work on iterating on my data and just generate a Jupyter notebook report (converted to HTML) for each experiment with some diagnostic plots.

• aim

  70 4,797 8.0 Python

  Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.

  

Same! Neptune is great. I'm also keeping an eye on Aim which is a very similar open-source solution but not as mature as Neptune yet. I can't wait to switch to this one because Neptune gets kind of pricey if working in a team.

• guildai

  16 856 8.8 Python

  Experiment tracking, ML developer tools

  

Guild AI now has support for running DvC stages as experiments. DvC uses git under the covers to manage project state for each experiment, along with the experiment results. Guild doesn't touch your git repo and instead copies your project source to a new run directory. This ensures that you have a correct record of your experiment without churning your project state.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project