[D] Why doesn’t your team use an experiment tracking tool?

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/MachineLearning

Our great sponsors
  • CodiumAI - TestGPT | Generating meaningful tests for busy devs
  • ONLYOFFICE ONLYOFFICE Docs — document collaboration in your environment
  • Sonar - Write Clean Python Code. Always.
  • InfluxDB - Access the most powerful time series database as a service
  • dvc

    🦉 Data Version Control | Git for Data & Models | ML Experiments Management

    Unfortunately, there are some issues with `dvc exp` --- the set of experiment tracking subcommands. In particular, I rely heavily on git submodules to partition the code that instantiates a model from the code that runs an experiment. But `dvc exp` doesn't work with submodules ATM. (Bug filed here.) This is unfortunate because, if `dvc exp` worked, it would make experiment tracking a little more convenient for us. It's not a deal breaker though. I use git branches to organize individual experiments and tags to organize stages of the same experiment. I use a shared dvc cache so that I can run multiple experiments at a time without using up too much workspace storage.

  • labml

    🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱

  • CodiumAI

    TestGPT | Generating meaningful tests for busy devs. Get non-trivial tests (and trivial, too!) suggested right inside your IDE, so you can code smart, create more value, and stay confident when you push.

  • ploomber

    The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

    I find the value proposition of experiment trackers a bit off, at least in my domain (classic ML, DL is a different story). When developing a model, what gives you the biggest performance improvements is better data cleaning, rarely, hyperparameters have an important effect. So I do not use experiment trackers because I mostly work on iterating on my data and just generate a Jupyter notebook report (converted to HTML) for each experiment with some diagnostic plots.

  • aim

    Aim 💫 — An easy-to-use & supercharged open-source AI metadata tracker (experiment tracking, AI agents tracing)

    Same! Neptune is great. I'm also keeping an eye on Aim which is a very similar open-source solution but not as mature as Neptune yet. I can't wait to switch to this one because Neptune gets kind of pricey if working in a team.

  • guildai

    Experiment tracking, ML developer tools

    Guild AI now has support for running DvC stages as experiments. DvC uses git under the covers to manage project state for each experiment, along with the experiment results. Guild doesn't touch your git repo and instead copies your project source to a new run directory. This ensures that you have a correct record of your experiment without churning your project state.


    ONLYOFFICE Docs — document collaboration in your environment. Powerful document editing and collaboration in your app or environment. Ultimate security, API and 30+ ready connectors, SaaS or on-premises

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts