dtplyr VS darts

Can’t say I’ve used it, but isn’t that what dtplyr is supposed to provide?

https://dtplyr.tidyverse.org/

# I am trying to follow the example at https://dtplyr.tidyverse.org/

FYI the developer of tidytable has been developing dtplyr for the Tidyverse. You might like that too!

If you need speed, you should consider using dtplyr (or tidytable), or even dbplyr with duckdb.

https://dtplyr.tidyverse.org/ ?

If you want data.table speed, but using dplyr/tidy then dtplyr is a good package to have handy. Personally I love R, and choose R + NodeJS as my gotos for everything I do, and use Python only when I have to.

Also, that dataset is quite big, and the "base" Tidyverse will be excessively slow. You should supplement the "base" Tidyverse packages (i.e. dplyr and tidyr) with either dtplyr or dbplyr (+ duckDB). I'd suggest starting with dtplyr, which should handle 10M+ rows fine.

If you want, there's a "substitute" for dplyr called dtplyr (also part of the Tidyverse), which "translates" your dplyr/tidyr code into data.table behind the scenes, and allows you to make your modifications apply directly to the original dataset by default:

Install the dtplyr package and change your code to:

Get the best of both worlds (tidyverse + data.tables) with dtplyr, a data.table backend for dplyr.

Don't use an LSTM. Get up to date with SoTA methods and read the papers in the field. LSTMs are not the way forward. Read the papers I suggested. It would be very useful to come to grips with both the Time Series Repository (https://github.com/thuml/Time-Series-Library) and Darts (https://github.com/unit8co/darts) as these are widely used for research and in industry.

As others have pointed out, Prophet is not a particularly good model for forecasting, and has been superseded by a multitude of other models. If you want to do time series forecasting, I'd recommend using Darts: https://github.com/unit8co/darts. Darts implements a wide range of models and is fairly easy to use.

The problem with time series forecasting in general is that they make a lot of assumptions on the shape of your data, and you'll find you're spending a lot of time figuring out mutating your data. For example, they expect that your data comes at a very regular interval. This is fine if it's, say, the data from a weather station. This doesn't work well in clinical settings (imagine a patient admitted into the ER -- there is a burst of data, followed by no data).

That said, there's some interesting stuff out there that I've been experimenting with that seems to be more tolerant of irregular time series and can be quite useful. If you're interested in exchanging ideas, drop me a line (email in my profile).

3. darts

Looking forward to checking this out! How does this compare with darts[1]?

[1] https://unit8co.github.io/darts/

Darts is an open-source Python library by Unit8 for easy handling, pre-processing, and forecasting of time series. It contains an array of models, from standard statistical models such as ARIMA to deep neural networks. https://unit8co.github.io/darts/

active support

The new Forecast menu, featuring the open-source Darts Time Series library, offers script-friendly functionality. It's also easy to use. Don't have any data to load yet? Enter through the Stocks or Crypto menus.

I would recommend Darts in Python [1]. It's easy to use (think fit()/predict()) and includes

* Statistical models (ETS, (V)ARIMA(X), etc)

* ML models (sklearn models, LGBM, etc)

* Many recent deep learning models (N-BEATS, TFT, etc)

* Seamlessly works on multi-dimensional series

* Models can be trained on multiple series

* Many models offer rich support for probabilistic forecasts

* Model evaluation is easy: Darts has many metrics, offers backtest etc

* Deep learning scales to large datasets, using GPUs, TPUs, etc

* There's even now an explainability module for some of the models - showing you what matters for computing the forecasts

* (coming soon): an anomaly detection module :)

* (also, it even include FB Prophet if you really want to use it)

Warning: I'm probably biased because I'm Darts creator.

[1] https://github.com/unit8co/darts