-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
My question is more related to the variance of the batchs. If one batch contains samples that skip a connection and samples that do not ('row' mode in the Torchvision implementation), even if the values are ajusted to preserve the expected value, the variance will be much higher because we have in practice two distributions (for x_n and x_n + f(x_n)/p), which will mess up with the update of the batch normalization. Also, at inference time, all forward passes will be done as x_{n+1} = x_n + f(x_n), which has a different variance. The torchvision implementation also offers a 'batch' mode that kinda reduce this issue (because the global variance computed this way will be the mean of both distribution variances, instead of the variance of the joint distribution) but it does not seem to be the default mode (it does not even exist in the timm implementation).
Found relevant code at https://github.com/yueatsprograms/Stochastic_Depth + all code implementations here