Our great sponsors
-
tensor2tensor
Discontinued Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
PE(1, 3) = cos(1 / 10000^(2*1 / 4)) = cos(1 / 10000^.5) ≈ 1
I also wondered if these formulae were devised with 1-based indexing in mind (though I guess for larger dimensions it doesn't make much difference), as the paper states
> The wavelengths form a geometric progression from 2π to 10000 · 2π
That led me to this chain of PRs - https://github.com/tensorflow/tensor2tensor/pull/177 - turns out the original code was actually quite different to that stated in the paper. I guess slight variations in how you calculate this embedding doesn't affect things too much?
Related posts
- [P] Why the Original Transformer Figure Is Wrong, And Some Other Interesting Tidbits
- Why the Original Transformer LLM Figure Is Wrong, and Other Interesting Tidbits
- What Are Transformer Models and How Do They Work?
- [P] Why I quit my lucrative job at Google to start Vectara? (neural search as a service for developers everywhere).
- Alias-Free GAN