Minimal PyTorch re-implementation of GPT

• minGPT

  35 18,932 0.0 Python

  A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

• iris

  8 756 1.9 Python

  Transformers are Sample-Efficient World Models. ICLR 2023, notable top 5%. (by eloialonso)

This is actually a pretty neat, self-contained implementation that can super easily extended beyond stereotypical natural language models, for example to create world models for video games [1] or to create robot models that can learn to imitate from large, chaotic human demonstration data [2] (disclaimer, I'm an author on the second one.) Basically, GPT (or minGPT) models are EXCELLENT sequence modelers, almost to the point where you can throw any sensible sequence data at it and hope to get interesting results, as long as you don't overfit.

Even though I have only been working on machine learning for around six years, it's crazy to see how the landscape has changed so fast so recently, including diffusion models and transformers. It's not too much to say that we might expect more major breakthroughs by the end of this decade, and end in a place we can't even imagine right now!

[1] https://github.com/eloialonso/iris

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• bet

  3 92 2.1 Python

  Code and website for Behavior Transformers: Cloning k modes with one stone.

• machine-learning-articles

  5 3,108 4.1

  🧠💬 Articles I wrote about machine learning, archived from MachineCurve.com.

For anyone else who was new to the phrase "isotropic model":

https://github.com/christianversloot/machine-learning-articl...

• machine-learning-articl

  1 - -

For anyone else who was new to the phrase "isotropic model":

https://github.com/christianversloot/machine-learning-articl...

• returnn-experiments

  2 152 6.4 Python

  experiments with RETURNN

  

This works for an architecture which has been well tuned and studied before, like LSTM or Transformer.

Once you do research on the model, testing out things, it often tends to become such kwarg monster in many frameworks.

Having everything (relevant) in one file (even in the config file itself with hyper params) allows you to copy the file for every experiment and modify it inplace. This avoids the kwargs mess. But then the config files are very complex, and can become messy in other ways (esp for research projects). Example: https://github.com/rwth-i6/returnn-experiments/blob/master/2...

Such approach makes it much more flexible and does not mess with the baseline code. As you say, it's more like an evolutionary DNA-like approach, where you then tend to do crossovers with other evolved good-performing configs, etc.

Suggest a related project