How we created an in-browser BERT attention visualiser without a server - TrAVis: Transformer Attention Visualiser

• bart-base-jax

  1 27 1.4 Python

  JAX implementation of the bart-base model

Firstly, we implemented the BART model from scratch using JAX. We chose JAX because it is an amazing deep learning framework that enables us to write clear source code, and it can be easily converted to NumPy, which can be executed in-browser. We chose the #BART model because it is a complete encoder-decoder model, so it can be easily adapted to other models, such as BERT, by simply taking a subset of the source code.

• jax

  82 27,842 10.0 Python

  Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

  

Firstly, we implemented the BART model from scratch using JAX. We chose JAX because it is an amazing deep learning framework that enables us to write clear source code, and it can be easily converted to NumPy, which can be executed in-browser. We chose the #BART model because it is a complete encoder-decoder model, so it can be easily adapted to other models, such as BERT, by simply taking a subset of the source code.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• d3

  277 107,600 8.4 Shell

  Bring data to life with SVG, Canvas and HTML. :bar_chart::chart_with_upwards_trend::tada:

  

Fourthly, we visualise the attention matrices in our web application using d3.js.

• word-piece-tokenizer

  1 5 3.8 Python

  A Lightweight Word Piece Tokenizer

Secondly, we implemented the HuggingFace BERT Tokeniser in pure Python, as it can be more easily executed in-browser. Moreover, we have optimised the tokenisation algorithm, which is faster than the original HuggingFace implementation.

• TrAVis

  1 49 10.0 Python

  TrAVis: Visualise BERT attention in your browser (by ayaka14732)

• bert

  49 36,945 0.0 Python

  TensorFlow code and pre-trained models for BERT

  

In the BERT Base Uncased model, for example, there are 12 transformer layers, each layer contains 12 heads, and each head generates one attention matrix. TrAVis is the tool for visualising these attention matrices.

Suggest a related project