How we created an in-browser BERT attention visualiser without a server - TrAVis: Transformer Attention Visualiser

1. bart-base-jax

   1 1 30 1.4 Python

   JAX implementation of the bart-base model

   

   Firstly, we implemented the BART model from scratch using JAX. We chose JAX because it is an amazing deep learning framework that enables us to write clear source code, and it can be easily converted to NumPy, which can be executed in-browser. We chose the #BART model because it is a complete encoder-decoder model, so it can be easily adapted to other models, such as BERT, by simply taking a subset of the source code.

2. Judoscale

   judoscale.com featured

   Save 47% on cloud hosting with autoscaling that just works. Judoscale integrates with Django, FastAPI, Celery, and RQ to make autoscaling easy and reliable. Save big, and say goodbye to request timeouts and backed-up task queues.

   

3. jax

   2 89 32,032 10.0 Python

   Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

   

   Firstly, we implemented the BART model from scratch using JAX. We chose JAX because it is an amazing deep learning framework that enables us to write clear source code, and it can be easily converted to NumPy, which can be executed in-browser. We chose the #BART model because it is a complete encoder-decoder model, so it can be easily adapted to other models, such as BERT, by simply taking a subset of the source code.

4. d3

   3 293 110,438 6.3 Shell

   Bring data to life with SVG, Canvas and HTML. :bar_chart::chart_with_upwards_trend::tada:

   

   Fourthly, we visualise the attention matrices in our web application using d3.js.

5. word-piece-tokenizer

   4 1 6 3.8 Python

   A Lightweight Word Piece Tokenizer

   

   Secondly, we implemented the HuggingFace BERT Tokeniser in pure Python, as it can be more easily executed in-browser. Moreover, we have optimised the tokenisation algorithm, which is faster than the original HuggingFace implementation.

6. TrAVis

   5 1 56 10.0 Python

   TrAVis: Visualise BERT attention in your browser (by ayaka14732)

   

7. bert

   6 55 39,057 0.0 Python

   TensorFlow code and pre-trained models for BERT

   

   In the BERT Base Uncased model, for example, there are 12 transformer layers, each layer contains 12 heads, and each head generates one attention matrix. TrAVis is the tool for visualising these attention matrices.

Suggest a related project