bert VS aws-cloudformation-coverage-roadmap

task-specific parameters, and is trained on the downstream tasks by simply fine-tuning all pre-trained parameters.

[0] https://arxiv.org/abs/1810.04805

The release of BERT in 2018 kicked off the language model revolution. The Transformers architecture succeeded RNNs and LSTMs to become the architecture of choice. Unbelievable progress was made in a number of areas: summarization, translation, text classification, entity classification and more. 2023 tooks things to another level with the rise of large language models (LLMs). Models with billions of parameters showed an amazing ability to generate coherent dialogue.

The general idea is that you have a particular task & dataset, and you optimize these vectors to maximize that task. So the properties of these vectors - what information is retained and what is left out during the 'compression' - are effectively determined by that task.

In general, the core task for the various "LLM tools" involves prediction of a hidden word, trained on very large quantities of real text - thus also mirroring whatever structure (linguistic, syntactic, semantic, factual, social bias, etc) exists there.

If you want to see how the sausage is made and look at the actual algorithms, then the key two approaches to read up on would probably be Mikolov's word2vec (https://arxiv.org/abs/1301.3781) with the CBOW (Continuous Bag of Words) and Continuous Skip-Gram Model, which are based on relatively simple math optimization, and then on the BERT (https://arxiv.org/abs/1810.04805) structure which does a conceptually similar thing but with a large neural network that can learn more from the same data. For both of them, you can either read the original papers or look up blog posts or videos that explain them, different people have different preferences on how readable academic papers are.

Could you post a link to "the BERT paper"? I've read some, but would be interested reading anything that anyone considered definitive :) Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" :https://arxiv.org/abs/1810.04805

Rust crate rust_bert implementation of the BERT language model (https://arxiv.org/abs/1810.04805 Devlin, Chang, Lee, Toutanova, 2018). The base model is implemented in the bert_model::BertModel struct. Several language model heads have also been implemented, including:

The achievement of training a BERT model to 90% of the GLUE score on a single GPU in ~100 hours is indeed impressive. As for the original BERT pretraining run, the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model and 4 days on 64 TPU chips for the BERT-Large model.

Regarding the translation of these techniques to the pretraining phase for a GPT model, it is possible that some of the optimizations and techniques used for BERT could be applied to GPT as well. However, the specific architecture and training objectives of GPT might require different approaches or additional optimizations.

As for the SOPHIA optimizer, it is designed to improve the training of deep learning models by adaptively adjusting the learning rate and momentum. According to the paper [2], SOPHIA has shown promising results in various deep learning tasks. It is possible that the SOPHIA optimizer could help improve the training of BERT and GPT models, but further research and experimentation would be needed to confirm its effectiveness in these specific cases.

[1] https://arxiv.org/abs/1810.04805

Click to Learn more...

Given AWS CloudFormation is AWS's native language and service for infrastructure as code, you will likely find more official quickstarts provided by AWS in the language. In addition to this, AWS Support will probably be more capable of assisting you with issues when you need help. AWS Support is essential for large enterprises, particularly those new to the cloud or slow to adopt. These types of organizations may have a skill gap within their organization regarding their cloud skill set, and in turn, they are more likely to use AWS Enterprise Support.

Today, I will show you how to build Amazon Location Service, which allows you to build location-based applications within your AWS environment using AWS Cloud Development Kit (AWS CDK) and AWS CloudFormation. I will also show examples of the recently popular CDK Migrate and AWS CloudFormation IaC generator.

AWS CloudFormation: Speed up cloud provisioning with infrastructure as code.

Infrastructure as Code (IaC) is an important part of any true hosting operation in the public cloud. Each of these platforms has their own IaC solution, e.g. AWS CloudFormation. But they also support popular open-source IaC tools like Pulumi or Terraform. A category of tools that also needs to be discussed is API gateways and other app-specific load balancers. There are applications for internal consumption, which can be called microservices if you have a lot of them. And often microservices use advanced networking options such as a service mesh instead of just the native private network offered by a VPC.

Cloudformation (IaC) does not need to be introduced to anyone, plus if you read the previous blogpost, the terraform provider (CC) we used is based on Cloudformation. Moreover, you will notice a lot of similarities, after all, we are implementing the same scenario, but with a different tool.

AWS Application Composer launched in the AWS Console at re:Invent one year ago, and this re:Invent it expanded to the VS Code IDE as part of the AWS Toolkit - but that’s not the only exciting part. When using App Composer in the IDE, users also get access to a generative AI partner that will help them write infrastructure as code (IaC) for all 1100+ AWS CloudFormation resources that Application Composer now supports.

CloudFormation

Of course, best practices today dictate that we should be deploying our infrastructure as code, using tools such as CloudFormation or Terraform.

If you're provisioning the above resources for the first time, you'll have to either configure Terraform to use specific AWS keys as you won't have OIDC connection yet. In my case, I chose to have those pre-requesites resources in a CloudFormation template and deploy them with StackSets.