esm VS typeguard

When evaluating this work, it’s important to remember that the functional labels on each of the 290 million input sequences were originally assigned by HMM as part of the pfam project, so the model is predicting a prediction.

Furthermore, the authors must engage a lot of human curation to ensure the sequences they generate are active. First, they pick an easy target. Second, they employ by-hand classical bioinformatics techniques on their predicted sequences after they are generated. For example, they manually align them and select those which contain specific important amino acids at specific positions which are present in 100% of functional proteins of that class, and are required for function. This is all done by a human bioinformatics expert before they test the “generated” sequences.

One other comment, in protein science, a sequence with 40% identity to another sequence is not “very different” if it is homologous. Since this model is essentially generating homologs from a particular class, it’s no surprise at a pairwise amino acid level, the generated sequences have this degree of similarity. Take proteins in any functional family and compare them. They will have the same overall 3-D structure—called their “fold”—yet have pairwise sequence identities much lower than 30–40%.

Not to be negative. I really enjoyed reading this paper and I think the work is important. Some related work by Meta AI is the ESM series of models [1] trained on the same data (the UniProt dataset [2]).

One thing I wonder is about the vocabulary size of this model. The number of tokens is 26 for the 20 amino acids and some extras, whereas for a LLM like Meta’s LLaMa the vocab size is 32,000. I wonder how that changes training and inference, and how we can adopt the transformer architecture for this scenario.

1. https://github.com/facebookresearch/esm

2. https://www.uniprot.org/help/downloads

What's your input data look like? If you're predicting structures of mutants where there's a wild type structure available you can use variant prediction tools like ESM-IF or some of the protein language models like ESM-2

Such an explosion of protein AI lately. It’s the absolute best time to be a protein scientist with an interest in ML. Every new model type is inevitably tried out on proteins. In this case, by grad students at a very famous protein design lab (Baker Lab at University of Washington). And they usually find some interesting application. Protein design presents tons of interesting challenges.

The very largest plain transformer models trained on protein sequences (analogous to plain text) are about 15B parameters (I am thinking of Meta AI’s ESM-2 [1]). These can do for protein sequences what LLMs do for text (that is, they can “fill in the blank” to design variations, generate new proteins that look like their training data—which consists of all natural protein sequences), and tell you how likely it is that a given sequence exists.

Some cool variations of transformers have applications for protein design, like the now-famous SE(3) equivariant transformer used in the structure prediction module of AlphaFold [2], now appearing in TFA

1. https://github.com/facebookresearch/esm

Likewise PyTorch is seeing a lot of sciml work, in particular to do with protein design. (See e.g. ESM2.)

As other folks have commented, type hints are now a big deal. For static typing the best checker is pyright. For runtime checking there is typeguard and beartype. These can be integrated with array libraries through jaxtyping. (Which also works for PyTorch/numpy/etc., despite the name.)

I got good use of the run-time type checking of typeguard [0] when I recently invoked it via its pytest plugin [2]. For all code visited in the test suite, you get a failing test whenever an actual type differs from an annotated type.

[0]: https://github.com/agronholm/typeguard/

[1]: https://typeguard.readthedocs.io/en/latest/userguide.html#us...

But you can use a library like typeguard to get runtime typechecking. Or run mypy over the code to get static typechecking.

Every point in this blog post strikes me as either (1) unaware of the tooling around python typing other than mypy, or (2) a criticism of static-typing-bolted-on-to-a-dynamically-typed-language, rather than Python's hints. Regarding (1), my advise to OP is to try out Pyright, Pydantic, and Typeguard. Pyright, especailly, is amazing and makes the process of working with type hints 2 or 3 times smoother IMO. And, I don't think points that fall under (2) are fair criticisms of type *hints*. They are called hints for a reason.

Otherwise, here's a point-by-point response, either recommending OP checks out tooling, or showing that the point being made is not specific to Python.

> type hints are not binding.

There are projects [0][1] that allow you to enforce type hints at runtime if you so choose.

It's worth mentioning that this is very analogous to how Typescript does it, in that type info is erased completely at runtime.

> Type checking is your job after all, ...[and that] requires maintenance.

There are LSPs like Pyright[2] (pyright specifically is the absolute best, IMO) that report type errors as you code. Again, this is very very similar to typescript.

> There is an Any type and it renders everything useless

I have never seen a static-typing tool that was bolted on to a dynamically typed language, without an `Any` type, including typescript.

> Duck type compatibility of int and float

The author admits that they cannot state why this behavior is problematic, except for saying that it's "ambiguous".

> Most projects need third-party type hints

Again, this is a criticism of all cases where static types are bolted on dynamically typed languages, not Python's implementation specifically.

> Sadly, dataclasses ignore type hints as well

Pydantic[3] is an amazing data parsing library that takes advantage of type hints, and it's interface is a superset of that of dataclasses. What's more, it underpins FastAPI[4], an amazing API-backend framework (with 44K Github stars).

> Type inference and lazy programmers

The argument of this section boils down to using `Any` as a generic argument not being an error by default. This is configurable to be an error both in Pyright[5], and mypy[6].

> Exceptions are not covered [like Java]

I can't find the interview/presentation, but Guido Van Rossum specifically calls out Java's implementation of "exception annotations" as a demonstration of why that is a bad idea, and that it would never happen in Python. I'm not saying Guido's opinion is the absolute truth, but just letting you know that this is an explicit decision, not an unwanted shortcoming.

[0] https://github.com/RussBaz/enforce

[1] https://github.com/agronholm/typeguard

[2] https://github.com/microsoft/pyright

[3] https://pydantic-docs.helpmanual.io

[4] https://github.com/tiangolo/fastapi

[5] https://github.com/microsoft/pyright/blob/main/docs/configur...

[6] https://mypy.readthedocs.io/en/stable/config_file.html#confv...

Personally I find working around mypy's quirks to be more effort than it's worth, so to offer another option: typeguard or beartype can be used to perform run-time type checking.

Never checked? They're statically checked.

Also, tooling like https://pydantic-docs.helpmanual.io/ can do runtime checking for important parts of your app or you can add use this https://github.com/agronholm/typeguard to enforce all types at runtime (although I haven't measured the performance impact, probably something to do in a separate environment than production?).

typeguard