syntaxdot VS jetson-inference

I am so happy about them releasing this. A few years ago I wrote a multi-task syntax annotator in Rust using Laurent Mazare's excellent tch-rs binding (it seems like he is also working on Candle):

https://github.com/tensordot/syntaxdot

However, the deployment story was always quite difficult. The PyTorch C++ API is not stable, so a particular version of tch-rs will only work with a particular PyTorch version. So, anyone wanting to use SyntaxDot always had to get exactly the right version of libtorch (and set some environment variables) to build the project.

The idea of making an abstraction over Torch and Rust ndarray (similar to Burn) crossed my mind several times, but there is only so much that I could do as a solo developer. So Candle would be a god-given if I was still working on this project.

Seeing Candle wants to make me port curated-transformers to Candle for fun:

https://github.com/explosion/curated-transformers

They are obviously not as good as in Python, but if you are willing to invest time, it's definitely doable. E.g. I made a multi-task transformer-based syntax annotator in Rust using the tch Torch binding:

https://github.com/tensordot/syntaxdot

In my current job, I do NLP with Python, Cython, and some C++. I don't think doing it in Rust was much more work. Once you are beyond the stage of implementing a small research project or toy model, most systems are going to contain a lot of custom, specialized code. You will have to do that work in any language.

What I like about PyTorch is that most of the functionality is actually available through the C++ API as well, which has 'beta API stability' as they call it. So, there are good bindings for some other languages as well. E.g., I have been using the Rust bindings in a larger project [1], and they have been awesome. A precursor to the project was implemented using Tensorflow, which was a world of pain.

Even things like mixed-precision training are fairly easy to do through the API.

[1] https://github.com/tensordot/syntaxdot

Huggingface fills the need for task based prediction when you have a GPU.

With model distillation, it should be possible to annotate hundreds of sentences per second on a single CPU with a library like Huggingface Transformers.

For instance, one of my distilled Dutch multi-task syntax models (UD POS, language-specific POS, lemmatization, morphology, dependency parsing) annotates 316 sentences per second with 4 threads on a Ryzen 3700X. This distilled model has virtually no loss in accuracy, compared to the finetuned XLM-RoBERTa base model.

I don't use Huggingface Transformers, but ported some of their implementations to Rust [1], but that should not make a big difference since all the heavy lifting happens in C++ in libtorch anyway.

tl;dr: it is not true that tranformers are only useful for GPU prediction. You can get high CPU prediction speeds with some tricks (distillation, length-based bucketing in batches, etc.).

[1] https://github.com/tensordot/syntaxdot/tree/main/syntaxdot-t...

Jetson Nano’s are obsolete and no longer supported; but to answer your question, this might be a good place to start.

I’m trying to do the tutorial, where they retrain the neural network to detect fruits (jetson-inference/pytorch-ssd.md at master · dusty-nv/jetson-inference · GitHub 1)

Jetson-Inference is another amazing resource to get started on. This will allow you to try out a number of neural networks (classification, detection, and segmentation) all with your own data or with sample images included in the repo.

Hello! I'm looking for some pre trained image classification models to use on a Jetson Nano. I already know about the model zoo and the pre trained models included in the https://github.com/dusty-nv/jetson-inference repo. For demonstration purposes, however, I need a model trained on small objects from the context of production, ideally nuts, bolts, and similar small objects. Does anyone happen to know a source for this? Thanks a lot!

> They provide some SSD-Mobilenet-v2 here: https://github.com/dusty-nv/jetson-inference

I was aware of that repository but from taking a cursory look at it I had thought dusty was just converting models from PyTorch to TensorRT, like here[0, 1]. Am I missing something?

> I get 140 fps on a Xavier NX

That really is impressive. Holy shit.

[0]: https://github.com/dusty-nv/jetson-inference/blob/master/doc...

[1]: https://github.com/dusty-nv/jetson-inference/issues/896#issu...

If you have a screen and keyboard and mouse plugged into the Nano, I would recommend starting with Hello AI World on https://github.com/dusty-nv/jetson-inference#hello-ai-world

Well, don't get me wrong. I do like my Jetson Nano. For a hobbyist who likes to tinker with machine learning in their spare time it's definitely a product cool and there are quite a few repositories on Github[0, 1] with sample code.

Unfortunately… that's about it. There is little documentation about

- how to build a custom OS image (necessary if you're thinking about using Jetson as part of your own product, i.e. a large-scale deployment). What proprietary drivers and libraries do I need to install? Nvidia basically says, here's a Ubuntu image with the usual GUI, complete driver stack and everything – take it or leave it. Unfortunately, the GUI alone is eating up a lot of the precious CPU and GPU resources, so using that OS image is no option.

- how deployment works on production modules (as opposed to the non-production module in the Developer Kit)

- what production modules are available in the first place ("Please refer to our partners")

- what wifi dongles are compatible (the most recent Jetson Nano comes w/o wifi)

- how to convert your custom models to TensorRT, what you need to pay attention to etc. (The official docs basically say: Have a look at the following nondescript sample code. Good luck.)

- … (I'm sure I'm forgetting many other things that I've struggled with over the past months)

Anyway. It's not that this information isn't out there somewhere in some blog post, some Github repo or some thread on the Nvidia forums[2]. (Though I have yet to find a reliably working wifi dongle…) But it usually takes you days orweeks to find it. From a product which is supposed to be industry-grade I would have expected more.

[0]: https://github.com/dusty-nv/jetson-inference

https://github.com/dusty-nv/jetson-inference#system-setup