v4l2rtspserver VS hls4ml

If you want to stream your webcam as an H264 encoded rtsp stream, it can be done with either rtsp-simple-server or V4l2rtspserver And it is possible to stream a linux desktop (and any program that might be open, like your web browser with the gcode viewer running on it) with the V4l2Loopback device and either of the two prgrams above. It is quite an involved process, but it is do-able.

v4l2rtspserver is about the only thing I've found for running a raspberry pi with the internal cable camera

I see that someone has already suggested hls4ml. I second that opinion. From my experience, it is extremely well documented. They have published papers which explain the scientific background. They have a really nice git page where they explain all the features of their tool. Additionally they also have an easy to follow tutorial of doing it from scratch using tensorflow networks. You can find all the information herehls4ml.

Open source project that originated at Fermilab https://github.com/fastmachinelearning/hls4ml (based on Xilinx Vivado which has been replaced by Vitis)

You may check the hls4ml framework: it's a "translator" from the ML model (Keras, PyTorch) to a synthesizable High-Level Synthesis (HLS) IP Core.

It really depends on the type of data you are using. There may (or may not) be some trade offs and sacrifices. There are frameworks which can basically translate your neural network information from a high level python code into equivalent HLS code which is optimized for low latency when inferred on FPGAs. Some frameworks which might be useful for you to explore are hls4ml and finn. These are some frameworks which can achieve low latency inference of neural networks on FPGAs using Xilinx Vitis HLS. These are what I found when I did a similar experiment but with much lower latency target (a few hundred ns) and a very simple MLP with 1D signal as input which was a year ago. Not sure if there are better alternatives available as of 2023. But conceptually all these work on the primary principle of having a supporting framework/methodology to first quantize the network and limit the precision of data to fixed point. The HLS then produced will also be a result of the framework applying dataflow techniques such that the resulting HLS code will produce an RTL which has the best overall latency.

Check hls4ml. Developed by CERN for fast classification in FPGA for high-energy physics experiments.

I would check out hls4ml. It's an open source project made by/for people at CERN to convert neural networks created in Python using QKeras (a quantization extension of Keras) into HLS, with Vivado HLS being the most well supported. There are some caveats though, and a fellow student and I have had trouble getting the generated HLS to match the Keras model and be feasible to synthesize, but it seems to work well for smaller neural networks.

Since you're looking for PL-only implementation, HLS4ML may fit your needs. It was developed to port TensorFlow models directly to FPGAs in particle physics experiments. Current development allows for implementation on SoC and MPSoC, though.