pythonic-cv VS interactive-projectivity-open

I did something similar in this example, where the “input” was someone covering up a webcam to switch to the next display source (in my case switching through a list of videos that would continue from where they left off).

If you’re wanting to continuously process your screen like a live video stream you might be interested in (my library) pythonic-cv, which supports MSS as a video input backend.

I’ve previously done something similar here, but the transition was triggered by covering a webcam (e.g. with a finger).

pythonic-cv (disclaimer: my library) includes MSS as an input stream option.

If that’s of interest, OpenCV provides a detailed stitching example, although my revision is likely a fair amount easier to follow and understand (but it’s been optimised for video, so if you’re wanting to use it with minimal modification you’ll need to provide your images in a sequence where each image has overlap with the one before it, not in a random order).

As something of a side note, you may be interested in (my library) pythonic-cv.

I wrote pythonic-cv because I found that pipelines regularly require pre and post processing that can be done in parallel across frames - you might want to take a look :-)

Likely areas for parallelisation depend on the operations that are happening - if there are independent stages then they can often be made to run at the same time in separate threads (or processes). Concurrency is similar, although more about doing something else while waiting for I/O, and can generally be solved with threads or asyncio coroutines. A common improvement for video-focused computer vision pipelines is reading in/capturing the next frame while the previous frame is being processed (e.g. like is done with pythonic-cv - disclaimer: my library), but given the frame rates you specified then that may help a bit but is likely not the main culprit. ML-based algorithms can often benefit from the inherent parallelisation of a GPU-based implementation, although that can be difficult or even impossible to achieve depending on the algorithm being used.

Personally I’d approach this using pythonic-cv - it has a VideoReader class that supports separate preprocess and process functions, and has threading built in. Then again, I wrote the library, so it’s not so surprising that I’d jump to using it.

I used an Extended Kalman Filter in my thesis. You can look in the Compensation section (3.5) for the details and a brief explanation of what I chose to do, and the last two references in the bibliography are the most useful tutorials/explanations of a Kalman filter that I found while learning about them originally. My relevant code is in code/interaction_detector.py.

In case it’s useful, here’s my Kalman implementation from my thesis. The tutorial I started off with is linked in the class docstring, and hopefully the code comments are helpful in figuring out what’s going on.