nut.js VS dogu

The number of Dislikes on that GitHub issue that the OP mentioned in the post has gone from 36 to the moon! https://github.com/nut-tree/nut.js/issues/577

Nut.js is great for this kind of thing, you could probably drop ahk with it

You could try nut.js. I am not sure if it includes any other dependencies, though.

One really cool little JS library I've been using for a bunch of desktop automation tasks lately is nut.js and the lower level libnut library it's implemented on top of:

https://github.com/nut-tree/nut.js

https://github.com/nut-tree/libnut

It provides a means to send user input (mouse movement/clicks and key presses) and read and react to changes in visual state (through screenshots), and works across Windows, Linux and MacOS. It automates at a much lower level of abstraction than the approaches mentioned in the article that script against programmatic APIs.

What I really like about this lower level approach is that you don't need to get anyone's permission to automate anything, since there's no programmatic API that the system owners has to provide for you and thus can limit or take away when it becomes inconvenient.

Any task that can be accomplished though looking at stuff on the screen and clicking the mouse and pressing keys on a keyboard (i.e. what a real person would do to accomplish the same task) can be automated, and it's actually surprisingly easy and effective to do this with nut.js. What really helps is that OpenCV has become ridiculously good and ridiculously fast at matching/identifying objects from a screenshot, with latencies usually in the low double digits, so latency-based flakiness isn't nearly as much of an issue as I remember it in the old days. I've also played around with OCR with tesseract but haven't had as much success with it in terms of perf, and remember seeing latencies of several seconds for even recognizing a single word from a tiny pre-cropped screenshot containing only the word itself.

The main tradeoff to this approach compared to automation through APIs is that because it works by simulating real user inputs, it's not very amenable to running in the background while a user is actively interacting with the same machine, so a separate machine or VM is often needed. That's an acceptable tradeoff for some use cases but complete deal breaker for others, so YMMV, but just wanted to bring this cool little tool to people's attention.

Dogu Technologies - Seamless Test Automation Platform

View on GitHub

Building DeviceFarm on-premise can solve the above problems. Here's an open source project to build Device Farm on-premise. => Dogu