Our great sponsors
-
-
marionette
Marionette is a test automation framework based on image and text recognition for .NET. (by asimmon)
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I started looking for UI testing solutions for desktop applications. I quickly discarded expensive UI testing suites. WinAppDriver was very interesting, but one of the requirements was to add some kind of identifiers on every UI element that we wanted to interact with. This would have required a huge effort from the team.
So, give it a try, and take a look at the sample project! It only works on Windows, because screen capture, mouse and keyboard interaction are only implemented for that OS so far.
In order to do so, I had to learn about the basics of image processing. I was able to find an image within another with a technique called template matching, and I used an OpenCV wrapper made for .NET. I will not go through the technical details here, but you can find the relevant source code in my GitHub repository.
In order to implement text recognition, I needed to use another library, Tesseract. It is an optical character recognition engine made by Google. There is also a .NET wrapper for this library. I had to use a few tricks to properly match arbitrary text with a library that is best suited for digitized books, and cleverly use page segmentation methods. You will find how I used Tesseract in this C# class.
In order to implement text recognition, I needed to use another library, Tesseract. It is an optical character recognition engine made by Google. There is also a .NET wrapper for this library. I had to use a few tricks to properly match arbitrary text with a library that is best suited for digitized books, and cleverly use page segmentation methods. You will find how I used Tesseract in this C# class.