secutils VS secutils-web-scraper

As a solo-developer for Secutils.dev, I operate with very limited resources and cannot accommodate every user's feature request, even if I wish to. On the other hand, prioritizing and developing features based on assumptions and limited upfront user feedback has its own challenges and risks. That's why, right from the start, I've been considering adding some sort of "extension points" into Secutils.dev that would allow users to customize the certain behavior of the utilities according to their needs.

After a refreshing winter-time blogging-break, I'd like to resume introducing new features of Secutils.dev through practical use cases. Ever wondered how to easily track something on the internet that does not offer subscribing to updates natively? If so, let me introduce you a recently released web content tracking utility that made its debut in December 2023 (v1.0.0-alpha.4). I'll explain how I use it for various tasks, well beyond its primary security focus. Additionally, I'll cover how it's made in case you're interested in developing a similar tool yourself. Read on!

Like anything we invest our time and energy in, I want to raise awareness about the work I've done, gauge interest, and hopefully receive constructive feedback. I'm not a fan of blunt self-promotion, so I'm going to try something different - demonstrating new features in action. Sometimes I'll show their business value, and other times it'll just be for fun and entertainment. In this post, I'll demonstrate how to use the new “Import content security policy” feature to learn a bit more about the websites you use every day. Let's dive in!

Last week, I kicked off the new "Q4 2023 – Oct-Dec" development and research iteration for Secutils.dev, the open-source toolbox designed for developing and testing secure applications. In this post, I'll take you through the significant features and changes that will be the focus of my work in the coming weeks and months: tracking arbitrary web content, user-specific webhook subdomains, inherited CSP, and more. Let's dive in!

Earlier this week, I wrapped up the "Q3 2023 – Jul-Sep" iteration and cut a new 1.0.0-alpha.3 release of Secutils.dev. In this post, I would like to quickly walk you through the major changes since 1.0.0-alpha.2: notifications, more powerful web page resource tracker, sharing capabilities and more. Let’s dive in!

In my previous post, I shared the update regarding the upcoming "Q3 2023 - Jul-Sep" milestone. While I briefly covered how I implemented the notifications subsystem in Secutils.dev, there are a few other important changes I've been working on for this milestone. One of these changes is related to the fact that I’m preparing to allow Secutils.dev users to inject custom JavaScript scripts into the web pages they track resources for (yay 🎉). As a result, I've spent some time hardening the Web Scraper environment's security and wanted to share what you should keep in mind if you’re building a service that needs to scrape arbitrary web pages.

Notifications, in general, and email notifications, specifically, are integral to any product that involves any monitoring or tracking activities. Secutils.dev already includes, and will continue to expand, features that require the ability to send notifications. Two notable examples include sending notifications for changes detected by the web page resources trackers and changes detected in the tracked content security policies (CSP).

ℹ️ ASK: If you found this post helpful or interesting, please consider showing your support by starring secutils-dev/secutils GitHub repository.

This newsletter is like JavaScript Weekly but for Rust! It provides updates from the Rust community, highlights updates from the vital Rust projects, shares new ideas and thoughts, offers guides, updates from the Rust core teams, and, sometimes, introduces brand new Rust crates. Rust is my passion and the primary programming language I use for the Secutils.dev Server. Every time I read this newsletter, I discover something new and exciting about Rust!

This is the second part of my reflection sparked by the recent “2023 State of Open Source Security” report from Snyk. It got me thinking about the price we pay for false positives in software security. In my previous post, “The Cost of False Positives in Software Security, Part 1: Small Applications”, I talked about how true and false positives affect smaller applications like Secutils.dev. Now, I want to take the same idea and apply it to a much larger software that’s a big part of my daily work: Kibana.

To begin, all functionality related to browser automation and web scraping lives in a dedicated service — Web Scraper. The primary rationale is that dealing with browsers and arbitrary user scripts is tricky from a security standpoint, and it's always a good idea to isolate such functionality as much as possible. You can read more about the security aspects of web scraping in the "Running web scraping service securely" post.

When it comes to web page resource scraping, Secutils.dev relies on a separate component - secutils-dev/secutils-web-scraper. I've built it on top of Playwright since I need to handle both resources that are statically defined in the HTML and those that are loaded dynamically. Leveraging Playwright, backed by a real browser, instead of parsing the static HTML opens up a ton of opportunities to turn a simple web resource scraper into a much more intelligent tool capable of handling all sorts of use cases: recording and replaying HARs, imitating user activity, and more.

While both Puppeteer and Playwright have their own advantages and disadvantages, I have chosen Playwright for Secutils.dev. Playwright not only allows us to access all browser APIs within the web page context to easily detect and extract inline resources, but also enables us to intercept all external dynamically loaded web page resources. Here's an example of the code (full code can be found here):