Spidey VS changedetection.io

Didn't get a chance to reply earlier but depending on what you're trying to do, you might want a web crawler. I have a crawler on Github that I built for scraping in instances where someone doesn't have an API. If you go this route, I suggest doing it as a background task and go off cached data.

This may be overkill but I have library out there for building web crawlers. Spidey is the library. I'm not suggesting you use it but you could look at it for ideas. It uses a multithreaded, producer/consumer approach that avoids recursion and stack overflow issues. Use a queue, pull from the queue for each url, push new urls on when you find them. Do need to optimize my code a bit more but if it helps at all. But your issue is most likely the fact that you're finding a link to the page you are currently on. HashSet or List of found URLs would solve the issue.

I use ChangeDetection,

- https://changedetection.io/#features

- https://github.com/dgtlmoon/changedetection.io

changedetection.io just donated to the awesome crew over at ntfy.sh

I used a business to business website that lists every single company in this industry by location and has a link for each to generate a list of companys and URLS. I monitor this for changes with changedetection.io

0.45.8 is out! Support for xPath2.0/3.0 means you can use logic in your selectors (for example, give the count of the number of divs that contain certain text, and much more). UI improvements, Selectable browser (plug it into Bright Data's scraping browser!) and more, check it out! Much ❤ from the changedetection team. https://github.com/dgtlmoon/changedetection.io / changedetection.io

The wiki is pretty easy to follow. The instructions here are the ones I followed. Simply uncommented the lines mentioned.

Hello, Is there a free alternative to this website changedetection.io ?

hey all - I noticed after showing our homepage ( changedetection.io ) to a lot of friends at the bar, they all click on the picture/screenshot at the top expecting a short video, but theres nothing there :) I remember there was some service that can build one for you? Or is it better to make it your self.. ? if so.. which tech do you use? (linux here)

Love from the changedetection.io crew ❤❤❤

You can use an open-source tool like this one: https://github.com/dgtlmoon/changedetection.io