spidermon
robotmk
spidermon | robotmk | |
---|---|---|
2 | 1 | |
510 | 52 | |
0.4% | - | |
6.9 | 9.6 | |
10 days ago | 4 days ago | |
Python | Rust | |
BSD 3-clause "New" or "Revised" License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
spidermon
-
Automated testing the scraping output
This is what Spidermon does.
- spidermon: Scrapy Extension for monitoring spiders execution
robotmk
-
Real-Time Performance Monitoring Software
We are using checkmk to monitor 1000s of locations in terms of reachability and performance. Basically you use the checkmk (www.checkmk.com) server as a central site that pings your network devices and your servers (both is important since a slow server doesn't mean "the network is so slow(tm)" ) In Checkmk you get nice graphs but you can also export your data to grafana (which is what we do) to build a smokeping like expirience. Smokeping is a nice tool, but it's rather old and does not scale too well. Checks from your network devices can be implemented using ipsla (cisco?). Theres a plugin for that: https://checkmk.com/de/integrations/cisco\_ip\_sla. If you want to monitor stuff from a (near) user perspective: Check MK supports a distributed setup that allows you to place sensors in differenent locations (if you don't want to implement the full end-to-end monitoring using something like Robot-Framework (https://github.com/simonmeggle/robotmk). If you want deeper network visibility then you could pair checkmk with ntopng (https://www.ntop.org/products/traffic-analysis/ntop/). This way you'll get a lot more than plain RTT and network interface load like ipfix, dpi, *flow, etc...
What are some alternatives?
undetected-chromedriver - Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
infracheck - Incredibly elastic and lightweight health check endpoint to cover ANY CASE, including infrastructure as well as applications
scrapeops-scrapy-sdk - Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
pytest-testinfra - Testinfra test your infrastructures
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
check-zfs-replication - This script checks yout ZFS replication an generates reports in different flavours or can act as checkmk agent plugin (local check).
Grab - Web Scraping Framework
Nagstamon - Nagios status monitor for your desktop.
estela - estela, an elastic web scraping cluster 🕸
prometheus - The Prometheus monitoring system and time series database.