Cerberus
detect-headless
Our great sponsors
| Cerberus | detect-headless | |
|---|---|---|
| 3 | 1 | |
| 3,106 | 224 | |
| 0.5% | 8.0% | |
| 7.4 | 10.0 | |
| 6 months ago | almost 4 years ago | |
| Python | JavaScript | |
| ISC License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Cerberus
-
Show HN: Config-file-validator – CLI tool to validate all your config files
I was expecting this to validate the configuration files are also valid for their use cases, not just valid JSON, TOML, etc.
If you're looking for that and Python is your jam, the library cerberus[0] is very good at it.
[0]: https://github.com/pyeve/cerberus
-
Do you think we need an open-source web scraping monitoring tool?
I wrote scrapy-test as a proof of concept for validating live pages for scrapy spiders if you're looking for some reference but if you're not using scrapy I'd recommend just adding validation tests using data validation tools like cerberus which is super underrated. I cover popular data validation techniques on this short blog I wrote if you want to learn more.
-
Can you suggest something more to grow in scraping?
Other than that, have you looked into testing scrapers? Since scrapers are working with highly dynamic data writing good tests is quite a challange. For example, for parser monitoring using cerberus is a very cool tool which allows you to define loose requirements like "phone number should always be 9 numbers" etc.
detect-headless
-
Can you suggest something more to grow in scraping?
Do you know how to detect headless browser? For example, you can check your bot here: https://github.com/infosimples/detect-headless. Try to make your bot undetectable.
What are some alternatives?
jsonschema - An implementation of the JSON Schema specification for Python
pydantic - Data validation using Python type hints
schema - Schema validation just got Pythonic
voluptuous - CONTRIBUTIONS ONLY: Voluptuous, despite the name, is a Python data validation library.
Schematics - Python Data Structures for Humans™.
colander - A serialization/deserialization/validation library for strings, mappings and lists.
valideer - Lightweight data validation and adaptation Python library.
python-codicefiscale - :it: :credit_card: italian fiscal codes encoding, decoding and validation - codifica, decodifica e validazione del Codice Fiscale italiano.
marshmallow - A lightweight library for converting complex objects to and from simple Python datatypes.
config-file-validator - Cross Platform tool to validate configuration files
scrapy-test - framework to test scrapy spiders