Our great sponsors
-
croncert-config
configuration and github actions for concertcloud.live (fka croncert.ch), a website that shows you concerts in various cities
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I am currently building a webscraper, called goskyr, that can be run from the command line and is supposed to be easily configurable. So instead of having to write code to scrape a website you'd just write a configuration snippet and run the scraper. I realize that there are a number of gui based scraping services that make it extremely easy to setup a scraping process for any website, so for people having no coding experience whatsoever that would probably be the easiest solution. I'm trying to come close to those gui based solutions in terms of functionality by providing a 'smart' way of finding potentially interesting data/fields and letting the user select a subset in a terminal based ui. Also date extraction & parsing and the newly added machine learning capability is probably worth mentioning. Still, those other, gui based solutions are really awesome, eg octoparse or scrapestorm.
I actually started this scraping project because of an idea I wanted to try, which is scraping concert data from as many websites as possible with as little effort as possible, see https://github.com/jakopako/croncert-config This seems to work better and better. Still I am wondering if there are any other valid use cases for such a terminal based scraper or if it's rather niche. What do you think?
I came to similar idea when i was scrape flight tickets: https://github.com/PxyUp/fitter