-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible).
I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here.
I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
Whilst the application is not available to the public (yet), all of the libraries I've created are now available.
I should have both a Rust and a Golang version for most / all of them.
On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites.
[0]: https://github.com/denysvitali/sbb-api-rs
[1]: https://github.com/denysvitali/homegate-rs
[2]: https://github.com/denysvitali/flatfox-rs
[3]: https://github.com/denysvitali/go-sbb-api
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible).
I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here.
I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
Whilst the application is not available to the public (yet), all of the libraries I've created are now available.
I should have both a Rust and a Golang version for most / all of them.
On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites.
[0]: https://github.com/denysvitali/sbb-api-rs
[1]: https://github.com/denysvitali/homegate-rs
[2]: https://github.com/denysvitali/flatfox-rs
[3]: https://github.com/denysvitali/go-sbb-api
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible).
I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here.
I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
Whilst the application is not available to the public (yet), all of the libraries I've created are now available.
I should have both a Rust and a Golang version for most / all of them.
On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites.
[0]: https://github.com/denysvitali/sbb-api-rs
[1]: https://github.com/denysvitali/homegate-rs
[2]: https://github.com/denysvitali/flatfox-rs
[3]: https://github.com/denysvitali/go-sbb-api
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible).
I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here.
I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
Whilst the application is not available to the public (yet), all of the libraries I've created are now available.
I should have both a Rust and a Golang version for most / all of them.
On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites.
[0]: https://github.com/denysvitali/sbb-api-rs
[1]: https://github.com/denysvitali/homegate-rs
[2]: https://github.com/denysvitali/flatfox-rs
[3]: https://github.com/denysvitali/go-sbb-api
I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM.
Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site.
https://github.com/PuerkitoBio/goquery
Sorry, I had no idea that there is a paywall. I use the extension[1] that removes majority of paywalls.
[1] https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean
I've received a PR suggesting just that [0], my concern is
> Only thing that concerns me is that it will spawn n parallel requests to the website which could consume a lot of memory since we're talking about spawning a new tab with Puppeteer (Chrome :sigh:) for each link.
[0]: https://github.com/mattrighetti/athome-scraper/pull/1