|5 days ago||9 months ago|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
A simple solution to rotate proxies or how to spin up your own rotation proxy server with Puppeteer and only a few lines of JS code
1 project | reddit.com/r/webscraping | 5 Mar 2021
I'm currently implementing concurrency conditions at project/proxy/domain/session level in https://github.com/get-set-fetch/scraper . On each level you can define the maximum number of requests and the delay between two consecutive requests.
Web scraping content into postgresql? Scheduling web scrapers into a pipeline with airflow?
1 project | reddit.com/r/webscraping | 15 Feb 2021
If you're familiar with nodejs give https://github.com/get-set-fetch/scraper a try. Scraped content can be stored in sqlite, mysql or postgresql. It also supports puppeteer, playwright, cheerio or jsdom for the actual content extraction. No scheduler though.
Web Scraping 101 with Python
5 projects | news.ycombinator.com | 10 Feb 2021
I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content.
What is your “I don't care if this succeeds” project?
42 projects | news.ycombinator.com | 1 Feb 2021
https://github.com/get-set-fetch/scraper - I've been working (intermittently :) ) on a nodejs or browser extension scraper for the last 3 years, see the other projects under the get-set-fetch umbrella. Putting a lot more effort lately as I really want to do those Alexa top 1 million analysis like top js libraries, certificate authorities and so on. A few weeks back I've posted on Show:HN as you can do basic/intermediate? scraping with it.
Not capable of handling 1 mil+ pages as it still limited to puppeteer or playwright. Working on adding cheerio/jsdom support right now.
Show HN: Plugin Based, Batteries Included, Web Scraper
1 project | news.ycombinator.com | 19 Jan 2021
How possible is it for a beginner to establish pipelines, data warehouse, and visualization solution as a team of 1?
1 project | reddit.com/r/dataengineering | 20 Feb 2021
This book will walk you through setting up a complete data engineering stack on GCP: https://github.com/Nunie123/data_engineering_on_gcp_book
Python & SQL knowledge needed for ETL?
1 project | reddit.com/r/dataengineering | 16 Feb 2021
As for resources, this book goes over a lot of these: https://github.com/Nunie123/data_engineering_on_gcp_book. However, this goes over the 'how', not the 'why'. The only method I know for understanding the 'why' is experience. Whether at work or personal projects.
Learning Python and SQL: What should be my next step?
1 project | reddit.com/r/dataengineering | 16 Feb 2021
Here's a good book to follow along to introduce you to common tooling and design patterns: https://github.com/Nunie123/data_engineering_on_gcp_book
Github Repo with All Data tranformation,Cleaning,Validation
I'm not sure if this is exactly what you're looking for, but here's a book on GitHub that talks about the tools and steps for building data pipelines into a data warehouse: https://github.com/Nunie123/data_engineering_on_gcp_book
What is the low hanging fruit for a brand new GCP data engineer to learn?
Check out this book: https://github.com/Nunie123/data_engineering_on_gcp_book
Unsure about overall process of data engineering
If you're interested in example of how to build a complete data engineering infrastructure, you should check out this book: https://github.com/Nunie123/data_engineering_on_gcp_book
[HELP] Airflow Reverse proxy + load balancer +docker
1 project | reddit.com/r/dataengineering | 10 Feb 2021
If you want to try Airflow without the setup headache, you can try Composer on GCP, which is a hosted version of Airflow. I wrote some info on how to do that here: https://github.com/Nunie123/data_engineering_on_gcp_book/blob/master/ch_2_orchestration.md
Transition from a Quality engineer to Data engineer
1 project | reddit.com/r/dataengineering | 9 Feb 2021
This book might be a good resource for you: https://github.com/Nunie123/data_engineering_on_gcp_book
Accepted a data engineer intern role at a Big N company - how do I learn as much as possible?
1 project | reddit.com/r/dataengineering | 6 Feb 2021
If you want a place to start on personal projects you can check out this book, https://github.com/Nunie123/data_engineering_on_gcp_book, which will walk you through the basics of setting up a full data engineering stack.
What tools, software, programming languages, and etc. does a data engineer need to have in 2021
1 project | reddit.com/r/dataengineering | 4 Feb 2021
If you are interested in tooling, here's a free book on setting up a basic data engineering tech stack on GCP: https://github.com/Nunie123/data_engineering_on_gcp_book
What are some alternatives?
playwright-python - Python version of the Playwright testing and automation library.
pyppeteer - Headless chrome/chromium automation library (unofficial port of puppeteer)
vopono - Run applications through VPN tunnels with temporary network namespaces
listudy - Listudy - chess training server
finsim - A personal finance simulator library in Rust.
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
exomind - A personal knowledge management tool hosted on your own personal cloud
commutative-algebra - An introduction to the basic ideas of commutative algebra
ppp_thing - A poorly written, minimum viable PPPoE client with session handoff between redundant FreeBSD routers
check-all-the-things - check all of the things!
Arthur - How to build your own AI art installation from scratch [Moved to: https://github.com/maxvfischer/DIY-ai-art]
Yue - A library for creating native cross-platform GUI apps