scaling-to-distributed-crawling
Angular
Our great sponsors
scaling-to-distributed-crawling | Angular | |
---|---|---|
5 | 699 | |
36 | 94,541 | |
- | 0.9% | |
0.0 | 10.0 | |
over 2 years ago | 1 day ago | |
HTML | TypeScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
scaling-to-distributed-crawling
-
DOs and DON'Ts of Web Scraping
We published a repository and blog post about distributed crawling in Python. It is a bit more complicated than what we've seen so far. It uses external software (Celery for asynchronous task queue and Redis as the database).
- Mastering Web Scraping in Python: Scaling to Distributed Crawling - ZenRows
- Mastering Web Scraping in Python: Scaling to Distributed Crawling – ZenRows
-
Mastering Web Scraping in Python: Scaling to Distributed Crawling
We will start to separate concepts before the project grows. We already have two files: tasks.py and main.py. We will create another two to host crawler-related functions (crawler.py) and database access (repo.py). Please look at the snippet below for the repo file, it is not complete, but you get the idea. There is a GitHub repository with the final content in case you want to check it.
Angular
-
Angular Signals, Reactive Context, and Dynamic Dependency Tracking
/** * https://github.com/angular/angular/blob/75a186e321cb417685b2f13e9961906fc0aed36c/packages/core/src/render3/reactivity/untracked.ts#L15 * * packages/core/src/render3/reactivity/untracked.ts * **/ export function untracked(nonReactiveReadsFn: () => T): T { const prevConsumer = setActiveConsumer(null); try { return nonReactiveReadsFn(); } finally { setActiveConsumer(prevConsumer); } }
- Episode 24/15: Wiz behind the curtain, Copilot in VSCode
-
Always unsubscribe. No exceptions. Debate closed.
source: https://github.com/angular/angular/issues/46542
-
Angular Signals: Best Practices
Besides the dangers, mentioned by Angular docs (infinite loops, change detection errors), there is another thing, that might be quite nasty: effects are executed in a reactive context, and any code you call in effect, will be executed in a reactive context. If that code reads some signals, they will be added as dependencies to your effect. Here Alex Rickabaugh explains the details.
-
Understanding control flow syntax in Angular 17
In June 2023, the Angular team raised a new RFC to implement control flow syntaxes within Angular. They gave the following rationale for introducing control flow syntax:
- Episode 24/09: Testing without TestBed, SSR & Hydration
-
Preparing our Code for Zoneless Angular
For scheduling, I use awesome code I found in the Angular source code.
-
⏰ It’s time to talk about Import Map, Micro Frontend, and Nx Monorepo
Just to give you more context, I led the migration of several AngularJS applications to the newer Angular Framework. My client finally decided to make that move following the AngularJS deprecation announcement (stay up to date please 🙏)️.
-
Conventional commit specification
Link — angular/CONTRIBUTING.md
-
Angular Control Flow: the complete guide
Angular v17 was released some months ago with a ton of new features, a brand new logo and the new blog angular.dev.
What are some alternatives?
celery - Distributed Task Queue (development branch)
Next.js - The React Framework
colly - Elegant Scraper and Crawler Framework for Golang
qwik - Instant-loading web apps, without effort
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
SvelteKit - web development, streamlined
Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
Alpine.js - A rugged, minimal framework for composing JavaScript behavior in your markup.
newspaper - newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
solid - A declarative, efficient, and flexible JavaScript library for building user interfaces.
PeARS-orchard - This is the development version of PeARS, the people's search engine. More compact but less robust than PeARS-lite. If you just want to use PeARS as a local indexer, use PeARS-lite instead.
lit - Lit is a simple library for building fast, lightweight web components.