article-extractor VS tarsier

Actually, I kept it simple - I use the original images from the news articles! When I fetch an article through RSS and extract its content using the @extractus/article-extractor library, it pulls the main image along with the content.

https://github.com/extractus/article-extractor

Agreed!

Apify's Website Content Crawler[0] does a decent job of this for most websites in my experience. It allows you to "extract" content via different built-in methods (e.g. Extractus [1]).

We currently use this at Magic Loops[2] and it works _most_ of the time.

The long-tail is difficult though, and it's not uncommon for users to back out to raw HTML, and then have our tool write some custom logic to parse the content they want from the scraped results (fun fact: before GPT-4 Turbo, the HTML page was often too large for the context window... and sometimes it still is!).

Would love a dedicated tool for this. I know the folks at Reworkd[3] are working on something similar, but not sure how much is public yet.

[0] https://apify.com/apify/website-content-crawler

[1] https://github.com/extractus/article-extractor

[2] https://magicloops.dev/

[3] https://reworkd.ai/

Edit: I found this library in NodeJs useful for article extraction. Anyone looking for something like you can take a look. https://github.com/extractus/article-extractor

Reworkd | Backend / Infrastructure | ONSITE San Francisco

At https://reworkd.ai/, we're building application layer LLM agents to extract web data at scale. We are foundational data infrastructure for startups today that are fine tuning models or building some web data constrained product. We're backed by YC, Paul Graham, AI grant, and many others.

We're looking for backend/infrastructure/full stack engineers to:

https://github.com/reworkd/tarsier/pull/115/files represents someone who does not know what git is used for

  Cloning into 'tarsier'...

Yes, thank you, I had the exact same experience. The actual project is probably https://reworkd.ai/

Reworkd (https://reworkd.ai/) | San Francisco (In-person) | Full-time

We're hiring a founding backend engineer to help us build infrastructure to run web agents at scale.

We're a super small scrappy team of four that's been working on the application layer of web agents since it's inception. Our projects have over 30k stars on GitHub, we're backed by PG himself + a bunch of great investors, and we have 3+ years of runway. Join us if you want to grind, and want a lot of ownership. (More info about the role in our job posting)

You can either apply through bookface (https://www.ycombinator.com/companies/reworkd/jobs/4f6BHpT-f...) or directly email me the following (asim@reworkd.ai) the following:

We run OCR on the screenshot & convert it to whitespace-structured text, that is passed to the LLM. The images below might make it clearer for you:

[1] https://github.com/reworkd/tarsier/blob/main/.github/assets/...

[2] https://github.com/reworkd/tarsier/blob/main/.github/assets/...

Agreed!

Apify's Website Content Crawler[0] does a decent job of this for most websites in my experience. It allows you to "extract" content via different built-in methods (e.g. Extractus [1]).

We currently use this at Magic Loops[2] and it works _most_ of the time.

The long-tail is difficult though, and it's not uncommon for users to back out to raw HTML, and then have our tool write some custom logic to parse the content they want from the scraped results (fun fact: before GPT-4 Turbo, the HTML page was often too large for the context window... and sometimes it still is!).

Would love a dedicated tool for this. I know the folks at Reworkd[3] are working on something similar, but not sure how much is public yet.

[0] https://apify.com/apify/website-content-crawler

[1] https://github.com/extractus/article-extractor

[2] https://magicloops.dev/

[3] https://reworkd.ai/