Scrapy VS colly

You could also visit the project's GitHub repository and add "/contribute" to the end of the URL. For example, visiting https://github.com/scrapy/scrapy/contribute will show you relatively approachable tasks for first-time contributors to the Scrapy project.

I used Scrapy for those, but you can also learn how to scrape and parse HTML with requests and something like Beautiful Soup.

I want to be able to run the Scrapy web crawling framework from within Django. Scrapy itself only provides a command line tool scrapy to execute its commands, i.e. the tool was not intentionally written to be called from an external program.

Yeah my hunch is that a combination of nltk, python-Levenshtein, numpy for language processing, pandas for gathering results and scrapy for web scraping should make it possible. Sadly such a project probably requires at least a month or two worth of training in Python to prototype. Good luck OP.

The scrapy framework would be a good fit for what you want to do (especially if you plan on hosting the crawler), however the learning curve is quite steep, especially if you don't have much experience using python.

I had a cool idea recently, coded up a scraper with scrapy. Decided it was a cool dataset, scraped it all into postgres with SQLalchemy. Built out a nice streamlit project so others users could interact with it as well. A core component of the project related to using spotify's API. Long story short, streamlit isn't totally mature and something like capturing the auth token after a redirect is a PITA. Plus, you are sort of locked into the visual design of streamlit to begin with.

Look into Scrapy

Don't know about any non scripting way. But i would use https://scrapy.org/ for this.

Web Crawler

Check out Scrapy - https://scrapy.org/

There is the scraper struct that spawns goroutine. Then, each goroutine scraps the web URL using go colly, and the structured MCQ questions are served from API endpoints. Users can choose between persistent Postgres model or in-memory data structure to store the questions and retrieve them from API. I have used dependency injection to inject the models via the interface to my application struct.

What follows is my thought process on how I collect all restaurant details from the Michelin Guide using Go with Colly. The final dataset is available free to be downloaded here.

The whole thing is written in Go on my end. Ingesting new headlines is handled in a goroutine that spawns within the process every 30 mins using a combo of the wonderful gofeed (https://github.com/mmcdole/gofeed) and colly (https://github.com/gocolly/colly) libraries.

When loading the front page, you're loading a 1-minute-cached HTML page of it that was constructed out of headlines already in my PostgreSQL database that were put there by the ingestion goroutine.

I like the idea of word clouds actually, I think you're on to something there. I think you just need to pre-generate them rather than doing it adhoc (if that's what you're doing here) for speed. Additionally, perhaps consider using sentiment in a way that orients stories based on positive and negative sentiment. Right now I am not seeing how I as a visitor/user can act on the sentiment analysis as it is presented now.

It would be neat to see a collection of uplifting stories grouped together through the sentiment analysis.

Anyway, food for thought. I hope you keep hacking away on it as it's just good fun to build things.

I've been looking at various scraping/crawling libraries like colly, but I'm not exactly sure if any of them can handle a specific use case I have.

In the process of trying to get better with Go, I have been trying to create a program that will scrape a subreddit and then save some info into a Postgres table. In this case, I decided to use r/frugalmalefashion and save the title and flair of recent posts into the table. I am using Colly to scrape data, and pgx as the database driver.

If you're familiar with Go, there's Colly too [1]. I liked its simplicity and approach and even wrote a little wrapper around it to run it via Docker and a config file:

https://gotripod.com/insights/super-simple-site-crawling-and...

[1] http://go-colly.org/

What's your problem with Colly? [0]

[0] http://go-colly.org/

If performance, Especially concurrency matters then we should include Go based libraries in the discussion as well.

Colly[1] is a all batteries included scrapping library, But could be bit intimidating for someone new to HTML scrapping.

rod [2] is a browser automation library based on DevTools Protocol which adheres to Go'ish way of doing things and so it's very intuitive but comes with overhead of running chromium browser even if its headless.

[1] https://github.com/gocolly/colly

[2] https://github.com/go-rod/rod

Before you write your own library for crawling, try some of the options out there. Many great Open Source libraries can achieve it: Scrapy, pyspider, node-crawler (Node.js), or Colly (Go). And many companies and services that provide you with scraping and crawling solutions.