Pandas VS Scrapy

pip install https://github.com/pandas-dev/pandas/archive/master.zip

Going through the venv site-packages I found 2 things: * Once your py files are converted into pyc or pyo you can delete the originals at the cost of debugability. It won't be a huge change but it's something. * Pandas specifically carries a huge tests folder it's discussed here. You can delete it. This is probably true for other libraries as well to some extent (carrying non production code into the package.

I'm Patrick Hoefler aka phofl and I'm one of the core team members developing and maintaining pandas (repo, docs), a popular data analysis library.

introducing optional behaviour comes with a huge maintenance cost (I started making such a proposal here, but then withdrew it)

I think this is an interesting question! I've opened https://github.com/pandas-dev/pandas/issues/51751

Personally polars' strictness is making me think about situations when in pandas we end up with object dtype, which we should probably avoid. Here's an example: https://github.com/pandas-dev/pandas/issues/50887 (polars would just error in such a case, which I think is the correct thing to do)

If you want to do data analysis or engineering, spend more time on Pandas or PySpark.

This Thursday we'll be hosting an AMA with some of the developers of pandas. The AMA will 'officially' start at 5:30pm UTC.

We released the release candidate for 2.0 last week, so the actual release is expected shortly, possibly next week. Please help us in testing that everything works through testing the rc :)

I chose to do this by using the popular Pandas library for tabular data analysis tasks.

You can use automation tools like Selenium or Playwright. You can work with a full-fledged framework such as Scrapy. I also recently discovered a Python tool like selectolax Lexbor, which allows you to extract data very quickly.

You may want to check out [estela](https://estela.bitmaker.la/docs/), which is a spider management solution, developed by [Bitmaker](https://bitmaker.la) that allows you to run [Scrapy](https://scrapy.org) spiders.

I wanted to invest my time and energy in learning the fastest, most efficient one, that can scale with my as my projects get more and more complex scrapy. After all, I want my projects to shine so bright in my cv it blinds the recruiter's eyes.

Try Scrapy (Python) https://scrapy.org/

Scrapy

If you weren't already aware, Scrapy has strong support for this via their HTTPCache middleware; you can choose whether to have it actually behave like a cache, choosing to returned already scraped content if matched or merely to act as a pass-through cache: https://docs.scrapy.org/en/2.7/topics/downloader-middleware....

Their OOtB storage does what the sibling comment says about sha1-ing the request and then sharding the output filename by the first two characters: https://github.com/scrapy/scrapy/blob/2.7.1/scrapy/extension...

Surprised at the name of this project given there is an incredibly popular project called scrapy related to web scraping. This project would really benefit from a rebrand.

I was looking for a used cars. I written a scraper using Scrapy, that was gathering all new offers, filtered by my criteria, every hour. Then it was sending me nicely formatted email.