13 ways to scrape any public data from any website

1. parsel

   1 6 1,332 6.5 Python

   Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

   

   variable.css(".X5PpBb::text").get() # returns a text value variable.css(".gs_a").xpath("normalize-space()").get() # https://github.com/scrapy/parsel/issues/192#issuecomment-1042301716 variable.css(".gSGphe img::attr(srcset)").get() # returns a attribute value variable.css(".I9Jtec::text").getall() # returns a list of strings values variable.xpath('th/text()').get() # returns text value using xpath

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. Scrapy

   2 197 62,120 9.5 Python

   Scrapy, a fast high-level web crawling & scraping framework for Python.

   

   scrapy is a high-level webscraping framework designed to scrape data at scale and can be used to create a whole ETL pipeline.

4. playwright-python

   3 37 14,718 8.2 Python

   Python version of the Playwright testing and automation library.

   

   playwright is a modern alternative to selenium. It can perform pretty much all interations as user would do i.e clicks, scrolls and many more.

5. Pandas

   4 449 48,900 10.0 Python

   Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

   

6. lxml

   5 20 3,035 9.3 Python

   The lxml XML toolkit for Python

   

   Parsel is a library build to extract data from XML/HTML documents with XPath and CSS selectors support, and could be combined with regular expressions. It's usees lxml parser under the hood by default.

7. insomnia

   6 258 38,472 9.7 TypeScript

   The open-source, cross-platform API client for GraphQL, REST, WebSockets, SSE and gRPC. With Cloud, Local and Git storage.

   

   Copy URL as CURL (Bash) and use it with online CURL runner or tools such as Insomnia.

Suggest a related project