memory_profiler
selectolax
memory_profiler | selectolax | |
---|---|---|
6 | 6 | |
4,214 | 970 | |
0.6% | - | |
3.7 | 7.7 | |
19 days ago | about 2 months ago | |
Python | Cython | |
BSD 3-clause "New" or "Revised" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
memory_profiler
- Ask HN: C/C++ developer wanting to learn efficient Python
-
8 Most Popular Python HTML Web Scraping Packages with Benchmarks
memory_profiler
-
Check Python Memory Usage
pythonprofilers/memory_profiler: Monitor Memory usage of Python code
-
Profiling and Analyzing Performance of Python Programs
# https://github.com/pythonprofilers/memory_profiler pip install memory_profiler psutil # psutil is needed for better memory_profiler performance python -m memory_profiler some-code.py Filename: some-code.py Line # Mem usage Increment Occurrences Line Contents ============================================================ 15 39.113 MiB 39.113 MiB 1 @profile 16 def memory_intensive(): 17 46.539 MiB 7.426 MiB 1 small_list = [None] * 1000000 18 122.852 MiB 76.312 MiB 1 big_list = [None] * 10000000 19 46.766 MiB -76.086 MiB 1 del big_list 20 46.766 MiB 0.000 MiB 1 return small_list
-
Profiling Python code with memory_profiler
What do you do when your Python program is using too much memory? How do you find the spots in your code with memory allocation, especially in large chunks? It turns out that there is not usually an easy answer to these question, but a number of tools exist that can help you figure out where your code is allocating memory. In this article, I’m going to focus on one of them, memory_profiler.
- What Is Your Favorite Profilerperformance Tool
selectolax
-
GitHub – GSA/code-gov: An informative repo for all Code.gov repos
https://github.com/rushter/selectolax#simple-benchmark )
(Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct )
datasette-graphql adds a GraphQL HTTP API to a SQLite database:
-
8 Most Popular Python HTML Web Scraping Packages with Benchmarks
selectolax
- High performance code in Python
-
Web Scraping with Python: Everything you need to know to get started (2022)
try this... https://github.com/rushter/selectolax
-
The State of Web Scraping in 2021
Lazyweb link: https://github.com/rushter/selectolax
although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_
> Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies.
although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor
---
> It looks like the author of the article just googled some libraries for each language and didn't research the topic
Heh, oh, new to the Internet, are you? :-D
- Show HN: Fast HTML5 parser for Python with multiple backends
What are some alternatives?
py-spy - Sampling profiler for Python programs
lxml - The lxml XML toolkit for Python
line_profiler
lexbor - Lexbor is development of an open source HTML Renderer library. https://lexbor.com
profiling
html5lib - Standards-compliant library for parsing and serializing HTML documents and fragments in Python
pyflame
pyppeteer - Headless chrome/chromium automation library (unofficial port of puppeteer)
Laboratory - Achieving confident refactoring through experimentation with Python 2.7 & 3.3+
pyquery - A jquery-like library for python
filprofiler - A Python memory profiler for data processing and scientific computing applications
gazpacho - 🥫 The simple, fast, and modern web scraping library