py-spy VS ripgrep

For CPU cycles, py-spy[0] is getting more and more used. For RAM, I would like to known too...

[0] -- https://github.com/benfred/py-spy

Theres also Py Spy, a profiling tool that can generate flame charts containing a mix of python and C (or C++) calls.

https://github.com/benfred/py-spy

It's worked really well for my needs

You should profile your consumer to check the bottlenecks. You can use the excellent py-spy(written in Rust). IMO a few usage of Numba there and there should solve your performance issues.

So as a first step you'll want to profile your program to figure out where it's slow, and hopefully that'll also tell you why it's slow. I'm the (biased) author of the Sciagraph profiler which is designed for this sort of application (https://sciagraph.com) but you can also try py-spy, which isn't as well designed for data processing/analysis applications (e.g. it won't visualize parallelism at all) but can still be informative (https://github.com/benfred/py-spy). Both are written in Rust ;)

Any advantages/disadvantages compared to py-spy [1]?

[1]: https://github.com/benfred/py-spy

Python profiling is enabled primarily through cprofile, and can be visualized with help of tools like snakeviz (output flame graph can look like this). There are also memory profilers like memray which does in-depth traces, or sampling profilers like py-spy.

A good profiler would be https://github.com/benfred/py-spy . If you run your app/benchmark with it, it should be able to draw a flamegraph telling you where the majority of time is spent. The info here is quite fine grained so it would already tell you where the bottleneck is. Without a full-fledged profiler you can also measure the timings in various parts of the code to understand where the bottleneck is.

Might be worth raising an issue on py-spy (a python profiler written in rust which "supports profiling native python extensions written in languages like C/C++ or Cython" to see if that can close the loop.

ripgrep - https://github.com/BurntSushi/ripgrep

Basic code searching skills seems like something new developers are never explicitly taught, but which is an absolutely crucial skill to build early on.

I guess the knowledge progression I would recommend would look something kind this:

- Learning about Ctrl+F, which works basically everywhere.

- Transitioning to ripgrep https://github.com/BurntSushi/ripgrep - I wouldn't even call this optional, it's truly an incredible and very discoverable tool. Requires keeping a terminal open, but that's a good thing for a newbie!

- Optional, but highly recommended: Learning one of the powerhouse command line editors. Teenage me recommended Emacs; current me recommends vanilla vim, purely because some flavor of it is installed almost everywhere. This is so that you can grep around and edit in the same window.

- In the same vein, moving back from ripgrep and learning about good old fashioned grep, with a few flags rg uses by default: `grep -r` for recursive search, `grep -ri` for case insensitive recursive search, and `grep -ril` for case insensitive recursive "just show me which files this string is found in" search. Some others too, season to taste.

- Finally hitting the wall with what ripgrep can do for you and switching to an actual indexed, dedicated code search tool.

live grep: ripgrep

The world has moved on though to opinionated tools, and Rust isn't even the furthest in that direction (That would be Go). The equivalent of those two lines in Cargo.toml would be this example of a basic configuration from the jacoco-maven-plugin: https://www.jacoco.org/jacoco/trunk/doc/examples/build/pom.x... - That's 40 lines in the section to do the "defaults".

Yes, you could add a load of config for files to include/exclude from coverage and so on, but the idea that that's a norm is way more common in Java projects than other languages. Like here's some example Cargo.toml files from complicated Rust projects:

Servo: https://github.com/servo/servo/blob/main/Cargo.toml

rust-gdext: https://github.com/godot-rust/gdext/blob/master/godot-core/C...

ripgrep: https://github.com/BurntSushi/ripgrep/blob/master/Cargo.toml

socketio: https://github.com/1c3t3a/rust-socketio/blob/main/socketio/C...

I'm not clear on why you're seeing the results you are. It could be because your haystack is so small that you're mostly just measuring noise. ripgrep 14 did introduce some optimizations in workloads like this by reducing match overhead, but I don't think it's anything huge in this case. (And I just tried ripgrep 13 on the same commands above and the timings are similar if a tiny bit slower.)

[1]: https://github.com/radare/ired

[2]: https://github.com/BurntSushi/ripgrep/discussions/2597

Explore o Ripgrep no repositório oficial: https://github.com/BurntSushi/ripgrep

🔎🗃️ ripgrep or ugrep (search fast, use regex patterns or fuzzy search, pipe output to bash/zsh shell for further processing V coloring)