hpy VS semgrep

There is a merge request up to add autogen rust bindings to hpy

https://github.com/hpyproject/hpy/pull/457

Are you referencing https://github.com/hpyproject/hpy?

I do hope it takes off.

The HPy project [0] seems like a promising way out of this.

[0] https://hpyproject.org/

socketify.py breaks the record for Python no other Python WebFramework/Server as able to reach 6.2 mi requests per second before in TechEmPower Benchmarks, this puts Python at the same level of performance that Golang, Rust and C++ for web development, in fact Golang got 5.2 mi req/s in this same round. Almost every server or web framework tries to use JIT to boost the performance, but only socketify.py deliveries this level of performance, and even without JIT socketify.py is twice as fast any other web framework/server in active development, and still can be much more optimized using HPy (https://hpyproject.org/). Python will get even faster and faster in future!

I think Propose a better messaging for Python is the option and a lot of languages will learn it from Rust, because rust erros are the best described errors I see in my life lol. Cargo is amazing and I think we will need a better poetry/pip for sure, HPy project will modernize extensions and packages 📦 too https://hpyproject.org/

It also show that PyPy3 will not magically boost your performance, you need to integrate in a manner that PyPy3 can optimize and delivery CPU performance, with a more complex example maybe it can help more. But why socketify is so much faster using PyPy3? The answer is CFFI, socketify did not use Cython for integration and cannot delivery the full performance on Python3, this will be solved with HPy.

HPy integration to better support CPython, PyPy and GraalPython

Absolutely everything in CPython is a PyObject, and that can’t be changed without breaking the C API. A PyObject contains (among other things) a type pointer, a reference count, and a data field; none of these things can be changed without (again) breaking the C API.

There have definitely been attempts to modernize; the HPy project (https://hpyproject.org/), for instance, moves towards a handle-oriented API that keeps implementation details private and thus enables certain optimizations.

NumPy is a sore point (works, but slow) and the missing spark to ignite PyPy adoption for a subset of users. The current hope seems to be HPy. If PyPy acquires good NumPy performance, a lot of people would migrate. Also of note is that conda-forge builds hundreds of packages for PyPy already (I think they started doing that in 2020).

I've battled with Python's packaging systems for more than a decade. Every couple of years some new thing comes along promising to fix package management, but it always fails spectacularly on some important, mainstream package or use case. I suspect a lot of this complexity comes down to the significant degree in which the Python ecosystem depends on C: not only do we need to package Python programs but C programs as well, and some people think we should ship C source code and build on the target machine and others think we should ship pre-built C binaries and dynamically link against them on the target machine thus Python supports both mechanisms; however, both are perilous in the general case.

And interestingly, Python depends so hard on C because CPython is very slow, and CPython is very slow because the C-extension API is virtually the whole of the CPython interpreter (many interesting optimizations to the interpreter would break compatibility in the ecosystem). So it certainly feels like all of Python's major problems come down to this decision to lean hard on C and specifically to make so much of the interpreter available for the C-extensions.

The way out as far as I can tell is to define some narrower API surface (e.g., https://hpyproject.org), get the ecosystem to consolidate around that, deprecate the old API surface, and then make the requisite breaking optimizations such that the ecosystem can feasibly do more in native Python. This requires leadership; however, and this has not historically seemed to be Python's strong suite--the Python maintainers seem unable to drive out big, necessary changes like this (which is certainly not to say that leadership is easy, particularly when Python is so established in many respects).

Personally, I've come to use Go for 99% of my Python use cases and it's been great. There are dramatically fewer C bindings, so the build/packaging tooling and performance are orders of magnitude better than in Python. Static typing works well out of the box, real static binaries are not only feasible but trivial (as opposed to Python where you try to build a zip file with your dependencies and the result is hundreds of megabytes and it's still missing the runtime, std libs, and various .so files). Further still, builds, tests, and every kind of tooling are far faster than with Python, and far simpler to install and manage. Unless you're doing data science, I don't think you'll regret the transition.

For the SAST stage, I used SonarQube tool. SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs and code smells on more than 30 programming languages. I preferred SonarQube instead of other SAST tools because it has a detailed documentation and plugins about integration with Jenkins and SonarQube works with Java projects pretty well. Of course you can similar multi-language-supported tools such as Semgrep or language-specific tools such as Bandit.

> Not sure I understand your point.

The problem is using Treesitter (for syntax highlighting and "semantic movements") and an LSP at the same time. So if your language has a LSP, using Treesitter additionally is redundant at best and introduces inconcistency at worst.

I'm not talking about using Treesitter as the parser for the LSP.

> Most popular languages have language-specific tools

I'd say even less popular langauges like Coq^H^H^HRocq, Lean 4, Koka, Idris, Unison, ... have their "own" tools, I do not know of a language that uses a Treesitter parser in its LSP, but I do know about tools like https://semgrep.dev/ (written in OCaml) and Github's code search which use Treesitter.

Well, when I seach for "semgrep", I get a very nice corporate landing page with a "Book Demo" button. Which is a level of hassle that just isn't worth it for smaller teams, because "Book Demo" usually means "We're going to try to do a dance to see how much money we can extract from you." Which smaller teams may only want to do for a handful of key tools.

(4 years ago, I was more willing to put up with enterprise licensing. But in the last two years, I've seen way too many enterprise vendors try to squeeze every penny they can get from existing clients. An enterprise sales process now often means "Expect 30% annual price hikes once you're in too deep to back out.")

There's also an open source "semgrep" project here: https://github.com/semgrep/semgrep. But this seems to be basically a vulernability scanner, going by the README.

Whereas AST-grep seems to focus heavily on things like:

1. One-off searching: "Search my tree for this pattern."

2. Refactoring: "Replace this pattern with this other pattern."

AST-grep also includes a vulnerability scanning mode like semgrep.

It's possible that semgrep also has nice support for (1) and (2), but it isn't clearly visible on their corporate landing page or the first open source README I found.

7. Semgrep

This project is a compilation of Semgrep rules derived from the OWASP Mobile Application Security Testing Guide (MASTG) specifically for Android applications. The aim is to enhance and support Mobile Application Penetration Testing (MAPT) activities conducted by the ethical hacker community. The primary objective of these rules is to address the static tests outlined in the OWASP MASTG.

For generally code analysis, I used Semgrep in the past.

For these cases, let me introduce you to my favorite static code analysis tool: semgrep. It's a free Open Source tool that you can install and use right now (it only starts costing money if you want to use their dashboard to view the results, which is entirely optional, and all code scanning runs on your device - code is never uploaded to any servers). As stated briefly, semgrep searches for code matching specific patterns, taking the semantics of the code into account (hence, semantic grep). You can use it for security checks based on a large set of detection rules curated by the semgrep community, but where it really shines is when you start writing rules for your own use cases.

Good idea! I opened an issue here: https://github.com/returntocorp/semgrep/issues/6331