warehouse VS infer

python3 -m pip install \ --trusted-host test.pypi.org --trusted-host test-files.pythonhosted.org \ --index-url https://test.pypi.org/simple/ \ --extra-index-url https://pypi.org/simple/ \ piper_whistle==$(python3 -m src.piper_whistle.version)

In my experience so far, I can use a vast amount of the Python Standard Library to build Wasm-powered serverless applications. The caveat I currently understand is that Python’s implementation of TCP and UDP sockets, as well as Python libraries that use threads, processes, and signal handling behind the scenes, will not compile to Wasm. It is worth noting that a similar caveat exists with libraries that I find on The Python Package Index (PyPI) site. While these caveats might limit what can be compiled to Wasm, there are still a ton of extremely powerful libraries to leverage.

We believe that poetry is currently the best tool for this purpose, besides of being the most popular one at the moment. This is why we will use poetry to manage the dependencies of our project throughout this series of posts. Poetry allows you to declare the libraries your project depends on, and it will manage (install/update) them for you. Poetry also allows you to package your project into a distributable format and publish it to a repository, such as PyPI. We strongly recommend you to learn more about this tool by reading the official documentation.

From there, I needed to learn a bit about PyPi or Python Package Index, which is the home for all the wonderful packages that you know if you have ever run the handy pip install command. PyPi has a pretty quick and easy onboarding, which requires a secured account be created and, for the purposes of submitting packages from CLI, an API token be generated. This can be done in your PyPi profile. Once logg just navigate to https://pypi.org/manage/account/ and scroll down to the API tokens section. Click “Add Token” and follow the few steps to generate an API token which is your access point to uploading packages. With all this in place, I was able to use twine to handle the package upload. First I needed to install twine, again as simple as pip install twine. In order for twine to access my API token during the package upload process, it needed to read it from .pypirc file that contains the token info. For some that file may exist already, for me I was required to create it. Working in windows I simply used a text editor to create it in my home user directory ($HOME/.pypirc). The file contents had a TOML like format looked like this:

I have published the package to Python Package Index, commonly called PyPi, and in this post, I'll be sharing the steps I had to follow in the process.

Register at PyPI.org

According to the stats on the original link, there are over 25,000 identified secret ids/keys/tokens in the data. And it looks like that's just identifiable secrets, e.g. "Google API Keys" that I'm guessing are identifiable because they have a specific pattern, and may be missing other secrets that use less recognizable patterns.

I mean, sure, compared to the 478,876 Projects claimed on https://pypi.org/, that's a pretty small minority. On the other hand, I'd guess a many Python packages don't use these particular services, or even need to connect to a remote service at all, so the area for this class of mistake should be even smaller.

And mistakes do happen, but that's a pretty big thing to miss if you are knowingly publishing your code with the expectation other people will be reading it.

PyPi package

Does pypi.org display the Union of all keywords, the keywords of the most recent release, the keywords of the first release or some other weird combination like the intersection?

https://peps.python.org/pep-0458/

Here's the in-progress roadmap: https://github.com/pypi/warehouse/issues/10672

If there's particular issues you believe you could pick off to help achieve the goal, much appreciated!

I think most development occurs on problems that can't be formally modeled anyway. Most developers work on things like, "can you add this feature to the e-commerce site? And can the pop-up be blue?" which isn't really model-able.

But that's not to say that formal methods are useless! We can still prove some interesting aspects of programs -- for example, that every lock that gets acquired later gets released. I think tools like Infer[0] could become common in the coming years.

[0]: https://fbinfer.com/

Using infer, someone else exploited null-dereference checks to introduce simple affine types in C++. Cppcheck also checks for null-dereferences. Unfortunately, that approach means that borrow-counting references have a larger sizeof than non-borrow counting references, so optimizing the count away potentially changes the semantics of a program which introduces a whole new way of writing subtly wrong code.

Meta/Facebook are long time OCaml users, their logo is on the OCaml website. Their static analysis tool and its predecessor are both written in OCaml.

Then this idea that the US government will tell tech companies how to write secure software. Let's get this straight, the private sector, especially big tech is miles ahead of US government in this regard. Microsoft literally invented threat modelling and modern exploit mitigations. Facebook has the best appsec processes pretty much in the whole world, including their own cutting edge code analyzer. AWS uses formal verification everywhere. Meanwhile the US government itself runs mission-critical systems that's almost literally held together by bubble gum and toothpicks. Maybe they could dial down the arrogance a tad, get their own shit together, learn how this cyber stuff is actually done and only then try lecturing everyone else.

Efforts: Dependabot, CodeQL, Coverity, facebook's Infer tool, etc

I notice there isn't fbinfer. It's pretty cool, and is used for this library.

"Move fast, break stuff" is a great approach when you aren't pushing the broken bits to production. Fuck, even Facebook, the big "move fast, break stuff" company, uses tools to detect errors in its continuous integration toolchain. https://fbinfer.com/

TBH, there's a non-zero amount of non-"ivory tower" tools you may have used that are written in functional languages. Say, Pandoc or Shellcheck are written in Haskell; Infer and Flow are written in OCaml. RabbitMQ and Whatsapp are implemented in Erlang (FB Messenger was too, originally; they switched to the C++ servers later). Twitter backend is (or was, at least) written in Scala.