conan VS spack

Conan is a package manager for C/C++. See: https://conan.io/.

The way it works is that you can provide "recipes", which are Python scripts, that automate the process of collecting source code (usually from a remote Git repository, or a remote source tarball), patching it, making its dependencies and transitive dependencies available, building for specific platform and architecture (via any number of build systems), then packaging up and serving binaries. There's a lot of complexity involved.

Here are the two recipes I mentioned:

libcurl: https://github.com/conan-io/conan-center-index/blob/master/r...

OpenSSL v3: https://github.com/conan-io/conan-center-index/blob/master/r...

Now, for the sake of this thread I want to highlight three things here:

- Conan recipes are usually made by people unaffiliated with the libraries they're packaging;

- The recipes are fully Turing-complete, do a lot of work, have their own bugs - therefore they should really be treated as software comonents themselves, for the purpose of OSS clearing/supply chain verification, except as far as I know, nobody does it;

- The recipes can, and do, patch source code and build scripts. There's supporting infrastruture for this built into Conan, and of course one can also do it by brute-force search and replace. See e.g. ZLib recipe that does it both at the same time:

https://github.com/conan-io/conan-center-index/blob/7b0ac710... -- `_patch_sources` does both direct search-and-replace in source files, and applies the patches from https://github.com/conan-io/conan-center-index/tree/master/r....

Now, good luck keeping track of what's going on there.

There were various approaches recommended depending on our language and ecosystem. My classmates who developed using Node.js were recommended npm, and PyPI or poetry for Python. Since my program is written in C++, I was recommended to look into one of vcpkg or conan, but I ultimately did not use either package manager.

Hi u/instinkt900, Conan maintainer here. Thanks for your feedback! Please remember that we actively monitor and respond to our issue tracker on GitHub (https://github.com/conan-io/conan/issues/new/choose), we’d love to hear about your specific use cases or pain points, so that we can improve your experience and that of other users. The motivation behind most of the updates in Conan 2.0 was precisely feedback from the community, and to improve our ability to continue delivering features in the constantly changing C++ ecosystem. We can certainly do this at a quicker pace, with some exciting new features recently released and in the pipeline: package metadata, transparent backup of downloaded package sources, cache least-recently-used cleanup, etc. A lot of the big decisions that we took for Conan 2.0 were taken with consensus from expert users and contributors (https://conan.io/tribe) and https://github.com/conan-io/tribe. Some specific workflows may not have 1:1 replacements in Conan 2.0, and are likely to affect some of the “less travelled roads” of Conan 1.x, including some features that were always marked as experimental. We are happy to hear feedback so that we can best satisfy these use cases. Conan 2.0 also includes a more sophisticated API to cover cases where the built-in integrations may not satisfy users needs. For what it’s worth - we have also heard very positive feedback from users about how Conan 2.0 simplifies their workflows when compared to Conan 1.x. The C++ tooling ecosystem is fragmented and moves at different speeds, including our users. So it’s always a fine balancing act, but we don’t want to leave anyone behind! An example is Conan Center - over 90% (~1200) of all recipes have been migrated to support Conan 2.0, while still maintaining compatibility with Conan 1.x, precisely to avoid breaking users that are still on Conan 1.x.

Solution: don't use git submodules - use a package manager like Conan or vcpkg.

The closest thing we have at the moment is conan[1]. It’s a cross platform package manager that attempts to implement “toolchains”, whereby different build systems can be integrated[2]. This is a big problem with package management in C/C++, there’s no single, standardised build system that most projects use. There isn’t even a standardised compiler! So when hosting your own packages using Conan, often you need to make sure you build your application for three different compilers, for three different platforms. Sometimes (for modern MacOS) also for two different architectures each.

If you control the compiler AND build system you can get away with just one package for most cases. This true for Microsoft’s C/C++ package manager, NuGet[3]

Historically, the convention has been to use the package manager of the underlying system to install packages, as there are so many different build configurations to worry about when packaging the libraries. The other advantage of using the system package manager is that dependencies (shared libraries) that are common can be shared between many applications, saving space.

[1] https://conan.io/

Anyway, the problems are today starting to get fewer, as more an more adopt standard cross-platform portable build systems, a.k.a. CMake and package managers such as vcpkg or Conan. Together this will take care of building, installing, linking and using the entire dependency tree.

Simply use a package manager: Crypto++ is available on both vcpkg and Conan.

If you want a specific version or provide a more integrated workflow that is easier to use across platforms and among many developers, use a package manager like vcpkg or Conan.

I do however always recommend using a package manager: vcpkg or Conan to install and integrate third party libraries (together with CMake). This normally solves all the typical problems with dependencies.

There's a few, look into Conan or vcpkg (the latter is my personal recommendation).

> Are we talking about the same autotools?

Yes. Instead of figuring out how to do something particular with every single software package, I can do a --with-foo or --without-bar or --prefix=/opt/baz-1.2.3, and be fairly confident that it will work the way I want.

Certainly with package managers or (FreeBSD) Ports a lot is taken care of behind the scenes, but the above would also help the package/port maintainers as well. Lately I've been using Spack for special-needs compiles, but maintainer ease also helps there, but there are still cases one a 'fully manual' compile is still done.

> Suffice it to say, I prefer to work with handwritten makefiles.

Having everyone 'roll their own' system would probably be worse, because any "mysteriously failure" then has to be debugged specially for each project.

Have you tried Spack?

* https://spack.io

* https://spack.readthedocs.io/en/latest/

Well, good luck with that, cause it's broken.

Previous release miscompiled Python [1]

Current release miscompiles bison [2]

[1] https://github.com/spack/spack/issues/38724

[2] https://github.com/spack/spack/issues/37172#issuecomment-181...

gh is available via Homebrew, MacPorts, Conda, Spack, Webi, and as a…

> I can't count the number of times I've seen people say "md5 is fine for use case xyz" where in some counterintuitive way it wasn't fine.

I can count many more times that people told me that md5 was "broken" for file verification when, in fact, it never has been.

My main gripe with the article is that it portrays the entire legal profession as "backwards" and "deeply negligent" when they're not actually doing anything unsafe -- or even likely to be unsafe. And "tech" knows better. Much of tech, it would seem, has no idea about the use cases and why one might be safe or not. They just know something's "broken" -- so, clearly, we should update.

> Just use a safe one, even if you think you "don't need it".

Here's me switching 5,700 or so hashes from md5 to sha256 in 2019: https://github.com/spack/spack/pull/13185

Did I need it? No. Am I "compliant"? Yes.

Really, though, the main tangible benefit was that it saved me having to respond to questions and uninformed criticism from people unnecessarily worried about md5 checksums.

[1] https://github.com/spack/spack/blob/develop/var/spack/repos/...

In Spack [1] we can express all these constraints for the dependency solver, and we also try to always re-cythonize sources. The latter is because bundled cythonized files are sometimes forward incompatible with Python, so it's better to just regenerate those with an up to date cython.

[1] https://github.com/spack/spack/

You want to look at the tools used for HPC systems, these are generally very well tried and tested and can be setup for single machine usage. Remote access - we use ssh, but web interfaces such as Open On Demand exist - https://openondemand.org/. For managing Jobs, Slurm is currently the most popular option - https://slurm.schedmd.com/documentation.html. For a module system (to load software and libraries per user), Spack is a great - https://spack.io/. You might also want to consider containerisation options, https://apptainer.org/ is a good option.

git clone https://github.com/spack/spack.git ./spack/bin/spack install gcc