-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
> I don't think there's any clear argument you can make on first principles for which of those is actually the case.
I don't understand why someone would try to argue from first principles here, it just seems like such a bizarre approach.
Anyway, it's not just a security issue. Malicious packages and security fixes are only part of the picture. Other issues:
- Despite a team's promise to use semantic versioning, point releases & "bugfix" releases will break downstream users
- Other systems for determining the versions to use are much more unpredictable and hard to understand than estimated (look at Dart and Cargo)
https://github.com/dart-lang/pub/blob/master/doc/solver.md
https://github.com/rust-lang/cargo/blob/1ef1e0a12723ce9548d7...
> I don't think there's any clear argument you can make on first principles for which of those is actually the case.
I don't understand why someone would try to argue from first principles here, it just seems like such a bizarre approach.
Anyway, it's not just a security issue. Malicious packages and security fixes are only part of the picture. Other issues:
- Despite a team's promise to use semantic versioning, point releases & "bugfix" releases will break downstream users
- Other systems for determining the versions to use are much more unpredictable and hard to understand than estimated (look at Dart and Cargo)
https://github.com/dart-lang/pub/blob/master/doc/solver.md
https://github.com/rust-lang/cargo/blob/1ef1e0a12723ce9548d7...
> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u ` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment.
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
[0]: https://news.ycombinator.com/item?id=30870862
[1]: https://github.com/caddyserver/caddy/blob/master/go.mod
>> The only commands that will change the go.mod (and therefore the build) are go get and go mod tidy. These commands are not expected to be run automatically or in CI, so changes to dependency trees must be made deliberately and have the opportunity to go through code review.
GO doesn't do jack shit to mitigate supply chain attacks. Version pinning with checksum and that is it. But what could they do? Solve supply chain attacks as a language feature? That doesn't even make sense.
Application developers using Go must prevent supply chain attacks against their applications. So go get some SAST for your pipeline.
Sure there is truth in saying: always verify your dependencies (and their dependencies) yourself with a code review on every update. But lets talk about collaborative vulnerability management instead. (yes there could be other attestations, but one thing at a time).
Let's say repositories that publishes go modules should also publish a curated list of known vulnerabilities (including known supply chain attacks) for the modules they publish. This curation is work: reports must be verified before being included in the list and they must be verified quickly. This work scales with the number of packages published. And worse, modules could be published in more than one repository, module publishing repository can be different from source code repositories, and lists of vulnerabilities can exist independent from these repository - so reports should be synced between different list providers. Different implementations and lack of common standards make this a hard problem. And implicit trust for bulk imports could open the door for takedown attacks.
There is an argument that vulnerability listing should be split from source and module publishing: each focusing on their core responsibility. For supply chain attacks especially this split in responsibilities also makes it harder for an attacker to both attack suppliers and suppress reports. But for all other issues it increase distance as reports must travel upstream. And it creates perverse incentives, like trying to keep reports exclusive to paying customers.
To pile on the insanity: reports can be wrong. And there are unfixed CVEs that are many years old (well ok maybe not for go... yet). Downstream there are "mitigated" and "wont-fix" classifications for reports about dependencies and many SAST tooling can't parse that for transitive dependencies.
Really, supply chain attacks are the easy case in vulnerability management, because they are so obviously a "must-fix" when detected. (and to please the never update crowd: for a downstream project "fix" can mean not updating a dependency into an attacked version)
Long story short: go get some SAST in your pipelines to defend against supply chain attacks. Like GitLabs Gemnasium ( https://gitlab.com/gitlab-org/security-products/gemnasium-db... ) or GitHubs Dependabot ( https://github.com/advisories?query=type%3Areviewed+ecosyste... ) among many, many others. (not recommendations, just examples!)
This helps you sort out supply chain attacks that other people have already found, before you update into them. (Collaboration!) is useful. Sure you are still left with reading the source changes of any dependency update, because who knows, you may be the first one to spot one, but hey, good for you.