jazelle VS core-js

But we also have teams working full time on monorepo stuff. I wrote tooling myself to bridge some of the gaps in the web ecosystem (e.g. we integrate w/ bazel to compute build graphs), but one still needs to integrate that w/ CI. Our solution uses dynamic pipeline generation, spot instances for elastic compute power, a bunch of crazy optimizations, a test flakiness detection system, a merge queue system (because we have hundreds of commits landing daily and HEAD must be green at all times)... A lot of this has no turn-key open source / commercial offerings and you're gonna need to invest time into integrations w/ the ones that do. Monorepos can be quite a commitment.

Yes, it's possible. At Uber we run a 1000+ package monorepo w/ Bazel and Yarn. Someone else mentioned rules_nodejs if you want to go with the popular option that is more or less in line with the Bazel philosophy[0]. We use a project called jazelle[1] that makes some different trade-offs that lets us do some interesting things like not busting the whole world's cache when lockfile changes, and source code generation (including automatic syncing of BUILD files to package.json)

[0] https://news.ycombinator.com/item?id=30959893

[1] https://github.com/uber-web/jazelle

I maintain the web monorepo at Uber (a fairly large codebase w/ ~1000 packages). Our monorepo uses a tool I wrote called jazelle that wraps over yarn workspaces and bazel.

3) Yes, you can have granular smart tests. Again, tooling can help. We use Bazel (I wrote a JS-oriented wrapper on top of it called jazelle), which can compute dependency graphs and only test things that changed. For example, with my tool, you can have a master job call jazelle changes to figure out what projects need to be checked given a git diff, and then we distribute the workload across multiple machines to parallelize tests/lint/bundle size analysis/etc (we can do a full run of several thousand compute hours for ~1000 packages in about 20 wall clock minutes). Other similar tools exist for dep graph computation, e.g. turborepo, and to some extent Rush and yarn workspaces + foreach, which might be sufficient for your scale.

Nice write-up. I have explored different repo strategies quite a bit myself in the course of a few efforts that I've been involved with. On one, we originally had a monolythic framework and everything the article said about cons is pretty spot on. However, I'll qualify by saying that I think the problems come less because of the nature of monolyths in general and more because of lack of experience with modular design.

We then wrote a new framework using a monorepo approach, with separate packages using Lerna. The problem here was tooling. Dependent builds were not supported and I've had to delete node_modules more times than I'd ever cared to count. The article talks about some github specific problems (namely, the issues list being a hodge-podge of every disparate package). We tried zenhub, it works ok, but it's a hack and it kinda shows. I've seen other projects organize things via tags. Ultimately it comes down to what the team is willing to put up with.

We eventually broke the monorepo out into multi-repos, and while that solved the problem of managing issues, now the problem was that publishing packages + cross-package dependencies meant that development was slower (especially with code reviews, blocking CI tests, etc).

Back to a monorepo using Rush.js (and later Bazel). Rush had similar limitations as Lerna (in particular, no support for dependent tests) and we ditched it soon afterwards. Bazel has a lot of features, but it takes some investment to get the most out of it. I wrote a tool to wrap over it[0] and setup things to meet our requirements.

We tried the "multi-monorepo" approach at one point (really, this is just git submodules), and didn't get very good results. The commands that you need to run are draconian and having to remember to sync things manually all the time is prone to errors. What's worse is that since you're dealing with physically separate repos, you're back to not having good ways to do atomic integration tests across package boundaries. To be fair, I've seen projects use the submodules approach[1] and it could work depending on how stable your APIs are, but for corporate requirements, where things are always in flux, it didn't work out well.

Which brings me to another effort I was involved with more recently: moving all our multi-repo services into a monorepo. The main rationale here is somewhat related to another reason submodules don't really fly: there's a ton of packages being, a lot of stakeholders with various degrees of commit frequency, and reconciling security updates with version drift is a b*tch.

For this effort we also invested into using Bazel. One of the strengths of this tool is how you can specify dependent tasks, for example "if I touch X file, only run the tests that are relevant". This is a big deal, because at 600+ packages, a full CI run consumes dozens of hours worth of compute time. The problem with monorepos comes largely from the sheer scale: bumping something to the next major version requires codemods, and there's always someone doing some crazy thing you never anticipated.

With that said, monorepos are not a panacea. A project from a sibling team is a components library and it uses a single repo approach. This means a single version to manage for the entire set of components. You may object that things are getting bumped even when they don't need to, but it turns out this is actually very well received by consumers, because it's far easier to upgrade than having to figure out the changelog of dozens of separate packages.

I used a single repo monolyth-but-actually-modular setup for my OSS project[2] and that has worked well for me, for similar reasons: people appreciate curation, and since we want to avoid willy-nilly breaking changes, a single all-emcompassing version scheme encourages development to work towards stability rather than features-for-features-sake.

My takeaway is that multi-repos cause a lot of headaches both for framework authorship and for service development, that single repos can be a great poor-mans choice for framework authors, and monorepos - with the appropriate amount of investment in tooling - have good multiplicative potential for complex project clusters. YMMV.

[0] https://github.com/uber-web/jazelle

[1] https://github.com/sebbekarlsson/fjb/tree/master/external

[2] https://mithril.js.org/

Bazel is still our bread and butter at Uber. Nothing in the JS space comes close in terms of features. I wrote a tool that wraps over it called jazelle

Oh sorry, we just moved it here https://github.com/uber-web/jazelle

This is surprisingly common. The other example off the top of my head, a single maintainer of a very popular project who had to temporarily abandon it due to lack of funds, is Denis Pushkarev (zloirock) and core.js (https://github.com/zloirock/core-js/blob/master/docs/2023-02...).

The majority of OSS projects have most of their contributions by one person (the project leader), and the vast majority of OSS contributors don't do it for their job. It seems nearly every single popular OSS project is like this (one unpaid, maybe sponsored, volunteer doing most of the work); it's not even worth listing projects and names, because you can just pick a couple projects you know and I bet at least one will be an example. Fortunately, most of these people seem to be well-off (probably in part due to the quality of programming jobs), but every once in a while there's someone who's not so fortunate. It should be more common to sponsor maintainers, especially if they are asking for donations provided they can prove that they really need the money (the world we live in, some people who have plenty fake issues to solicit donations, then others who genuinely need and deserve the money are scolded and left unfunded because of them).

Funny you say that, I was just thinking earlier today back to the core-js drama.

In short: the creator of a NPM package that is used by approximately everyone, everywhere, was facing a legal battle. He had been developing this package full time for years and did not have the cash on hand to hire a lawyer. He added a console log that ran on installing his package that said something like "If you're using core-js please consider donating". Queue an absolute shitstorm of people screaming at him in the github issues and him going to prison for around 10 months. Luckily he seems to be back on the grind nowadays, with a decently robust cross-platform slush fund to boot (~200k USD across Pateron, Open Collective, Bitcoin).

It can be a rough world out there for the folks building for the "focus, productivity and anti-distraction" platform.

https://github.com/zloirock/core-js

I think everyone here knows that, at some point, the site would start costing a lot of money and would need to be funded in some way. I would love for the Wikipedia donation model to work for a site like this but everything I find points to that not being the case. Reddit gold not covering server costs and open source devs not tied to a corporation struggling to continue working on their projects being two prime examples. If anyone has anything that can convince me to give it a try, please let me know and I will switch this to a non-profit.

Open source developers are not being paid. They published under licenses that allow zero cost and businesses won't pay.

If you want to write open source code for living, you have to find a business model that works. In this case, it is even under permissive license.

* code freeze - code is under open source license only a certain time after commit/release. Maybe add "support", aka you get security fixes in timely manner.

* open core - put some features behind commericial door.

* go ImageSharp way of split license. That one is fun, because MS deprecated/killed (throws exceptions on attempt to use) official image/font library and that was was intended replacement. Rather blatant offloading of costs.

This has been rehashed several time (core-js recently https://github.com/zloirock/core-js/blob/master/docs/2023-02...).

The gist of it is: Companies are not going to pay if they don't have to. That is the reality and it's not going to change.

This is one of the few examples. https://github.com/zloirock/core-js/blob/master/docs/2023-02-14-so-whats-next.md

Open source: It sounds pretty nice. Open to everyone... But it sucks in general. People really don't care to contribute to open-source. (e.g. here). It is a really good resource for development but for people who don't know anything about development, it is not important. There needs to be some financial income / support for good open-source.

there is https://github.com/zloirock/core-js but is more or less a 1 guy team and he is grossly under paid and well just read this https://github.com/zloirock/core-js/blob/master/docs/2023-02-14-so-whats-next.md im shocked he still works on it

Choice is good to a point but at some point it becomes crippling. It still haunts me on Rails. Is it yarn, is it brunch, is it npm, is it webpacker, is it esbuild, is it import maps... plus personally the pad-left debacle left a bad taste in my mouth and this little nugget about core-js was heartbreaking. For me it's hard to pick JS for anything other than what I absolutely must.

There's a real story behind that (but the software is core-js, not nginx)

npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3. \> [email protected] postinstall /home/daniel/src/test/node_modules/core-js > node -e "try{require('./postinstall')}catch(e){}" Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library! The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: > https://opencollective.com/core-js > https://www.patreon.com/zloirock Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)