Probably the best tools to develop and build your monorepo.
These days, many tools can run “npm install” and “npm run build” in 20 different folders. But, not all tools can facilitate a proper monorepo.
Facilitating a proper monorepo development means solving challenges like operating the test and build processes for decoupled modules, being able to independently publish modules from the project, and managing the partial impact of changes on every affected dependant module in the project.
The list of challenges goes on, to include even “trivial” things like how you manage issues and PRs, which can get hard as you scale development.
Note that a monorepo is NOT a monolith application(!) — it is not built or deployed all at once. It is a group of applications developed separately.
In this roundup, I’ve gathered some of the best tools in the world to construct a “monorepo”, where you can build multiple modules inside a single project, with a decent developer experience that scales.
The list isn’t ranked and aims to outline the strengths of each tool based on its own merits. I hope this can help you save time and find the right tool.
Feel free to comment below and share your own insights. Cheers!
Late addition: I recommend using pnpm workspaces as first choice. Both are great really.
Yarn Workspaces aims to make working with monorepos easy, solving one of the main use cases for yarn link
in a more declarative way. Your dependencies can be linked together, which means that your workspaces can depend on one another while always using the most up-to-date code available. This is also a better mechanism than yarn link
since it only affects your workspace tree rather than your whole system.
Workspaces helps with a few issues, making it a great monorepo setup:
- It sets up a single
node_modules
without repetitions or cloning dependencies throughout different packages in the project. - All your project dependencies will be installed together, giving Yarn more latitude to better optimize them.
- Yarn will use a single lockfile rather than a different one for each project, which means fewer conflicts and easier reviews.
- It allows you to change the code of one of your packages and have those changes instantly visible to the other packages that use it. Any modification to one’s source code is instantly applied to the others.
As such, Yarn workspaces makes a very powerful combo with pretty much any tool on the list, and especially tools like Bit, Nx, and Lerna, acting as a lower-level layer of your monorepo management abstraction.
However, you can also publish directly with workspaces. When a workspace is packed into an archive, it dynamically replaces any workspace:
dependency with a package version, so you can publish the resulting packages to the remote registry without having to run intermediary steps — consumers will be able to use published workspaces as any other package. Cool!
Bit is a next-generation tool for building modular projects.
In short, it solves the key problems of a monorepo:
- Configurations on the workspace and component level
- Code sharing on the next level via composability
- Standardization of code, tech, and dev tasks
- Ownerships over components / features / experiences
- Build, test, and deploy changes on a graph
- Auto-defining and managing dependencies at app/component level
- Dev experience that stays simple at scale
It reates a workspace that lets you easily create and compose many components and applications, taking away the pains of configurations, and makes it easy to update changes, run build/test tasks, and even deploy changes across components and apps with next-to-zero configurations and in a super fast and modular way on the graph of components.
With just a few CLI commands you can setup an entire monorepo with React apps, a component library, packages, and super-performant build and test pipelines that work with just about any tool possible for testing etc.
$ bit new react my-workspace$ bit create react-app apps/my-react-app1 component(s) were created
my-name.my-scope/apps/my-react-app
location: my-scope/apps/my-react-app
env: teambit.harmony/<aspect> (set by template)$ bit run apps/my-react-appapps/my-react-app app is running on http://localhost:3000
With Bit, many of the common problems in a monorepo no linger exist. For example, code-sharing; You can consume and use any app, feature or component from any application or library to another with ease.
Bit also prevents and solves common pains with monorepo dependency management — here’s a short overview written by the maker of pnpm.
In essence, in with Bit you can easily develop, version, manage, build, test, document, deploy, and publish many “components” in the same project. Bit “knows” which component and app depend on each other, how, and where — so every change you make propagates smoothly across the moorepo.
Bit’s workspace manages all the relationships between components in the project. When you make changes to any component, Bit builds and tests it in isolation, and propagates the change up the dependency graph.
Components can be bulk published, as independent packages, to NPM and/or to the bit.dev platform for collaboration, consumption, and documentation.
Bit’s UI helps you view the development of your monorepo. As you code, and each component is documented, tested, built etc, you can visually see what’s happening with live feedback and hot-reloading.
Bit provides decoupled dev environments — Reusable and customizable modules that configure and “bundles” together different services needed throughout the life-cycle of an independent component such as compiling, bundling, testing, linting, documenting, and more.
NX is an advanced set of extensible dev tools for monorepos, with a strong emphasis on modern full-stack web technologies.
NX aims to provide a holistic dev experience via CLI (with editor plugins), and capabilities for controlled code sharing, and consistent code generation. It also provides incremental builds, so it doesn’t rebuild and retest everything on every commit you make, resulting in faster build times.
NX’s command execution allows for consistent commands to test, serve, build, and lint each project. It uses a distributed computation cache so if someone has already built or tested similar code, Nx will speed up the command for everyone else instead of rebuilding or retesting the code from scratch.
With Nx you can use your preferred framework, integrate with modern tools you’re probably already using. For example, NX lets you use out of the box integration with Cypress, Jest, Typescript, Prettier, and other tools.
The NX team also provides NX cloud, with smart computation memoization in the cloud and faster builds to help your team working with NX deliver faster.
Rush is a powerful monorepo infrastructure by Microsoft + Open Source. It aims to help you, well, build and publish many packages in one repository.
Some of the key features of rush include a single NPM install (also works with Yarn and pnpm) so you can install all dependencies for all projects into a common folder, using isolated symlinks to reconstruct an accurate “node_modules” folder for each project.
This also helps to ensure there are no phantom dependencies, so you won’t accidentally import a library that was missing from your package.json, and there are no dependency duplications where you find 10 copies of lib in your node_modules.
Automatic local linking means all your projects are automatically symlinked to each other, and when you make a change, you can see the downstream effects without publishing anything, and without any npm link
headaches.
Rush’s unique installation strategy produces a single shrinkwrap/lock file for all your projects that installs fast. Rush detects your dependency graph and builds your projects in the right order, so if two packages don’t directly depend on each other, Rush parallelizes their build as separate processes.
If you only plan to work with a few projects from your repo, Rush provides subset and incremental builds so rush rebuild --to <project>
does a clean build of just your upstream dependencies. After you make changes, rush rebuild --from <project>
does a clean build of only the affected downstream projects. And rush build
delivers a powerful cross-project incremental build. Rush even handles cyclic dependencies, by separating versions for projects.
When you want to release, Rush supports bulk publishing, so it detects which packages have changes, automatically bumps all the relevant version numbers, and run npm publish
in each folder.
Rush also helps to implement and enforce development policies. For example, when a PR is created, you can require developers to provide a major/minor/patch log entry for the affected projects, which will later be aggregated into a changelog file upon publishing. It also helps you enforce stuff like pre-publish reviews, specific dependency versions, and so on.
Lerna (named after the home of Hydra, the multi-headed beast) is a “tool for managing JavaScript projects with multiple packages”.
Lerna was created to solve the multi-package for Babel, to optimize the workflow of managing multi-package repositories with git and npm. It’s essentially tools and scripts to effectively manage and publish many independently versioned packages in a single Git repository.
my-lerna-repo/
package.json
packages/
package-1/
package.json
package-2/
package.json
The two primary commands in Lerna are lerna bootstrap
and lerna publish
.bootstrap
will link dependencies in the repo together. publish
will help publish any updated packages.
You can manage your project using one of two modes: Fixed or Independent.
Fixed mode Lerna projects operate on a single version line. The version is kept in the lerna.json
file at the root of your project under the version
key. When you run lerna publish
, if a module has been updated since the last time a release was made, it will be updated to the new version you're releasing. This is the mode that Babel is currently using.
Independent mode Lerna projects allows maintainers to increment package versions independently of each other. Each time you publish, you will get a prompt for each package that has changed to specify if it’s a patch, minor, major or custom change. Independent mode allows you to more specifically update versions for each package and makes sense for a group of them.
The ‘lerna.json’ file is a list of globs that match directories containing a package.json
, which is how lerna recognizes "leaf" packages (vs managing the dev dependencies and scripts for the entire repo). Example:
{
"version": "1.1.3",
"npmClient": "npm",
"command": {
"publish": {
"ignoreChanges": ["ignored-file", "*.md"],
"message": "chore(release): publish",
"registry": "https://npm.pkg.github.com"
},
"bootstrap": {
"ignore": "component-*",
"npmClientArgs": ["--no-package-lock"]
}
},
"packages": ["packages/*"]
}
Even if you’re not intending to publish to NPM, Lerna can still be helpful in managing versioning and common development tasks in a monorepo.
Google introduced the Bazel build system. It is an open-source build and test tool similar to Make, Maven, and Gradle that uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. It supports large codebases in a big monorepo or across multiple repositories, and large numbers of users.
Uber Developers uses Bazel to build their Go monorepo. Uber writes most of its back-end services and libraries in Go, which in 2018 were all grouped into a large Go monorepo which now have over 100,000 files. Bazel allowed this project to scale, cutting build times, and supporting its growth.
Here’s a nice small open-source project with Bazel as a demo:
Bazel is designed to work at scale and supports incremental hermetic builds across a distributed infrastructure, which is necessary for a large codebase. With Bazel’s remote cache, build servers can also share their build artifacts. Bazel caches all previously done work and tracks changes to both file content and build commands. A package is built and tested only when something has changed either in the package or its dependencies.
Bazel runs on Linux, macOS, and Windows. Bazel can build binaries and deployable packages for multiple platforms, including desktop, server, and mobile, from the same project. Many languages are supported, and you can extend Bazel to support any other language or framework.
Buck is a build system that encourages the creation of small, reusable modules consisting of code and resources, and supports a variety of languages on different platforms.
It is developed and used by Facebook, and has gained its fame by serving as the official build system of the FB monolith and thanks to being used by teams like Uber Developers to greatly reduce build times. The team at AirbnbEng marked 50% faster builds and 30% smaller apps.
Buck is designed to build a monorepo, and support for the monorepo design motivated Buck’s support for cells and projects.
It is Facebook’s experience that maintaining all dependencies in the same repository makes it easier to ensure that all developers have the correct version of the code and simplifies the process of making atomic commits.
Buck is commonly used for Android and iOS development. You can find more use-cases for buck here in this showcase page.
Buck can help you and your team in many ways:
Like other tools on the list, Buck builds independent artifacts in parallel to take advantage of multiple machine cores. It reduces incremental build times by keeping track of unchanged modules so that the minimal set of modules is rebuilt. And, Buck only uses the declared inputs, which means everybody gets the same results.
More cool features include buck query
, which helps you better understand your dependencies and what is required to build your product. And buck project
helps your IDE better “understand” the project your building.
In 2014, Twitter introduced its monorepo build system called Pants. Today, on v2, Pants aims to be a fast, scalable build system for growing codebases. It’s currently focused on Python, with support for other languages coming soon.
Pants uses a fine-grained workflow, and isolates each work unit from side effects, so it can utilize all available cores. Some of Pant’s best features include explicit dependency modeling, fine-grained invalidation, shared result caching, concurrent execution, remote execution, and extensibility and customizability via a plugin API.
The Pants engine is written in Rust, for performance. The build rules are written in typed Python 3, for familiarity and simplicity. The engine is designed so that fine-grained invalidation, concurrency, hermeticity, caching, and remote execution happen naturally, without rule authors' intervention.
Not using Python? Move on, nothing to see here. But if you do, you might want to take Pants for a test run before setting up your build system.
Please is a cross-language build system with an emphasis on high performance, portability, extensibility and correctness.
Please makes sure build steps are executed in their own hermetic environment, with access to only files and env variables that they have been given access to. Incremental builds means it only builds what it needs to, and it also provides task parallelism, and distributed caching, for a reliable and performant build system at scale.
Please also aims to focus on dev experience, so you can enjoy a famlar CLI and define aliases for common tasks leveraging auto-completions. Written in Go, Please provides all this user experience with no runtime dependency. And, there’s no single large workspace file with too much configs to handle.
It as an intriguing option to explore as your monorepo’s build system.
Oao isn’t the most mature, rich, or easily usable tool on the list, but it’s interesting none the less. It is a Yarn-based, opinionated monorepo management tool thatp provide monorepo features like installing all dependencies, adding/removing/upgrading sub-package dependencies, validating version numbers, determining updated sub-packages, publishing everything at once, updating the changelog, etc.
Oao lets you run a command or package.json
script on all sub-packages, serially or in parallel, optionally following the inverse dependency tree. And, it supports yarn workspaces, optimising the monorepo dependency tree as a whole and simplifying bootstrap as well as dependency add/upgrade/remove.
Support for non-monorepo publishing: benefit from oao’s pre-publish checks, tagging, version selection, changelog updates, etc. also in your single-package, non-monorepos. Note that Oao uses a synchronized versioning scheme so a master version is configured in the root-level package.json
, and sub-packages will be in sync with that version . You can try it out here.
Boltpkg aims to be a a “ Super-powered JavaScript project management tool”.
Bolt implements the idea of workspaces on top of Yarn. Bolt CLI is largely a replacement of the Yarn CLI. You can use it on any Yarn project.
As we know, workspaces are nested within a larger project/repo. Each workspace can have its own dependencies with its own code and scripts. Workspaces can also be grouped into sub-directories for organization.
Using Bolt, you can install the dependencies for all of these packages at once (and you can do it really really fast). And, when you specify a dependency from one workspace to another, it will get linked to the source. This way, when you go to test your code, all your changes get tested together.
Watch out, some of these are no longer actively maintained. ✋
FAQs
Is monorepo a good idea? ›
Monorepos definitely bring a lot of benefits when it comes to organizing teams working with related projects. They help you improve the way you work, save time with less code and even share devs between projects a lot easier. That is all true, if you have a very well-defined and accepted set of rules.
What is monorepo tool? ›What is a monorepo? A monorepo is a version-controlled code repository that holds many projects. While these projects may be related, they are often logically independent and run by different teams. Some companies host all their code in a single repository, shared among everyone. Monorepos can reach colossal sizes.
Should I use a monorepo for microservices? ›A monorepo removes barriers and silos between teams, making it easier to design and maintain sets of microservices that work well together. Standardization. With monorepos, it is easier to standardize code and tooling across the teams.
Does Microsoft use a monorepo? ›At Microsoft, we build monorepos with hundreds of projects.
Does Netflix use monorepo? ›Netflix has a multi-repo, we use library version management very extensively. We made the decision early on not to do monorepo, and it has a lot of tradeoffs in terms of velocity versus stability.
Does Google still use a monorepo? ›This practice dates back to at least the early 2000s, when it was commonly called a shared codebase. Google, Meta, Microsoft, Uber, Airbnb, and Twitter all employ very large monorepos with varying strategies to scale build systems and version control software with a large volume of code and daily changes.
Does Facebook use monorepo? ›Facebook has one such example of a monorepo: With thousands of commits a week across hundreds of thousands of files, Facebook's main source repository is enormous—many times larger than even the Linux kernel, which checked in at 17 million lines of code and 44,000 files in 2013.
Does AWS use a monorepo? ›Product Manager Technical, AWS. Amplify Console recently launched monorepo support, providing developers with mono-repositories a better experience connecting apps to the Amplify Console. A mono-repository is a repository that contains more than one logical project, each in it's own repository.
Does twitter use a monorepo? ›"Source", Twitter's monorepo, spans almost 20 million lines of hand-crafted code and ten times as much of generated code. Most of it is Scala, but we also support Java, Python, and to a lesser extent NodeJS, Go and C/C++.
What is the opposite of a monorepo? ›Polyrepos or Multirepos are the opposite of Monorepos. Multiple distinct projects in different “git” repositories is considered a Polyrepo. Contrarily, in a Monorepo, we have all our applications in the same “git” repository, and they can be deployed separately.
Is angular a monorepo? ›
A multi-project workspace is suitable for an enterprise that uses a single repository and global configuration for all Angular projects (the "monorepo" model).
Is monorepo micro frontend? ›Application. We going to build the apps in monorepo, which is going to have an external store as a micro frontend app, dashboard, and header. All of them are gonna be consumed by the host app. The host will use remote config to consume the store , dashboard , and header .
Is Linux a monorepo? ›The Linux kernel is a monorepo and it's probably still at least an order of magnitude bigger than your project (it is bigger than ours anyway, despite having hundreds of engineers involved at this point). Basically, you're not Google or Microsoft.
How do you organize a monorepo? ›In order to properly manage a monorepo, you should organize the various components of your codebase to be separated in a logical directory structure. Related code should be close in the directory tree, and work that involves different teams should be separated into different directories.
What backend does Netflix use? ›All these apps are written using platform-specific code. Netflix web app is written using react JS, and react. js was influenced by several factors the first is startup speed, the second is a runtime performance of react. js, and the third one is modularity.
Why do big companies use monorepo? ›The big companies that predate git and such used monorepos because that was the norm at the time, and it was easier to do with the tools at the time, and as they scaled, they just scaled their process instead of changing everything.
How big is Google's monorepo? ›Google teams had not necessarily planned to create one of the world's largest and most valuable repositories. Google's heritage and active asset are of considerable size: 2 billion lines of code representing 86 Tb of storage.
What is monorepo vs Polyrepo? ›Monorepo means using one repository that contains many projects, and polyrepo means using a repository per project.
Is GitLab a monorepo? ›While some might believe that monorepos are a no-no, there are valid reasons why companies, including Google or GitLab (that's right! We operate a monolithic repository), choose to do so.
Why doesn t Google use Git? ›There's a number of principles that make developing code at Google very different from how it's done in most companies, whereas Git has been designed with a specific use case in mind. Schematically, development at Google is very centralized, and the original Git project was very decentralized.
What is a NX monorepo? ›
A monorepo is a single git repository that holds the source code for multiple applications and libraries, along with the tooling for them.
What does NASA use AWS for? ›AWS allows NASA to quickly and easily deploy and manage large-scale applications and services. This helps NASA to improve its efficiency and ensure that its data is always accessible and reliable.
Is NASA using AWS? ›On Mars, powered by the cloud: Mars 2020 rover launches
The images taken from the Mars 2020 rover will be available on NASA JPL's public website, and will be shared with people around the world. All images returned from the mission will be hosted on the AWS Cloud.
JPMorgan Chase & Co. is using Amazon Web Services (AWS) to evolve its business for today's technology revolution.
When would you use monorepo and when multi repos? ›The monorepo approach entails storing the code for different libraries or projects — and even all code from a company — in a single repository. And the multi-repo system divides the code into units, such as libraries or services, and keeps their code hosted in independent repositories.
What is a monorepo and why you should care? ›A monorepository is a code management and architectural concept whereby you keep all your isolated bits of code in one super repository instead of managing multiple smaller repositories—like a single repository for your website and mobile apps.
What backend does Twitter use? ›Websites | Popularity (unique visitors per month) | Back-end (Server-side) |
---|---|---|
290,000,000 | C++, Java, Scala, Ruby (Ruby On Rails) | |
Bing | 285,000,000 | C++, C# |
eBay | 285,000,000 | Java, JavaScript, Scala |
MSN | 280,000,000 | C# (ASP.NET) |
Because of the nature of monorepos, certain issues can be experienced. These include unclear boundaries, less sense of ownership, and collaboration. Consequently, projects in a monorepo are prone to depend on each other, creating tight coupling. This leads to brittle software.
Should microservice have its own DB? ›An important rule for microservices architecture is that each microservice must own its domain data and logic. Just as a full application owns its logic and data, so must each microservice own its logic and data under an autonomous lifecycle, with independent deployment per microservice.
Which is the best suited practice for microservices development? ›- The Single Responsibility Principle. ...
- Have a separate data store(s) for your microservice. ...
- Use asynchronous communication to achieve loose coupling. ...
- Fail fast by using a circuit breaker to achieve fault tolerance. ...
- Proxy your microservice requests through an API Gateway.
Should I use gRPC for microservices? ›
By using the built-in features in gRPC, you can improve both the reliability of your product and the productivity of your entire team by using Go Microservices. gRPC provides better performance and security than other protocols like REST with JSON or XML communication, as it uses Protocol Buffers and HTTP/2 over TLS.