This blog, the second in this series, introduces the advantages of a monorepo in the development of multi-service applications.
Why and What is Monorepo?
Modern web-based applications typically comprise multiple services, such as a back-end API and a front-end client. As projects grow larger, scaling may become an issue and services can be divided into multiple microservices. This raises the question of how to effectively manage the source code. One solution is to use a monorepo – a single repository for all source code in the project.
The monorepo approach provides easy access to the entire code base, and numerous benefits such as simplified dependency management and the ability to reuse code. However, this approach also has its drawbacks. For instance, per-service semantic versioning and deployment processes become more complex.
As the monorepo expands and more projects are added, inter-project dependencies may emerge. This can lead to constraints on the order in which jobs related to project A are executed with respect to those of project B, which depends on A.
What to look out for
Monorepos have many advantages, but there are important considerations. Managing dependencies is crucial. Services should not depend on each other, and changes outside the common area should not impact other projects.
Another consideration is handling updates to common libraries. The process for updating dependent projects must be clear, and you need a plan for how to handle cases where certain services need to remain on older versions of the library.
Security is also critical. Secrets should never be committed to the repository, and security protocols should be in place to prevent unauthorized access. Long-running tests can be a challenge, as testing each service and integration becomes increasingly time-consuming as the repository grows. Finally, repository issues such as git status errors or failed workflows can cause significant interruptions for teams working with projects in the repository.
CICD process in Monorepo architecture
Monorepos can be complex, but by leveraging CI/CD (Continuous Integration/Continuous Deployment), you can find the optimal solution.
Since Monorepos typically contain multiple projects, each maintained by separate developers or teams, it’s common to have a shared CI/CD workflow. However, running the entire pipeline from start to finish on every commit can be time-consuming and inefficient.
By implementing CI/CD optimization techniques, such as testing and building only the projects that have been changed, you can streamline the development process and reduce unnecessary workload. This approach, known as Continuous Integration, allows you to catch errors early and ensure that the code is always in a working state.
In addition to CI, you can also use Continuous Deployment to automate the deployment process. This ensures that changes are deployed quickly and consistently, without requiring manual intervention.
Lets look at CI/CD components and strategy.
Trigger pipelines for what has changed
In a Monorepo it is best to treat each solution or project as a separate entity. This means that when changes are made, you should only run testing, building, versioning, and releasing for that specific solution.
There are two ways to accomplish this. The first is to manually add triggers to the pipelines so they only run for the impacted solutions. This can be time-consuming and error-prone, as developers may forget to add necessary triggers.
The second option is an automated process that runs pipelines based on chained folders. This allows the pipeline to automatically detect which solutions have changed and only run the necessary tests and builds. This saves time and reduces the risk of errors caused by manual triggers.
Separate workflow for each path
In GitHub workflows, there is an option to specify a path filter. This allows the workflow to only run when changes have been made to a specific directory or file.
By combining the path filter with a branch filter, the workflow will only run if both filters are satisfied. For example, if a change is made to a file in a specific folder and that change is committed to the specified branch, then the workflow will start. This is helpful in a Monorepo because it allows developers to run automated processes only for the specific solutions that have been changed, rather than running them for the entire repository.
When working with a Monorepo, there may be many developers or teams working on the same repository at the same time. As a result, there’s a need for multiple releases per day. This can be challenging to manage when the repository is very active.
You can address this challenge by using Mainline Development. This involves merging all code changes, including features and bug fixes, directly into the main branch (also known as the master branch or mainline). The main branch is then tested and deployed.
For example: Team 1 creates a branch called “feature/abc-256/some-new-thing.” When it’s time for a release, instead of merging changes into a separate release branch, merge them directly into the main branch. Then test the main branch and if everything is good, release.
It’s important to note that Mainline Development may not be suitable for all projects. It works best when changes to the repository are frequent and small, and when there is a strong culture of testing and quality assurance. Mainline Development can also increase the risk of conflicts and regressions, so it’s important to have good monitoring and rollback mechanisms in place.
To enhance security, it is important to implement security scanning. This is particularly important when there are multiple applications in the same repository, and commits are frequent. Identifying security bugs in this scenario can be challenging, so it is recommended to perform security scans at the merge level. This way you can proactively detect and address security vulnerabilities before they become bigger problems.
There can be many more components in CICD including: Performance Testing, Automated Code Review, Dependency Management, and Environment Management.
Monorepo is a software development approach that involves storing all projects and application components in a single repository. Monorepos have many benefits including: improved collaboration, code sharing, and easier maintenance. They also require careful consideration and planning. Take into account scalability, dependency management, and versioning to ensure the Monorepo remains manageable and efficient.
When implementing a CI/CD process in a Monorepo, you should consider:
- trigger pipelines for what has changed
- separate workflows for each path
- release strategies
- security scanning
- infrastructure as code
- performance testing
- automated code review
- dependency management, and
- environment management.
Overall, the Monorepo approach can lead to faster and more efficient development and deployment, but requires a careful and strategic approach. By implementing a comprehensive CI/CD process with the right components, organizations can realize the benefits of Monorepo while managing accompanying complexity and challenges.
Darshan is an experienced platform engineer with a strong background in software development and over 6 years of experience. He specializes in Infrastructure-as-Code (IaC) and optimizing system performance to deliver reliable and scalable infrastructure.