How Informed Customizes CircleCI Workflows for Efficient Monorepo Deployment

How Informed Customizes CircleCI Workflows for Efficient Monorepo Deployment Informed

This is the fourth post in our CI/CD series. We’ll discuss how CircleCI helps with deploying monorepo components in a microservice way. Additionally CircleCI’s support for monorepo, better observability and debugging tools, and synchronous deployment of apps and infrastructure, make it a great tool for this purpose.

CircleCI’s Support for Monorepo

CircleCI allows us to create a single repository to store code for all of our applications. This makes it easier to manage and track changes across applications. CircleCI also supports a primary-secondary approach. In this method, a config file sits at the top-level of a repo under the .circleci folder – the primary config file. CircleCI drives all of its workflow from this file. Secondary config files sit at the application level and drive the workflow for that particular application.

Better Observability and Debugging Tools

CircleCI’s better observability and inherent debugging tools are helpful – we can quickly identify and resolve issues in our CI/CD pipeline. We also use the pre-defined orbs to support our workflows. These orbs are similar to pre-built Docker images, providing a modularized approach to creating workflows.

Synchronous Deployment of Apps and Infrastructure

Based on any changes, CircleCI triggers workflows that are coupled together, ensuring synchronous deployment of apps and infrastructure. We have multiple environments, and the same deployment pattern is used to deploy to each environment.There are many CircleCI triggers such as merge to the main branch, external calls, Slack messages, or other third-party integrations.

Creating Customized Workflows Based on Changes Made in Monorepo

In software development, it’s common to have multiple services or applications dependent on each other. Managing the deployment and integration of these services becomes complicated when changes need to be made across multiple components.

To simplify the management of multiple projects or services within a monorepo codebase, the team developed a “commons” component that includes modularized jobs. These jobs can be reused across the monorepo, allowing for modularized workflows that can be referenced anywhere in the application workflows.This approach makes it easier to customize workflows based on changes, enabling the team to streamline the deployment process.

We utilize a modularization approach for our CI/CD pipeline that provides flexibility. The CircleCI master config file automates the deployment process for changes that affect both the application and infrastructure. A job is triggered upon merging a PR into the monorepo, which sets a flag and creates a customized workflow. This allows us to easily trigger the deployment of a particular service in a microservice-style approach, ensuring consistency and efficiency in the deployment process. These changes range from small updates to the introduction of a new feature.

This modularization approach provides several benefits:

  1. It enables the team to treat each service as a microservice deployment, including dependencies. This simplifies the deployment process, making it easier to manage and maintain.
  2. It allows the team to make changes to the monorepo while minimizing the impact on other services. This reduces the likelihood of errors or conflicts between services, improving stability. 
  3. Finally, it enables the team to scale their CI/CD pipeline more efficiently, by reusing modularized workflows across multiple services, saving time and resources. 

Below is the file structure of monorepo which shows common components consisting of different jobs and config.yml placed at different applications’ root level.

How Informed Customizes CircleCI Workflows for Efficient Monorepo Deployment Informed

Now let’s look at custom workflows in detail.

Implementing different Workflow types for Efficient Service Development and Deployment

We understand the importance of maintaining a robust and reliable software development lifecycle (SDLC) to ensure the delivery of high-quality software. To achieve this, we have implemented three custom workflows that cover various aspects of the SDLC. These include security checks and unit testing, artifact creation, and deployment to different environments. 

These workflows are designed to work seamlessly with our DevOps toolchain, providing a streamlined and efficient development experience. Below, we’ll provide an in-depth overview of these workflows, highlighting their key features, benefits, and technical components.

Workflow 1: Pull Request

The Pull Request Workflow is triggered when a pull request is raised in a repository. This workflow is responsible for checking:

  • All security aspects, 
  • Running unit tests, 
  • Enforcing best practices. 
  • If the repository contains infrastructure code written in Terraform, the workflow also checks the formatting and naming conventions.

The workflow starts by cloning the repository and running a security scanner to detect any vulnerabilities in the code. The scanner checks for vulnerabilities in third-party libraries, code dependencies, and the code itself. If any are detected, the pull request is marked as failed and notified dev, and the code must be updated to fix the issues.

Next, the workflow runs the unit tests to ensure that the code changes have not broken any existing functionality. The tests are run in a circleCI environment to prevent interference with the production environment. 

Finally, if the repository contains Terraform code, the workflow checks the formatting and naming conventions to ensure consistency across. If any issues are found, the pull request is rejected, and the code is updated to fix the issues.

Workflow 2: Artifact Creation

The Artifact Creation Workflow is triggered when changes are merged into the main branch of the repository. This workflow is responsible for building the code and creating an artifact that can be deployed to any environment.

The workflow starts by building the code and packaging it into an artifact format suitable for deployment. The type of artifact depends on the type of application. For example, if the application is a Java application, the artifact could be a Java Archive (JAR) file.

Next, the workflow tags the commit with a version number to identify the artifact. This ensures that the same version of the artifact is deployed consistently across all environments.

Finally, the workflow uploads the artifact to a package repository, such as ECR or S3 bucket, where it can be accessed by the Promote Artifact Workflow.

Workflow 3: Promote Artifact

The Promote Artifact Workflow is triggered manually using the invoke tool when an administrator wants to promote an artifact to a different environment, such as staging or production.

This workflow starts by retrieving the artifact from the package repository. Next, it deploys the artifact to the target environment. This could involve provisioning infrastructure, configuring load balancers, attaching layers, updating DNS records or deploying lambda with the appropriate artifact from an S3 bucket or ECR based on the lambda type. 

Finally, the workflow notifies the administrator of the success or failure of the deployment. If the deployment was successful, the administrator can promote the artifact to the next environment, such as  production. If the deployment failed, the administrator can roll back to a previous version of the artifact.

It’s important to note that the workflows we’ve described are just examples and can be customized based on  specific needs. For example, some applications may require additional security checks or compliance testing, while others have unique build or deployment requirements. By tailoring workflows to each application’s needs, we ensure that our development process is efficient, reliable, and secure.

To recap: The Pull Request Workflow ensures that changes are thoroughly tested before being merged into the main branch. The Artifact Creation Workflow ensures that artifacts are built and packaged consistently. The Promote Artifact Workflow ensures that artifacts are deployed consistently across all environments.

Conclusion

Our deployment structure, consisting of three types of workflows, enables us to maintain code quality, automate testing and promote artifacts across different environments. With this approach, we are able to deliver high-quality software releases to our customers in a timely and efficient manner.

author avatar
Darshan Shah Platform Engineer
Darshan is an experienced platform engineer with a strong background in software development and over 6 years of experience. He specializes in Infrastructure-as-Code (IaC) and optimizing system performance to deliver reliable and scalable infrastructure.

Upcoming Webinar: Crossing the chasm into the new digital world: The impact of AI and automation in creating a fully digital auto ecosystem

X