CI/CD buyer’s guide: How to choose a cloud CI/CD platform

Cloud CI/CD platforms explained

If your goals are high-velocity software development and frequent delivery of working builds to production, you need to automate at least part of the testing and delivery process. Ideally, that means implementing continuous integration/continuous delivery (CI/CD) pipelines for your projects, along with test suites to catch errors before customers see the software, and scripts that implement the steps of the pipelines.

Continuous integration (CI) is a methodology for automating software builds, packaging, and tests in a consistent way. CI helps to give a team some confidence that changes they check into source-code version control will not break the build or introduce bugs into the software. The endpoint of CI is typically a completed check-in to the main branch of a software repository.

[ Download our editors’ PDF cloud CI/CD enterprise buyer’s guide today! ]

In this buyer’s guide

• Cloud CI/CD platforms explained
• What to look for in cloud CI/CD platforms
• Leading vendors for cloud CI/CD platforms
• Essential reading

Continuous delivery (CD) automates the delivery of tested software to infrastructure environments. That doesn’t usually mean throwing it directly into production to see if customers complain. Typically, organizations start by pushing the build to a development environment. After the developers themselves beat on the new build and release it, it usually goes to a test environment, where it is used by a broader group of users (sometimes just dedicated internal testers, sometimes a larger cadre of users signed up for beta testing or “dog-fooding”) and monitored closely. Finally, if all goes well, the testers sign off and push the new version to a production environment.

At each stage of CD, there are options to revert quickly to an older build and generate bug report tickets for the developers to address in the new build. The goal is not to push lots of builds into production, but rather to continuously improve and enhance the software without introducing regressions. Another term for these practices is devops.

Hosting a CI/CD platform in your own data center is a viable option, especially for companies that mandate hosting their applications and data inside the firewall. The disadvantage of doing this is that you need a dedicated team to maintain the infrastructure, and you incur some capital expenditures for servers.

If you are allowed to host in the cloud, that is usually a better option. The cost of hosting in the cloud is modest, and that operating expense is offset by the services provided: onboarding, infrastructure maintenance, security maintenance, support, and CI/CD software maintenance. Hosting your CI/CD software in the cloud often makes it easier and faster for the pipelines to interact with your source code repositories, if they are also in the cloud. If your developers and testers are geographically distributed, hosting your repositories in the cloud frequently gives developers a better experience than hosting in remote servers behind firewalls.

It’s also possible to deploy CI/CD on a hybrid of on-premises and cloud servers. Several of the latest CI/CD offerings run in containers on Kubernetes clusters, which are equally happy running on-premises and in the cloud. In a hybrid deployment scenario, you can place each component where it makes the most sense given the physical location of the developers themselves and the network locations of the other servers in the development infrastructure.

What to look for in cloud CI/CD platforms

CI/CD will be a critical part of your infrastructure once you have it fully implemented. Bear that in mind as you get up to speed. It’s important to perform a rigorous proof of concept before starting to roll out the CI/CD pipelines. Shake down the CI portion before beginning the CD phase. Make sure you exercise your test suites and rollback capabilities before connecting any CI/CD pipelines to production instances, and keep people in the loop until you’re very sure that the automation is rock-solid.

There are seven key areas to investigate before choosing a cloud CI/CD platform.

1. CI/CD must integrate with your repositories: As you may have gathered when you read “the endpoint of CI is typically a completed check-in to the main branch of a software repository” earlier, repos are essential to CI and CD. Beyond being the endpoint of the check-in and test process, software repositories are the preferred place to store your CI and CD scripts and configuration files. Yes, many of the CI/CD platforms can store scripts and other files internally, but you are usually better off having them in version control outside of the tool.

Almost all CI/CD tools can interact with Git. Some also integrate directly with GitHub, GitHub Enterprise, GitLab, and/or Bitbucket. A few also support Subversion and/or Mercurial.

2. Your CI/CD tools need to support your programming languages and tools: Each programming language or language group (JVM languages, LLVM compiled languages, .Net languages, and so on) tends to have its own build tools and testing tools. To be useful to you, a CI/CD tool must support all the languages that are part of a given project. Otherwise, you might need to write one or more plug-ins for the tool.

Docker images are becoming more and more important to distributed, modular, and microservices software deployments. It helps a lot if your CI/CD tool knows how to deal with Docker images, including creating an image from your source code, binaries, and prerequisites, and deploying an image to a specific environment. Again, lacking this, you might need to write plug-ins or scripts to implement the Docker functionality you need. Similarly, you’ll want your CI/CD tool to support Kubernetes and any other container orchestration systems that you use in your environments.

3. Do your developers understand CI/CD and the tools you’re considering? The principles of CI and CD may seem obvious, but the details are not. The various CI/CD tools have differing levels of support and documentation. There are multiple books on Jenkins, which isn’t surprising because it’s the oldest of the bunch. For other products, you may have to investigate the documentation and support forums and paid support options as part of your due diligence in picking a tool.

For general background on CI, consider the Addison-Wesley book Continuous Integration by Duvall et al. Similarly, for general background on CD, consider Continuous Delivery by Humble and Farley. Both books won Jolt awards when they were published.

4. You can choose different CI/CD tools for different projects: While this guide is about choosing a CI/CD platform, please don’t assume that one platform will be optimal for all your software development projects. Most shops use multiple programming languages and environments, and not every CI/CD platform will support all of them well.

Feel free to pick the CI/CD platforms that work best for each of your projects rather than finding a single compromise platform. The general principles of CI and CD carry over from one platform to another, even though the scripts you write for them may not always be portable. While the additional onboarding time for each new platform may cost your devops team some time, that’s most likely less expensive than needing to customize a CI/CD tool extensively.

5. Plan for future CI/CD migration: Along the same lines, please don’t assume that a given CI/CD platform will serve the needs of your projects forever. Always hedge your bets, for example by storing scripts in repositories rather than within the CI/CD tool.

6. Prefer serverless CI/CD where appropriate: In general, cloud container deployments are less expensive than cloud server instance deployments, and serverless cloud deployments are less expensive than container deployments. Unfortunately, few CI/CD platforms can run serverless as of this writing.

Serverless means that the container running the process of interest is instantiated as necessary, generally in response to an event. For CI/CD, the triggering event is generally a code check-in to a specific repository branch; the repository Webhook then launches the serverless process. When the process completes, the resources are released.

One the few CI/CD platforms that can run serverless is Serverless CI/CD, part of Serverless Framework Pro, an enhanced version of the open source Serverless Framework. Serverless CI/CD is optimized for deploying serverless apps and currently runs only on AWS. You’ll have to determine whether it supports your application well enough to use.

7. Where are your current cloud assets? To optimize a cloud-based CI/CD configuration (or any cloud application), it helps if all your assets are near each other. “Near” in this context partially refers to geographic location, and it also partially refers to network location.

For example, if your database is in Australia and your application is in North America, you’re going to have a large lag every time the application needs to read or write data. On a smaller scale, if your application and database are in the same availability zone (AZ), the latency between them will be minimized. If they are in different zones in the same region, the latency will be higher, but not as high as if they were in different regions.

Similarly, if your database is on Google Cloud Platform in Virginia and your application is on Microsoft Azure in Virginia, the latency will be higher than if both were on the same cloud provider in the same AZ. All of this also applies to your repository (which is essentially a database), your CI/CD software, your actual application, and your developers and testers. It helps if all are “nearby,” although the effects of lag aren’t as blatant in this situation as they would be for, say, a real-time interactive game.

If the developers can push code commits into the master repository reliably and without noticeable wait times, they’ll usually be happy. Similarly, if the users and testers are “near” enough to the application to get sub-second response times, they’ll also be happy. For the CI/CD software, the key is reliable connections to the points of deployment. A little lag can be tolerable as long as it doesn’t cause time-outs or dropped packets.

Leading vendors for cloud CI/CD platforms

There are a surprising number of cloud-based services that support devops. I’ve listed 14 cloud CI/CD product vendors here as examples. Note that inclusion in this list is not a recommendation, and exclusion is not a condemnation.

AWS CodePipeline: CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. CodePipeline automates the build, test, and deploy phases of your release process every time there is a code change, based on the release model you define. This enables you to deliver features and updates rapidly and reliably. You can easily integrate AWS CodePipeline with third-party services such as GitHub or with your own custom plugin.

Azure DevOps: Azure DevOps provides developer services that help teams to plan work, collaborate on code development, and build and deploy applications. You can work in the cloud using Azure DevOps Services or on-premises using Azure DevOps Server. Azure DevOps provides integrated features that you can access through your web browser or integrated development environment (IDE) client. Azure Pipelines provides build and release services that support continuous integration and delivery. Other standalone services include Azure Repos (Git and Team Foundation source control), Agile Boards (a suite of agile tools), Azure Test Plans (tools for manual/exploratory testing and continuous testing), and Azure Artifacts (allows teams to share packages from public and private registries).

Bitbucket Cloud: Atlassian’s Bitbucket Cloud combines Git code hosting and source control and a built-in CI/CD solution in Bitbucket Pipelines. It also offers integrations with Atlassian’s Jira (feature and issue tracking) and Trello (kanban-style project management) tools, giving development teams one central place to plan projects, collaborate on code, test, and deploy.

Bitrise: Bitrise is a devops and CI/CD platform geared for developers of mobile applications. It supports a wide range of mobile technologies, including Cordova, Flutter, Ionic, Java, Kotlin, Objective-C, React Native, and Swift. You can integrate Bitrise with Bitbucket, GitHub, GitLab, and other Git source control services, both in the cloud and on premises. Bitrise is billed as a “full-featured mobile CI/CD in the cloud, for any platform, with no hardware required.” This means that, in addition to builds running on Linux machines, macOS builds are also fully included in all plans, including those for free users and open-source projects.

CircleCI: CircleCI can automate builds across multiple environments, both cloud-based and on on-premises, integrates with Bitbucket, GitHub, and GitLab, and deploys to targets ranging from Amazon Web Services, Google Cloud, and Microsoft Azure to Artifactory and NPM. You can choose from thousands of existing integrations or create your own, optimize builds for speed and run jobs in parallel, and ensure that your projects are built, deployed, and maintained securely. Build environments include Docker, Linux (including Android emulators), macOS, Windows, GPUs, and Arm.

CloudBees: CloudBees is a software delivery automation solution for large and compliance-first organizations that provides software development teams with Jenkins-based CI/CD, release orchestration, feature management, and value-stream management capabilities, while providing management with risk mitigation, compliance, and governance capabilities. The CloudBees platform provides tools that allow organizations to create self-service, fast, and secure workflows and to connect software delivery to business outcomes.

Codefresh: Codefresh is a software delivery platform for Kubernetes powered by Argo, using a GitOps model where infrastructure as code is versioned in a repository. Codefresh uses Argo CD, Argo Workflows, Argo Events, and Argo Rollouts, extended with functionality and features essential for enterprise deployments. Codefresh offers security, maintainability, traceability, and a single control plane for all stakeholders, be they developers, operators, product owners or project managers. Codefresh supports GitHub, GitLab, and Bitbucket.

GitHub Actions: GitHub Actions is a GitHub-hosted CI/CD platform that allows you to automate your build, test, and deployment pipeline. You can create workflows that build and test every pull request to your repository, or deploy merged pull requests to production. GitHub Actions also lets you run workflows when other events happen in your repository. (For example, you can run a workflow to automatically add the appropriate labels whenever someone creates a new issue in your repository.) GitHub provides Linux, macOS, and Windows virtual machines to run workflows, or you can run them in your own virtual machines, in the cloud or on-premises, using self-hosted runners. GitHub Actions supports Go, Java, .Net, Node.js, PHP, Python, Ruby, Rust, and Swift.

GitLab CI/CD: GitLab CI/CD is the part of GitLab that you can use for all of the continuous methods (continuous integration, continuous delivery, and continuous deployment). With GitLab CI/CD, you can build, test, deploy, and monitor your applications, with no third-party application or integration needed. GitLab automatically detects your programming language and uses CI/CD templates to create and run default pipelines to build and test your application. Then, you can configure deployments to deploy your apps to staging and production, and set up Review Apps to preview your changes per branch. Thus GitLab CI/CD helps you to catch bugs and errors early in the development cycle and ensure that all the code deployed to production complies with the code standards you established for your app.

Google Cloud Build: Google Cloud Build is a cloud-hosted, fully managed CI/CD platform that lets you run builds across Google Cloud, other public clouds, and on-premises systems, or strictly within your own private network. You can create pipelines as a part of your build steps to automate deployments, and deploy using built-in integrations to Google Kubernetes Engine, Google App Engine, Google Cloud Functions, and Google Firebase. You can set up triggers to automatically build, test, or deploy source code when you push changes to GitHub, Google Cloud Source Repositories, or a Bitbucket repository. Cloud Build can be used with Spinnaker for creating and executing complex pipelines.

IBM Cloud Continuous Delivery: IBM Cloud Continuous Delivery provides toolchains, pipelines, and tool integrations for software delivery. You can create toolchains that include IBM services, open source tools. or third-party tools that make development and operations repeatable and easier to manage. You can build, test, and deploy in a repeatable way with minimal human intervention by leveraging Tekton-based delivery pipelines. You can manage your source code and track work with Git repositories and issue tracking hosted by IBM and built on GitLab Community Edition. And you can quickly provision toolchains for devsecops, deployments to Kubernetes, Cloud Foundry, virtual machines, and more with shareable, customizable templates that include IBM, open-source, and third party tools.

Spinnaker: Spinnaker is an open source, multicloud CD platform designed to help you release software changes with high velocity and confidence. Spinnaker combines application management features that allow you to view and manage your cloud resources with application deployment features that allow you to construct and manage continuous delivery workflows. A mature and widely productionalized CD platform, Spinnaker can apply the expertise of Amazon, Google, Microsoft, and Netflix to your software development life cycle.

TeamCity: TeamCity is a general-purpose CI/CD solution from JetBrains that allows the flexibility for all sorts of workflows and development practices. TeamCity lets you write CI/CD configuration using Kotlin, giving you the power of a full-featured programming language and its toolset. TeamCity is integrated with all version control systems, not just Git. It natively supports Java, .Net, Python, Ruby, and Xcode, plus other languages through plugins. TeamCity is also integrated with Bugzilla, Docker, Jira, Maven, NuGet, Visual Studio Team Services, and YouTrack.

Travis CI: Travis CI is a CI/CD tool that provides flexible automation for your builds, tests, and deployments. In addition to automatically building and testing code changes, Travis CI can automate other parts of your development process by managing deployments and notifications. This means you can have jobs depend on each other by using Build Stages, set up notifications, prepare deployments after builds, and perform many other tasks. Travis CI was the first CI service that provided services to open source projects for free, and continues to do so. Travis CI supports Assembla, Bitbucket, GitHub, and GitLab, as well as more than 30 programming languages.

Essential reading

• What is CI/CD? Continuous integration and continuous delivery explained
• 5 best practices for securing CI/CD pipelines
• Version control: Track the who, what, and when of software changes
• What is devops? Bringing dev and ops together to build better software
• What is Kubernetes? Your next application platform