What to look for (and look out for) in container registries

There has been a lot of movement in the world of container registries lately. And, with companies increasingly betting their businesses on container builds in their CI/CD pipelines, the stakes for container registries have never been higher. When CI/CD goes down, development grinds to a halt. That means we need to build resilience into our CI/CD systems, and the registry server is a key component for doing so.

A registry server is essentially a fancy file server that is used to store container images for Kubernetes, devops, and container-based application development. Developers can store and share container images by uploading to (pushing) and downloading from (pulling) a registry server. When a container image is pulled to a new system, the original application contained within it can be run on that system, as well.

In addition to container images, registries can store objects such as source code (source containers), security signatures (sigstore and cosign), application definitions for Kubernetes (Helm Charts) and even operating system updates themselves (RHEL for Edge). The registry server is quickly becoming a de facto standard for all kinds of data, making it ever more critical as an infrastructure component.

Choices, choices, choices…

In the past, the choice of container registry was hardly any choice at all: Docker Hub was pretty much it. Organizations relied on this service, and, not unlike GitHub, if it went down, their CI/CD systems went down with it. That’s still pretty much the case on both counts. Docker Hub (public and private) is still synonymous with container registries, and the health of a registry (and images within a registry) directly impacts organizations’ ability to quickly develop and deliver apps.

However, in the last few years a number of other container registries have sprouted up. For example, Quay has become a significant registry player. GitHub is also starting to invest heavily in its registry server. Meanwhile, each of the Big Three public cloud providers (AWS, Google Cloud, and Microsoft Azure) has its own registry server, and more and more companies are establishing their own private registry servers and/or using commercially supported private registry services.

Organizations put implicit trust in a registry server simply by using it, but it can’t be blind trust. The ease with which developers can pull images from any registry they want facilitates the quick adoption of new software (and, hence, quicker software delivery), but it also creates potential for security, compliance, and reliability problems.

Organizations must determine not only how much to trust the content provided by a registry, but also how much to trust a registry itself.

The convenience factor

Many dev teams decide to use a registry because it’s local. For example, it makes sense that a dev team using Azure Pipelines is going to use the Azure registry. It’s important, however, to ensure that a provider’s registry has enterprise-class capabilities, including support for multiple authentication systems, role-based access control management, vulnerability scanning capabilities, auditable logs, and automation.

In fact, most of the differentiation among container registries comes from tooling, and there will likely be two camps in an organization when determining which capabilities matter most. There will be a build use case, i.e., developers want a registry with a ton of content and a bunch of cool tools, and there will be a production use case, i.e., the prod team wants a registry that is super-reliable with strong security features, role-based access control, and resiliency capabilities.

As with any service, it’s likely that an organization might have one registry server for development work and a completely different, highly controlled registry server for distribution of container images in production clusters. There’s no need for any tension between development and operations about which capabilities matter more—they can each have their own registry server as necessary.

One big thing organizations need to ensure is that the registry is based on open standards. Fortunately, this is almost a non-issue today. Specifically, the Open Container Initiative (OCI) Distribution and Image specifications guarantee that everybody is pushing and pulling images to and from registry servers that are compatible with each other.

The one thing to watch out for is legacy and niche container technologies that don’t completely comply with OCI standards or only marginally comply with them. Pay attention to the technologies that are being adopted by the big technology companies, as they will generally protect you from adopting niche technology that does not comply with OCI standards.

The bigger picture

More generally, organizations need to be really thoughtful about how they are using container images and what’s going on in the industry.

In terms of the former, it’s all over the map. Some companies only allow the operations team to pull images from the internet. The ops team places the images into a private registry, and the dev team can pull only from this private registry. This approach creates a very controlled, almost air-gapped environment.

On the flip side, other companies let developers pull from wherever they want, which is kind of like letting every contractor manage its own supply chain contract. Nobody does that in manufacturing—everyone is super-careful about the supply chain, and rightly so. When it comes to the container supply chain, it’s too easy to pull in an image that was hacked. Most companies will be somewhere in the middle when it comes to where (and how) developers can pull down container images.

Changes in the industry can also affect the resilience of CI/CD systems. For example, Docker recently made a change to its terms and services that basically limited how often an image could be pulled (rightfully, to save bandwidth costs for free users). Docker provided warnings about the change, but not everyone heeded them, and many CI/CD systems broke as a result.

Organizations may not have paid much (if any) attention to Docker’s terms, as the Docker Hub service had been unlimited up until that time. However, with something as critical as the build system, everything must be done on purpose—nothing can be taken for granted. Developers didn’t expect the registry server to be the point of failure in their CI/CD system, but it turned out to be.

Container pushmi-pullyu

Operations and security teams need to have a hand in every container image that comes into an organization, as well as in the setup and maintenance of registry infrastructure. Operations teams should control the base images, and the lower layers of the software that come into the organization, and development should have control to put software on top of those base layers. This creates a clean demarcation between areas of responsibility (and non-repudiation). If OpenSSL gets hacked in a lower layer, it’s the responsibility of the operations team. If a Python library gets hacked in a higher layer, it’s the development team’s responsibility.

With so much riding on container registries, it is critical that organizations take nothing related to registries for granted. Understanding how the market is shifting, the role that open standards play, and the ways in which developers are pushing and pulling from registries is key to ensuring the health and resilience of the CI/CD pipeline—and, by extension, organizations’ ability to create, innovate, problem-solve, and compete.

At Red Hat, Scott McCarty helps to educate IT professionals, customers and partners on all aspects of Linux containers, from organizational transformation to technical implementation, and works to advance Red Hat’s go-to-market strategy around containers and related technologies.

—

New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to newtechforum@infoworld.com.