Exposing External-Facing Services In Kubernetes

What is Kubernetes?

Kubernetes is an open-source clustered container orchestration platform that works across clusters of machines.

In Kubernetes, containers belonging to each application are grouped into work units called pods which are scheduled on specific worker nodes in the cluster on a resource-availability basis.

This is analogous to how an operating system’s CPU scheduler decides which processes receive CPU cycles at any given moment. Other OS process scheduling concepts like affinity and priority also have Kubernetes equivalents as well. If a pod needs more resources, it can be scaled vertically by changing the resource limits in its manifest.

Situations do arise when it is necessary to have multiple pods running the same container for reasons such as load balancing or high availability; an action known as horizontal scaling. Pods are horizontally scaled and managed using deployments. Pods running the same application are grouped together into services which provide a single point of access for the pods they represent; a service’s IP remains the same even when its backend pods are scaled or rescheduled. Services employ a selector which identifies backend pods based on their labels, arbitrary tags that can be used to group pods together.

When a service is deployed in Kubernetes it is easily accessible from other services in the cluster via kube-dns. But what if you want to host a service meant to be accessed from outside the cluster?

The Options

There are three ways to expose an external-facing service in Kubernetes which differ in their ServiceType specification; as a NodePort, LoadBalancer, or ClusterIP with Ingress.

NodePort LoadBalancer ClusterIP with Ingress
Each service is accessible on its own port in the form <NodeIP>:<Port>. Each service provisions its own LoadBalancer from the cloud provider. All services share a LoadBalancer and requests are proxied to each service.
The method most commonly used in production environments is ClusterIP with Ingress. This is due to high scalability, accessibility, and low infrastructure usage; reasons which this post will elaborate upon! When services are exposed in this way, there is only one load balancer and wildcard DNS record required and clients do not have to keep track of a service’s port number. All services use a common point-of-entry which simplifies development, operations, and security efforts. Routing rules and provisioning are done within the manifests of Kubernetes native objects, allowing for easy configurability.

ClusterIP With Ingress

In this example, a request to xyz.bar.com is directed to xyz-container and a request to foo.bar.com is directed to foo-container. This type of proxying is known as name-based virtual hosting. The ingress-controller pod runs a container which handles the proxying – most commonly this is implemented using nginx, an open-source reverse proxy and load balancer (among other cool features!) although other implementations such as HAproxy and Traefik exist. Proxy rules are defined in the nginx configuration file /etc/nginx/nginx.conf and kept up-to-date by a controller binary that watches the Kubernetes API for changes to the list of external-facing services, known as ingresses; hence the name ingress controller. When an ingress is added, removed, or changed, the controller binary rewrites nginx.conf according to a template and signals nginx to reload the configuration. Nginx proxies requests according to its configuration by looking at the host header to determine the correct backend.

Each exposed service requires the following Kubernetes objects:

Deployment Ingress Service
Specifies the number of pods and the containers that run on those pods. Defines the hostname rules and backends used by the controller for the service. Lists the destination pods and ports according to the selector in the Endpoints API.

Here’s what the Kubernetes manifest describing these objects for the xyz application might look like in YAML:

The ingress controller requires its own deployment and service just like any other application in Kubernetes. The controller’s service provisions the load balancer from the cloud provider with optional provider-specific configuration. Additionally, the controller can use a ConfigMap, a Kubernetes object for storing configuration key-value pairs, for global customization and consume per-service annotations, which are key-value metadata attached to Kubernetes objects, specified in each ingress. Examples of these configuration options include the use of PROXY protocol, SSL termination, use of multiple ingress controllers in parallel, and restriction of the load balancer’s source IP range.

Let’s return our attention to the two other methods of exposing external-facing services. Firstly, let’s revisit ServiceType LoadBalancer:

LoadBalancer

From the client’s perspective, the behaviour is the same. A request to xyz.bar.com still reaches xyz-container and foo.bar.com still hits the foo-container. However, note the duplicated infrastructure: An extra load balancer and CNAME record is needed. If there were a thousand unique services, there would have to be an equivalent number of load balancers and DNS records, all needing to be maintained; what happens if a change needs to be made to every load balancer or record, or perhaps only a specific subset of them? Updating DNS records can also prove a hassle with this configuration, as changes will not be instant due to caching as opposed to a nginx configuration reload, which is a near-zero downtime operation. This is a network that simply does not scale and wastes resources if each service does not receive enough requests to justify having its own load balancer.

For completeness, here is the equivalent ServiceType NodePort network diagram:

NodePort

Here, client request behaviour is radically different. To access the xyz-container, a request to node.bar.com:30000 must be made, and for the foo-container, the request is made to node.bar.com:32767. This port-based virtual hosting is extremely unfriendly to the client: port numbers must be tracked and kept up-to-date in case they are changed, and this is additional overhead on the node; while there are thousands of possible ports, the finite number of them is still a limitation to take into account. These port numbers are by default auto-assigned by Kubernetes; it is possible to specify the service’s external port, but the responsibility for avoiding collisions falls on the developer. While it is possible to solve some of these issues with SRV records or service discovery, those solutions add a layer of complexity which negates the advantage of the simplicity of this method.

The Choice

In comparison, ClusterIP with Ingress provides a powerful, transparent, and intelligent way to allow Kubernetes services to be accessed from outside the cluster. It allows services to assert control over how they expose themselves by moving per-service routing rules from cloud provider infrastructure like DNS records and load balancers to Kubernetes object manifests inside the cluster. In most cases, it is the preferred solution to expose an external-facing service with advantages in scalability, accessibility, and infrastructure over NodePort or LoadBalancer. Hopefully you’ve come away from this post with a general overview of the three methods of exposing external-facing services in Kubernetes!

About The Author

Winfield Chen is a high school co-op on the Operations team, recently graduated from Centennial Secondary School. He is attending Simon Fraser University’s School of Computing Science in September.