AWS Networking Overview, Part 2

In this three-part series, we deep dive into the Kubernetes Pod networking options on Amazon, and provide a bit of guidance around the various trade-offs involved in selecting a particular Kubernetes Network technology for your cluster on Amazon. Please see here for Part 1. 

The other part to understanding networking on Kubernetes running on Amazon is the underlying Amazon network technology. AWS started out with a simple flat network, but due to customer demand for segmented networks, and to provide a more full-featured network implementation, it now includes VPC (virtual private cloud).

VPC allows for more traditional network configurations.

VM’s come with the option to set up a set number of elastic network interfaces (ENIs) on your VM (the exact number depends on the instance type), which attach to the VPC network using public or private subnets. VPC also provides fine-grained controls for routing and also packet filtering of traffic between these subnets. The configuration of your VPC can be done through the Amazon console or programmatically using a variety of languages and tools.

With VPC, you can create ENI’s that:

• Directly attach to a public subnet 
• Directly attach to a private subnet for non-public facing services
• Use Network Address Translation to map public IP addresses to specific ENI’s on private subnets

Furthermore, you can:

• Use VPC native tools to control traffic between subnets using configurable packet filters
• Setup dedicated VPN connections between your internal datacenter and the VPC

This combination of features provides the ability to create a full-featured network topology, controlled by Amazon’s VPC interface. Philosophically, Amazon has chosen to allow maximum flexibility in VPC network design, allowing customers to mirror traditional network topologies. The advantages of this system are that it allows network administrators to implement topologies that mimic traditional zone based network segmentation.

Amazon VPC is a mature product, and Amazon has also implemented a number of helpful technologies to improve operations and security inside of a VPC.

• VPC Flow Logs to track accepted traffic, rejected traffic, or all traffic to a subnet, interface, or an entire VPC. These flow logs can be used to track network activity into, out of, and within your AWS network.
• Security Groups, which act as virtual firewalls on your instances. Security groups allow stateful traffic filtering.
• Network ACLs that operate as virtual, horizontally scalable, stateless packet filtering devices at the subnet level.
• VPC Routing Tables that work at the VPC and subnet layer to control traffic routability within your network.

These tools allow for a flexible, secure, in-depth approach to service networking. Routing tables, network ACL’s and Security Groups can all be applied to prevent unwanted network traffic.

However, there are a couple of key points of philosophical divergence between this AWS model and the internal Kubernetes network model are:

• Not all VM’s are routable from one another
• NAT is allowed and even encouraged for public facing services

Of course, it is entirely possible to simplify the AWS VPC model down to map directly to a Kubernetes cluster in which all nodes are routable to one another and NAT is not used for inter-service communication. Amazon has worked with Heptio to create a CNI provider which does exactly that. Creating a CNI plugin that operates by attaching secondary ENI’s in one or more mutually routable subnets to worker nodes in your Kubernetes cluster, and assigning them directly to Pod IP’s

However, there are also cases where the constraints of the VPC model (the number of EIC’s available per node in particular) become an issue that may push you towards using another CNI provider on top of the Amazon VPC.

In Part 3, I explore those options and tradeoffs in more detail.

For further reading on production-ready Kubernetes download our latest whitepaper “Your Guide to a Production Ready Kubernetes Cluster”.