R is definitely for rabbit holes.
I was reading a time travel trilogy, and the author Rysa Walker inserted short stories 1.5 and 2.5 to bridge the three novels. The short stories presented interesting observations about the paradoxes of time travel. I wonder if she was stuck in a similar way to me in my four-part series on Kubernetes? I got lost in so many interesting rabbit holes as I was researching these blog posts. While I’m discussing Kubernetes, containers and scale-out applications, I was reminded a few times during the research about the advent of commodity multicore CPUs. It struck me in retrospect that the limit to CPU scale-up of maximum clock frequency and the subsequent roll-out of multicore microprocessors was the scale-up vs. scale-out conundrum simply playing out once again.
So, let’s allow ourselves a diversion, call this blog post 2.5, and remember the multicore crisis.
Karl Rupp has an interesting and comprehensive chart combining trends for microprocessors over 42 years.
When the CPU clock (frequency) wall was hit around 2004, the commodity CPU industry pivoted to producing multicore/multithreaded microprocessors leading to a relative explosion of logical cores/threads per microprocessor. Realize that prior to this, CPUs/platforms gained performance consistently generation over generation by increasing the CPU clock rate. There were camps of DIYs that overclocked production CPUs to gain more performance.
Two problems arose with the wide availability of multicore microprocessors. First, legacy applications were slow to evolve (if at all) to take advantage of commodity multiprocessing. Second, popular operating systems provided little in the way of enterprise-level sandboxing of both resources and performance of multiple applications on the same OS platform. The appearance of commodity multicore processors led to unused CPU cycles in the data center which in turn resulted in the rapid emergence of VMware and virtualization as the blueprint for data centers moving forward as virtualized instances neatly allowed people to create “application sandboxes” multicore microprocessors. Where each application had its own operating system. While virtualization solved the unused CPU problem, it solved it in a heavyweight fashion with lots of overhead.
Krup’s graph above motivates our understanding of the need for software-enabled multiprocessing and why platform virtualization became so popular. But this series is about the more recent introduction of Kubernetes.
Kubernetes, again, is an orchestration layer that allows you to manage a pool of platforms as worker nodes executing one or more groups (pods in Kubernetes parlance) of containers. Kubernetes schedules the work according to the capabilities and current loading of the servers being assigned work. Unlike traditional virtualization that allows unmodified applications to run in their own copy of the operating system, Kubernetes requires the application to be containerized – written in terms of container APIs.
Kubernetes and containers share much more of the infrastructure of the operating system amongst containers and provide similar sandboxing capabilities with much less overhead than traditional virtualization. Applications are specifically written for or ported to container APIs. As part of the containerization process, a design approach of microservices is encouraged. Each application is broken down into a cooperating set of microservices, and microservices and Kubernetes are inherently networked. Differing number of cooperating instances/microservices/containers are spun up across a network of worker nodes in smaller increments than would be typical for traditional virtualized infrastructure. The aggregate set of microservices provides the larger customer-facing service/experience.
The design model and the much lighter-weight container framework for applications, in the end, allow you to pack more actual applications (containers) into any given multicore CPU platform as compared to virtualized application deployments.
Because of superior application utilization and support for larger numbers of applications, these microservices and containers can easily number in the thousands vs. the tens or hundreds of traditional (heavyweight) virtualized instances. The container approach of smaller building blocks – on top of the lighter weight container framework managed by Kubernetes – increases the efficiency of CPU utilization across a sea of worker nodes as they can be best fit into available cores. Because Kubernetes is inherently responding to load feedback before assigning work, a heterogeneous set of servers (say through a slow-rolling upgrade to a new generation platform with a larger number of cores and memory per physical instance), the load balancing and work scheduling are robust in the face of a large dynamic pool of worker nodes.
Mark Nelson did a post arguing the multicore crisis (hitting the clock frequency wall and scaling out cores) was not all it was cracked up to be. He observed at 2 or even 4 cores current operating systems and applications would benefit. But he caveats that at 128 cores, we may have a problem. And here we are. As I observed at the start, the creation of multicore CPUs is the scale-out response to hitting the upper limits on scale-up (clock) in commodity microprocessor. In the small, and in the large (think the emergence of Map-Reduce about this same time) scale-out techniques for application development is on a roll. And Kubernetes and containers is the rapidly emerging scale-out approach leaving legacy monolithic application architectures behind.
Perhaps traditional virtualization was only transitional technology until we arrived at containers and Kubernetes?