Of Pets, Cattle, and Cabbage!

Of Pets, Cattle, and Cabbage!


N.B. This is  the text of a position paper I submitted to the 17th International Workshop on High Performance Transaction Systems.

In the past 10 years or so, Ethernet link and switching bandwidth improvement has been improving at a rate greater than expected from Moore’s Law. This year 100Gbps NICs & switches are commonly available, and 400Gbps is expected by late 2018. In the same time frame, Ethernet has added significant new capabilities such as RDMA and loss avoidance. These changes allows system designers to think “out of the box” and have led to current industry efforts like NVMe over Fabrics, Ethernet-attached hard drives, Rack Scale Architectures, FPGA based Smart NICs, Software Defined Infrastructure, remote GPUs, etc. So what would it mean if we just quit using system “buses” like PCI Express and just put everything on Ethernet?  Imagine a data center with servers that are defined not by what is contained in some sheet metal, but rather by the dynamic association of various hardware components. Such an architecture would enable much more efficient use of hardware resources that might otherwise be stranded inside servers, allow rapid reconfiguration in the event of component failure, clarify the distinction between stateful (storage) and stateless (compute) components, and encourage the further commoditization of hardware.

There are, of course, substantial problems raised by this. In the data plane, latency is the greatest limiting factor – CPU is not going to be separated from DRAM by any type of network because that interface is so fine tuned to low latency. On the other hand, low enough latency is achievable for anything deeper in the storage hierarchy. Reduction of latency and conservation of bandwidth both require the locality and topology aware binding of resources and placement of software. But the management issues are much tougher than the data plane problems.

Today’s servers define, de-facto, the behavior of Ethernet end points, and come with management problems, kludges, and tradeoffs that are directly evolved from the original IBM PC. In the last decade, the industry has made great strides in managing servers, treating them now as cattle instead of pets. Yet even cattle have minds of their own, develop strange behaviors, and leave trails of patties to be cleaned up.

Fully dis-aggregated Ethernet attached components would exist in numbers an order or two more in magnitude than servers, so they must be far easier to manage – like vegetables instead of animals – like cabbage instead of cattle. Why cabbage? The root word for cabbage means “head”, and we refer to “heads” of cabbage – perhaps implying that they’re smarter than other vegetables.

About the Author:

Tom Lyon is a computing systems architect, a serial entrepreneur and a kernel hacker. Prior to founding DriveScale, Tom was founder and Chief Scientist of Nuova Systems, a start-up that led a new architectural approach to systems and networking. Nuova was acquired in 2008 by Cisco, whose highly successful UCS servers and Nexus switches are based on Nuova’s technology. He was also founder and CTO of two other technology companies. Netillion, Inc. was an early promoter of memory-over-network technology. At Ipsilon Networks, Tom invented IP Switching. Ipsilon was acquired by Nokia and provided the IP routing technology for many mobile network backbones. As employee #8 at Sun Microsystems, Tom was there from the beginning, where he contributed to the UNIX kernel, created the SunLink product family, and was one of the NFS and SPARC architects. He started his Silicon Valley career at Amdahl Corp., where he was a software architect responsible for creating Amdahl’s UNIX for mainframes technology. Tom holds numerous U.S. patents in system interconnects, memory systems, and storage. He received a B.S. in Electrical Engineering and Computer Science from Princeton University.

Leave A Comment