N.B. This is the text of a position paper I submitted to the 17th International Workshop on High Performance Transaction Systems.
In the past 10 years or so, Ethernet link and switching bandwidth improvement has been improving at a rate greater than expected from Moore’s Law. This year 100Gbps NICs & switches are commonly available, and 400Gbps is expected by late 2018. In the same time frame, Ethernet has added significant new capabilities such as RDMA and loss avoidance. These changes allows system designers to think “out of the box” and have led to current industry efforts like NVMe over Fabrics, Ethernet-attached hard drives, Rack Scale Architectures, FPGA based Smart NICs, Software Defined Infrastructure, remote GPUs, etc. So what would it mean if we just quit using system “buses” like PCI Express and just put everything on Ethernet? Imagine a data center with servers that are defined not by what is contained in some sheet metal, but rather by the dynamic association of various hardware components. Such an architecture would enable much more efficient use of hardware resources that might otherwise be stranded inside servers, allow rapid reconfiguration in the event of component failure, clarify the distinction between stateful (storage) and stateless (compute) components, and encourage the further commoditization of hardware.
There are, of course, substantial problems raised by this. In the data plane, latency is the greatest limiting factor – CPU is not going to be separated from DRAM by any type of network because that interface is so fine tuned to low latency. On the other hand, low enough latency is achievable for anything deeper in the storage hierarchy. Reduction of latency and conservation of bandwidth both require the locality and topology aware binding of resources and placement of software. But the management issues are much tougher than the data plane problems.
Today’s servers define, de-facto, the behavior of Ethernet end points, and come with management problems, kludges, and tradeoffs that are directly evolved from the original IBM PC. In the last decade, the industry has made great strides in managing servers, treating them now as cattle instead of pets. Yet even cattle have minds of their own, develop strange behaviors, and leave trails of patties to be cleaned up.
Fully dis-aggregated Ethernet attached components would exist in numbers an order or two more in magnitude than servers, so they must be far easier to manage – like vegetables instead of animals – like cabbage instead of cattle. Why cabbage? The root word for cabbage means “head”, and we refer to “heads” of cabbage – perhaps implying that they’re smarter than other vegetables.