Gigabit TCP with Trapeze

Some of the press coverage about the experiment is misleading. We would like to emphasize, clarify, and expand on key points touched on in the original Duke press release:

• What we did was to combine several software techniques, some of which are not novel, to produce what we believe is the first demonstration on public record of TCP/IP protocols running faster than a gigabit per second. Our contribution included drivers and FreeBSD operating system kernel enhancements for zero-copy sockets (written by Andrew Gallatin) and the Trapeze/Myrinet firmware (written by Ken Yocum).

• We did not improve the TCP/IP protocol software and we did not build or modify high-speed LAN hardware. These are significant technical achievements that we claim no credit for. The credit for the LAN is due to Myricom, and the credit for gigabit-capable TCP/IP software is due to others, e.g., Van Jacobson, Dave Borman, and many others who have contributed to TCP/IP over the years, as well as to its implementation in FreeBSD. Moreover, our result was made possible by advancing endstation hardware, most notably 64-bit PCI.

• Typical TCP socket implementations running over typical gigabit LANs (e.g., a Gigabit Ethernet using the standard 1500-byte MTU) deliver about half a gigabit per-second. We set up a DEC/Compaq Alpha 21264 (XP1000) workstation with prototype LANai-7 Myrinet cards donated by Myricom, installed our firmware, booted an enhanced FreeBSD kernel, and measured TCP bandwidths above 1.1 gigabits per second -- twice as fast as typical workstations and operating systems connected to current gigabit-per-second LANs.

• It is possible to transmit data with TCP faster than half a gigabit per-second without our software. Certainly there are other LANs on the market capable of TCP bandwidths higher than half a gigabit per second. In fact, our published results show that Gigabit Ethernet products from Alteon (using Jumbo Frames) are capable of end-to-end TCP bandwidths of about 970 Mb/s on the XP1000 even without our enhancements, assuming a driver with support for checksum offloading. Alteon's web site currently reports several experiments in which a host services several simultaneous TCP connections which together approach one gigabit per-second.

• The previous end-to-end TCP bandwidth record listed at netperf.org was 750 Mb/s. However, it is unlikely that the netperf.org site is regularly updated with new results as LAN and workstation hardware continue to advance. It is quite possible that others have approached or even exceeded the gigabit milestone without promoting the result. The previous highest bandwidth recorded on Myricom's web site is 687 Mb/s, set by our group last year with an earlier generation of Myricom's LAN and DEC Alpha workstations.

• The work is one aspect of a larger project funded by the National Science Foundation and equipment grants from Intel Corporation and Myricom Inc.