DMA engine vs. CPU for Memory Bandwidth

When using a network interface such as Myrinet that is capable of delivering a true Gigabit per-second, its critical to run on an architecture that has adequate memory bandwidth to support both the network interface and the CPU at better than the rate required to handle network traffic (128MB/sec for a 32-bit Myrinet card). Unfortunately, many current platforms fall far short of this goal. The DEC 'pyxis' (or 21174) core-logic-chipset found in the Miata platform (Digital Personal Workstation "a" & "au" series, and Digital Semiconductor's PC164LX series of machines) does the best job of any platform we've been able to test so far.

On May 27th, we received a field-test of the new version of the Miata (known as the MiataGL). This version is based on the new version of the Pyxis chipset. The MiataGL we have tested appears to correct the the flaws in the previous version. We have experianced no correctness problems, and we have measured host to PCI DMA transfers in excess of 100 MB/sec. We whole heartedly recommend the MiataGL for use with Myrinet.

Please note that the initial revision of the Miata had serious correctness flaws in its PCI bus implementation which made it impossible to use for high performance networking with Myrinet. Digital has corrected these problems, but we DO NOT RECOMMEND older Miata platforms for use with Myrinet.

Below are a series of graphs that were generated by running a memory copy benchmark (Lmbench's bw_mem_cp) concurrently with a PCI bus benchmark (Myricom's hswap) that repeatedly runs a Myrinet board's DMA engine to read or write 'DMA Transfer size' bytes to or from host memory.

Please note when reading these graphs that the axis differ (so, in fact the AS500 is much worse than it appears at first glance, and the Personal Workstation is much better). These are not presentation quality graphs and this benchmark may not be statistically valid. However, we've found it to be a good predictor of the maximum throughput of processed data through our gigabit network. Also note that the miata graphs are from 1st generation miatas, not the current revision. Current revision Miatas are certain to do much better.