Topics in High-Performance Messaging

Robert A. Van Valzah

Todd L. Montgomery

Eric Bowden

December 2009


Table of Contents
1. Introduction
2. TCP Latency
3. Group Rate Control
4. Ethernet Flow Control
5. Packet Loss Myths
6. Monitoring Messaging Systems
7. UDP Buffering Background
8. UDP Buffer Sizing
9. Multicast Loopback
10. Sending Multicast on Multiple Interfaces
11. TTL=0 to Keep Multicast Local
12. TTL=1 and Cisco CPU Usage
13. Intermittent Network Multicast Loss
14. Multicast Address Assignment
15. Multicast Retransmissions
16. Messaging Latency Budget
17. Sources of Latency
18. Latency from Interrupt Coalescing
19. Latency Measurement Overview
20. Measuring CPU Scheduling Latency
21. Measuring CPU Contention Latency

Abstract

We have been working together in the field of high-performance messaging for many years. We have seen many deployed messaging systems that worked well and many that didn't. The background information needed for successful deployment isn't widely available; most of what we know we had to learn in the school of hard knocks. In hopes of saving others a knock or two, we have tried to collect here background information and commentary on some of the issues involved in successful deployments. This information is organized as a series of topics around which there seems to be confusion or uncertainty. Please contact us through 29West if you have questions or comments.

1. Introduction

In the field of high-performance messaging systems, performance tends to be the dominant factor in making design decisions. In this context, "performance" can indicate high message rates, high payload data transfer rates, low latency, high scalability, high efficiency, or all of the above. Such factors tend to be important in applications like financial market data, satellite, telemetry, and military command & control.

Successful deployment of high-performance messaging systems requires cross-disciplinary knowledge. System-level issues must be considered including network, OS, and host hardware issues.

Copyright 2004 - 2010 29West, Inc.