2. Focus More on Measuring Application Message Latency and Less on Measuring Data Rates

We know of no better early warning indicator than application message latency. It's the "canary in the coal mine." Increases in application message latency will often precede more significant problems in a system. Application message latency may be caused by problems in the network, the operating system, the messaging system, or within the application itself.

If you only measure one thing, measure the latency for each message passing through your application. Compare the latency message-by-message. Compare it day-by-day. Use it for forensic analysis when something goes wrong.

Once you're measuring application message latency, you can work to improve it. In contrast, market data arrival rates are generally outside your control. It seems better to measure things that you can change.

Data rate measurements may be useful, but we've found them to be less useful than application message latency measurements. Rate measurements suffer from a "hidden average" effect while message latency measurements do not.

Rate measurements are often expressed in units like messages per second or bits per second with no apparent average. But rate measurements must always be made (and averaged) over an interval that is often unspecified. Even when specified, the measurement interval is often far too long to provide data that's useful in diagnosing latency problems.

For example, network traffic rates are often given in terms of megabits per second yet measured only once every 30 seconds. This averages the data rate over a 30-second interval. Even when a 100 mbps network shows only 20% utilization, it still may have been 100% busy for over 5 seconds if the measurement interval was 30 seconds.

If data rate measurements are to be used in diagnosing latency problems, then we recommend making measurements at intervals that are a fraction of the latency boundary for the application. See Section 4 below for details.

It's very important to make latency measurements in production applications running under real market conditions. Latency should also be measured end-to-end. How much time elapsed between the time a market data message was produced (ideally taken from the exchange time stamp) till the time of final consumption by your application? It's good to make intermediate latency measurements along the way, but don't stop till you hit the end of the chain of events triggered by the arrival of a market data message.

In summary, it may seem counter-intuitive to focus on measuring latency instead of rates when the goal is dealing with increasing rates. However, the actual measurement of consistently low latencies is the only certain evidence we know of that demonstrates effective handling of current market data rates. Coming sections will suggest ways that preparedness for future increases in market data rates can also be demonstrated.