21. Measuring CPU Contention Latency

A special form of CPU scheduling latency called CPU contention latency happens when there are more runnable processes than CPUs. In effect, this separates the fixed portion of CPU scheduling latency from the variable portion. In our final round of tests, we wanted to investigate the effect of CPU contention on latency. In the preceding tests, 10 receiver processes had to contend for time on 4 physical CPUs. Note that a single multicast packet wakes up all 10 receiver process at the same time. Assuming an otherwise idle machine, 4 processes get scheduled onto CPUs immediately while 6 wait.

For this test, we ran only 3 receiver processes on the same 4-CPU machine. The results are shown in the lower line on the plot below. The upper line shows the usual 10 receiver case from above. But note that the vertical axis covers only about 1/7 the latency range of the preceding plots. The range of the vertical scale was reduced to show more detail.

The lower band is much thinner than the upper one since there were fewer receivers than CPUs and hence no CPU contention. The difference in thickness is a good indication of how long the additional receivers in the earlier tests had to wait for a CPU to run them (the CPU contention latency). From inspection of the plot above, we'd estimate the CPU contention latency for the 10 receiver case to be about 1.5 ms.

With the magnified vertical scale of the above plot, the clock drift over the duration of the test is also apparent. This makes it easy to visualize the operation of ntpd in trying to keep the system clocks synchronized. Note that the approximately 7 ms. of clock drift during the test run significantly exceeds the measured latency for almost all of the test. Also note that the clock skew was negative for the entire 20-minute duration of the test.