Kernel threads in Linux are similar to user processes but they run in kernel mode. This means they have scheduling policies and priorities too. When we changed the scheduling policy of the FWQ benchmark from Linux’s default completely fair scheduler to real-time scheduler with the highest possible priority, we found that VM status update threads were never scheduled due to their lower priorities, thus removing the values between 30-35us in the Figure 3. If applications use a higher priority than kernel-level non-critical threads, it might be possible to achieve a consistent low-level OS clock tick noise profile with task pinning and turning off the scheduler load balancer. However, even with the realtime policy, FWQ benchmark gave the same results; the real-time scheduler is not enough to reduce the noise footprint of the OS.
We also ran scientific codes and popular HPC benchmarks in the same 24 core machine and found that there is a slight (less than 0.5%) performance gain with the lowest OS noise footprint compared to other configurations. We believe that the insignificance of the performance gain is due to relatively small scales; exascale-grade hardware is expected to contain many more CPUs per node and the performance gain would be amplified in that case. Another interesting finding was the elimination of the variability in the running times of applications. Although the average running times did not change significantly, variability almost disappeared.