1. Introduction. Shell Sort is an algorithm that generalizes the method of sorting by
insertion. It is essentially several stages of insertion sort. The algorithm was proposed
in Shell (1959). The method received considerable attention over the past quarter of a
century after Knuth’s 1973 book popularized it (Knuth, 1973). From a practical point of
view, the interest in Shell Sort stems from the fact that it is a rather practicable method of
in situ sorting with little overhead and can be implemented with ease. From a theoretical
standpoint, the interest is that insertion sort has an average of 2.n2/ running time to sort n
random keys, whereas the appropriate choice of the parameters of the stages of Shell Sort
can bring down the order of magnitude. For instance, by a certain choice of the structure
of the stages, a 2-stage Shell Sort can sort in O.n5=3/ average running time. Ultimately,
an optimized choice of the parameter can come close to the information-theoretic lower
bound of O.n ln n/ average case.
The analysis of Shell Sort has stood as a formidable challenge. Most research on
Shell Sort has gone in the direction of making good choices for the parameters of
the stages to obtain good worst-case behavior (see, for example, the review paper of
Sedgewick (1996)). We propose here to take the research along a different axis and
to analyze the stochastic structure of the algorithm. We rederive the limit result of
Louchard (1986) for .2; 1/-Shell Sort, which he proved by essentially combinatoric
arguments, and we generalize the approach to the analysis of .h; 1/-Shell Sort. The
integrated alsolute value of the Brownian bridge appears in the limiting distributions;
the moments of the distribution of this random variable were found by Shepp (1982),
and Johnson and Killeen (1983) gave an explicit characterization of the distribution
function.