DOI: 10.1007=s00453-001-0048-0
Algorithmica (2001) 31: 442–457 Algorithmica
© 2001 Springer-Verlag New York Inc.
Stochastic Analysis of Shell Sort
R. T. Smythe1 and J. Wellner2
Abstract. We analyze the Shell Sort algorithm under the usual random permutation model. Using empirical
distribution functions, we recover Louchard’s result that the running time of the 1-stage of .2; 1/-Shell Sort has
a limiting distribution given by the area under the absolute Brownian bridge. The analysis extends to .h; 1/-
Shell Sort where we find a limiting distribution given by the sum of areas under correlated absolute Brownian
bridges.Avariation of .h; 1/-Shell Sort which is slightly more efficient is presented and its asymptotic behavior
analyzed.
Key Words. Empirical distribution functions, Brownian bridge, Sorting algorithm, Random permutation
model, Asymptotic distribution.
1. Introduction. Shell Sort is an algorithm that generalizes the method of sorting by
insertion. It is essentially several stages of insertion sort. The algorithm was proposed
in Shell (1959). The method received considerable attention over the past quarter of a
century after Knuth’s 1973 book popularized it (Knuth, 1973). From a practical point of
view, the interest in Shell Sort stems from the fact that it is a rather practicable method of
in situ sorting with little overhead and can be implemented with ease. From a theoretical
standpoint, the interest is that insertion sort has an average of 2.n2/ running time to sort n
random keys, whereas the appropriate choice of the parameters of the stages of Shell Sort
can bring down the order of magnitude. For instance, by a certain choice of the structure
of the stages, a 2-stage Shell Sort can sort in O.n5=3/ average running time. Ultimately,
an optimized choice of the parameter can come close to the information-theoretic lower
bound of O.n ln n/ average case.
The analysis of Shell Sort has stood as a formidable challenge. Most research on
Shell Sort has gone in the direction of making good choices for the parameters of
the stages to obtain good worst-case behavior (see, for example, the review paper of
Sedgewick (1996)). We propose here to take the research along a different axis and
to analyze the stochastic structure of the algorithm. We rederive the limit result of
Louchard (1986) for .2; 1/-Shell Sort, which he proved by essentially combinatoric
arguments, and we generalize the approach to the analysis of .h; 1/-Shell Sort. The
integrated alsolute value of the Brownian bridge appears in the limiting distributions;
the moments of the distribution of this random variable were found by Shepp (1982),
and Johnson and Killeen (1983) gave an explicit characterization of the distribution
function.
1 Department of Statistics, Oregon State University, Corvallis, OR 97331, USA.
2 Department of Statistics, University of Washington, Seattle, WA 98195, USA.
Received June 6, 2000; revised September 27, 2000. Communicated by K. Kemp and H. Prodinger.
Online publication August 9, 2001.
Stochastic Analysis of Shell Sort 443
Section 2 gives a brief description of the algorithm; readers in search of more detail are
advised to consult a source such as Sedgewick (1988), Sedgewick and Flajolet (1996),
or Mahmoud (2000). In Section 3 the limiting distribution of the running time of .2; 1/-
Shell Sort is derived using order statistics and empirical distribution functions. This is
extended in Section 4 to .h; 1/-Shell Sort. Section 5 introduces a variation of .h; 1/-Shell
Sort and gives the asymptotic distribution of its running time. In Section 6 we make some
brief observations about .h; k; 1/-Shell Sort.
2. The Algorithm. To sort by insertion, one progressively adds keys to an already
sorted file. This is achieved by identifying the rank of the next key by searching the
available sorted list. The search can, of course, be done in many different ways. We
restrict our attention to linear search, since it is a form that integrates easily into Shell Sort.
Shell Sort performs several stages of insertion sort and is well suited to arrays. We
assume the data reside in a host linear array structure of size n. If the chosen Shell Sort
uses k stages, a k-long integer sequence decreasing down to 1 is chosen to achieve a
faster sort than plain insertion as follows. Suppose the sequence is tk ; tk¡1; : : : ; t1 D 1.
In sorting n keys, the first stage sorts keys that are tk positions apart in the list. Thus
tk subarrays of length at most dn=tke each are sorted by plain insertion. In the second
stage the algorithm uses the increment tk¡1 to sort tk¡1 subarrays of keys that are tk¡1
positions apart (each subarray is of length at most dn=tk¡1e), and so on, down to the last
stage, where an increment of 1 is used to insert-sort the whole array. Thus, in the last
stage the algorithm executes plain insertion sort.
As an example, .2; 1/-Shell Sort sorts the array
3 2 6 5 9 8 1 4 7
in two stages. In the first stage the increments of 2 are used—the subarray of odd indexes
is sorted by regular insertion sort, and the subarray of even indexes is sorted by regular
insertion sort. The two interleaved arrangements
sorted odd positions: 1 3 6 7 9
sorted even positions: 2 4 5 8
are then sorted by one run of the regular insertion sort on the whole array. The stage of
Shell Sort that uses the increment tj will be referred to as the tj -stage of the algorithm.
We use the usual notation Z. j / to denote the j th order statistic among Z1; : : : ; Zr .
(Technically, we should use Z. j : r / since Z. j : r / and Z. j : s/ may differ for s 6D r ; however,
when the second subscript is understood, we drop it for simplicity.)
The notion of an inversion in a permutation is at the core of our analysis. Let
.¼1; : : : ; ¼n/ be a permutation of f1; : : : ; ng. One says that the pair .¼i; ¼j / is an inversion
if ¼i > ¼j , for i < j , that is when ¼i and ¼j are out of their natural order.
3. Analysis of .2; 1/-Shell Sort. Our goal in this paper is to discuss the stochastic
behavior of Shell Sort when it operates on an array of n raw data. The usual probability
model is the random permutation model, whereby the ranks of the data are equally likely
444 R. T. Smythe and J. Wellner
to be any of the permutations of f1; : : : ; ng, each occurring with probability 1=n!. This
model is natural and is used as the standard for the analysis of sorting algorithms. The
model covers a wide variety of real-world data models; for instance the entire class of
data drawn from any continuous distribution follows the random permutation probability
model. Henceforth the term random will specifically refer to data from this model. In
this section we consider only .2; 1/-Shell Sort.
The difficulty in the stochastic analysis of Shell Sort lies in the fact that after the first
stage the resulting data are no longer random. Instead, tk sorted subarrays are interleaved.
The second and subsequent stages may not then appeal to the results known for insertion
sort. For example, the second stage of the .2; 1/-Shell’s sort does not sort a random array
of size n. The first stage somewhat orders the array as a whole, and many inversions are
removed (some new ones may appear, though; see for example the positions of 5 and 6
before and after the 2-stage of our example).
Suppose our data are n real numbers from a continuous probability distribution.
Because ordering is preserved by the probability integral transform, we may (and do)
assume that the probability distribution is uniform on .0; 1/.We call the elements in odd
positions X’s and those in the even position Y ’s. Thus, if n is odd, initially our raw array
prior to any sorting is
X1; Y1; X2; Y2; : : : ; Ybn=2c; Xdn=2e;
and if n is even the initial raw array is
X1; Y1; X2; Y2; : : : ; Xn=2; Yn=2:
The 2-stage of the algorithm puts the X’s in order among themselves and the Y ’s in
order among themselves. Let Sn be the number of comparisons that .2; 1/-Shell Sort
makes to sort n random keys, and let Cn be the number of comparisons that insertion
sort makes to sort n random keys. The 2-stage of .2; 1/-Shell Sort makes two runs of
insertion sort on the subarrays X1; : : : ; Xdn=2e and Y1; : : : ; Ybn=2c, thus requiring
Cdn=2e © QC
bn=2c
comparisons, where QC
j
DD
Cj , and for all feasible i and j , Ci is independent of QC
j .
The 1-stage now comes in, requiring additional comparisons to remove the remaining
inversions. When we are about to insert Y. j /, we place it among
fX.1/; : : : ; X. j /g [ fY.1/; : : : ; Y. j¡1/g:
Because the 2-stage has sorted the Y ’s, fY.1/; : : : ; Y. j¡1/g do not have any inversions with
Y. j /. Only fX.1/; : : : ; X. j /g can introduce inversions. It is well known that the so-called
sentinel version of insertion sort makes
C.5n/ D n C I .5n/
comparisons to sort a permutation 5n with I .5n/ inversions. Let Vj be the number of
inversions Y. j / makes with all the elements that precede it, that is
Vj D 1fX.1/>Y. j /g C¢ ¢ ¢C1fX. j />Y. j /g;
Stochastic Analysis of Shell Sort 445
for j D 1; : : : ; bn=2c. A symmetric argument applies to the insertion of X. j /; define Wj
as the number of inversions X. j / makes with all the elements that precede it:
Wj D 1fY.1/>X. j /g C¢ ¢ ¢C1fY. j¡1/>X. j /g;
for j D 1; : : : ; dn=2e. The number of inversions after the 2-stage is thus
In D
dXn=2e
jD1
Vj C
bXn=2c
jD1
Wj :
The 1-stage then requires n C In comparisons
The overall number of comparisons Sn of .2; 1/-Shell Sort is therefore given by the
convolution
Sn D Cdn=2e © QC
(1) bn=2c © n © In:
Lent and Mahmoud (1996) developed Gaussian laws for the entire class of treegrowing
search strategies, which includes linear search, the search method of Shell Sort.
In particular, Cn, the number of comparisons that insertion sort performs to sort n random
keys, is asymptotically normally distributed:
Cn ¡ 14
n2
n3=2
D ¡! N.0; 1
36 /:
We focus on the limiting distribution of In. To avoid working with clumsy floors and
ceilings, we assume n is even.