The SPIMI algorithm is shown in Figure 4.4. The part of the algorithm that
parses documents and turns them into a stream of term–docID pairs, which
we call tokens here, has been omitted. SPIMI-INVERT is called repeatedly on
the token stream until the entire collection has been processed.
Tokens are processed one by one (line 4) during each successive call of
SPIMI-INVERT. When a term occurs for the first time, it is added to the
dictionary (best implemented as a hash), and a new postings list is created
(line 6). The call in line 7 returns this postings list for subsequent occurrences
of the term.
A difference between BSBI and SPIMI is that SPIMI adds a posting directly
to its postings list (line 10). Instead of first collecting all termID–docID
pairs and then sorting them (as we did in BSBI), each postings list is dynamic
(i.e., its size is adjusted as it grows) and it is immediately available to collect
postings. This has two advantages: It is faster because there is no sorting
required, and it saves memory because we keep track of the term a postings list belongs to, so the termIDs of postings need not be stored. As a result, the
blocks that individual calls of SPIMI-INVERT can process are much larger
and the index construction process as a whole is more efficient.
The SPIMI algorithm is shown in Figure 4.4. The part of the algorithm thatparses documents and turns them into a stream of term–docID pairs, whichwe call tokens here, has been omitted. SPIMI-INVERT is called repeatedly onthe token stream until the entire collection has been processed.Tokens are processed one by one (line 4) during each successive call ofSPIMI-INVERT. When a term occurs for the first time, it is added to thedictionary (best implemented as a hash), and a new postings list is created(line 6). The call in line 7 returns this postings list for subsequent occurrencesof the term.A difference between BSBI and SPIMI is that SPIMI adds a posting directlyto its postings list (line 10). Instead of first collecting all termID–docIDpairs and then sorting them (as we did in BSBI), each postings list is dynamic(i.e., its size is adjusted as it grows) and it is immediately available to collectpostings. This has two advantages: It is faster because there is no sortingrequired, and it saves memory because we keep track of the term a postings list belongs to, so the termIDs of postings need not be stored. As a result, theblocks that individual calls of SPIMI-INVERT can process are much largerand the index construction process as a whole is more efficient.
การแปล กรุณารอสักครู่..
