1. INTRODUCTION
With the prevalence of the Internet and digital cameras,
effective and efficient image retrieval techniques have be-
come an important research direction in both commercial
and academic circles. There are mainly two basic problems
in image retrieval. The first one is query formulation, that
is how to interpret an implicit query in a user’s mind such as
Copyright is held by the International World Wide Web Conference Committee
(IW3C2). Distribution of these papers is limited to classroom use,
and personal use by others.
WWW 2010, April 26–30, 2010, Raleigh, North Carolina, USA.
ACM 978-1-60558-799-8/10/04.
Figure 1: An implicit query in a user’s mind: i.e. “I
want to find a scene in which a couple are standing
together by the sea at sunset.”
Bing Search:
MindFinder:
Figure 2: Top row: the top 3 search results of Bing
Image Search using query “sunset sea couple mountain”
. Bottom row: the top 3 search results of
MindFinder.
“I want to find a scene in which a couple is standing together
by the sea at sunset”, as shown in Fig. 1, into an explicit
query expressed by some features the computer can easily
process. The second one is query matching, that is how to
find the images that best fit for the explicit query. Since
the solution to the second problem highly depends on the
first problem, query formulation should be given primary
importance in image retrieval.
Currently, based on the types of query formulation meth-
ods, text-based and content-based image retrieval frame-
works become very popular in commercial and academic
circles. Although both these two kinds of frameworks have
been widely studied and applied in commercial and academic
systems, their query formulation methods are far from satis-
factory for a user to express his/her boundless imagination.
For example, a user may want to find a scene in which a
couple is watching sunset by the sea, which is simply illu-
minated in Fig. 1. It is really not easy to search images
similar to such a complex scene. The top results using key-
words “sunset sea couple mountain” as the query in Bing
WWW 2010 • Demo April 26-30 • Raleigh • NC • USA
1309
Image Search1 are shown in Fig. 2. It is quite clear that
the results are not satisfactory. For query by example, it is
also difficult to interpret the user’s mind by finding a query
image for retrieval. While the ongoing work about specific
image retrieval [1] seems far from practical applications.
In order to better formulate a user’s implicit query in
mind, some interactive techniques have been developed, most
of which can be classified into two categories, i.e. search
result-based interactive methods and query-based interactive
methods. Search result-based interactive methods try to
catch users’ intentions by interactively refining the search
results guided by users’ interactions. Relevance feedback [2]
is a typical approach in this category. Query-based inter-
active approaches become more and more popular in recent
years, which try to enable more user interactions by provid-
ing certain attributes that could be specified by users. For
example, Xcavator2 enables users to draw points or lines
on the query image, and then use them to emphasize key
color features and their spatial relationships during search.
Color-structured image search [3] enables users to draw a
few color strokes to indicate the intent to improve search
quality. SkyFinder [4] defines several attributes for sky im-
age retrieval, which could be specified by users.
In spite of the success of existing interactive image search
techniques, most of them are one-side interactive search and
only consider how to leverage users’ effort to catch their
intentions, rather than help users to express their queries
by leveraging the image database. Furthermore, most of
them only use one type of interaction. For example, rel-
evance feedback approaches only use interactive indication
from users to tell whether those results are relevant or not.
Xcavator and color-sturcture image search only involve vi-
sual content features. SkyFinder is particularly designed for
sky image retrieval. Recently, Chen et al. [5] develop an
image montage system, i.e. Sketch2Photo, to stitch several
images together in agreement with the sketch and tags pro-
vided by users. In spite of the leverage of both sketching
and tagging of the queries, users need to draw the implicit
query in mind onto the query panel totally in one time and
there is no user interaction at all. Moreover, the purpose of
Sketch2Photo is to stitch images representing different ob-
jects into the resulting image, rather than to find images in
the database to meet what in the user’s mind.
In this work, we develop the MindFinder system, which
is a bilateral interactive image search engine by interactive
sketching and tagging. Different from existing interactive
image search engines, most of which only provides querybased
or search result-based interaction, MindFinder enables
a bilateral query$search result interactive search, by consid-
ering the image database as a huge repository to help users
express their intentions. Moreover, MindFinder also enables
users to tag during the interactive search, which makes it
possible to bridge the semantic gap. Multiple actions are en-
abled for users to flexibly design their queries in a bilateral
interactive manner by leveraging the whole image database,
including tagging, refinding query by dragging and dropping
objects from search results, as well as editing objects. Af-
ter each action, the search results will be updated in real
time to provide users up-to-date materials to further formu-
late query. Besides the contributions in the query formu-
1http://www.bing.com/images
2http://www.xcavator.net
lation stage, in order to support the real time interactions
between the system and users, a novel object-based indexing
and retrieval algorithm is also developed for query match-
ing. Therefore, MindFinder not only trys to enable users to
present on the query panel whatever they imagine in their
mind, but also returns to users the most similar images to
the picture in users’ mind. Fig. 2 shows the top 3 images
retrieved by MindFinder according to the query in a user’s
mind shown in Fig. 1. In this technical demonstration, users
can try their own searching on the MindFinder system.