Efficient Relevance Feedback

for Content-Based Image Retrieval

by Mining User Navigation Patterns
Ja-Hwung Su, Wei-Jyun Huang, Philip S. Yu, Fellow, IEEE, and
Vincent S. Tseng, Member, IEEE
AbstractNowadays, content-based image retrieval (CBIR) is the mainstay of image retrieval systems. To be more profitable,
relevance feedback techniques were incorporated into CBIR such that more precise results can be obtained by taking users
feedbacks into account. However, existing relevance feedback-based CBIR methods usually request a number of iterative feedbacks
to produce refined search results, especially in a large-scale image database. This is impractical and inefficient in real applications. In
this paper, we propose a novel method, Navigation-Pattern-based Relevance Feedback (NPRF), to achieve the high efficiency and
effectiveness of CBIR in coping with the large-scale image data. In terms of efficiency, the iterations of feedback are reduced
substantially by using the navigation patterns discovered from the user query log. In terms of effectiveness, our proposed search
algorithm NPRFSearch makes use of the discovered navigation patterns and three kinds of query refinement strategies, Query Point
Movement (QPM), Query Reweighting (QR), and Query Expansion (QEX), to converge the search space toward the users intention
effectively. By using NPRF method, high quality of image retrieval on RF can be achieved in a small number of feedbacks. The
experimental results reveal that NPRF outperforms other existing methods significantly in terms of precision, coverage, and number of
feedbacks.
Index TermsContent-based image retrieval, relevance feedback, query point movement, query expansion, navigation pattern
mining.

1 INTRODUCTION
M
ULTIMEDIA contents are growing explosively and the
need for multimedia retrieval is occurring more and
more frequently in our daily life. Due to the complexity of
multimedia contents, image understanding is a difficult but
interesting issue in this field. Extracting valuable knowl-
edge from a large-scale multimedia repository, so-called
multimedia mining, has been recently studied by some
researchers. Typically, in the development of an image
requisition system, semantic image retrieval relies heavily
on the related captions, e.g., file-names, categories, anno-
tated keywords, and other manual descriptions [19], [20].
Unfortunately, this kind of textual-based image retrieval
always suffers from two problems: high-priced manual
annotation and inappropriate automated annotation. On one
hand, high-priced manual annotation cost is prohibitive in
coping with a large-scale data set. On the other hand,
inappropriate automated annotation yields the distorted
results for semantic image retrieval.
As a result, a number of powerful image retrieval
algorithms have been proposed to deal with such problems
over the past few years. Content-Based Image Retrieval (CBIR)
is the mainstay of current image retrieval systems. In
general, the purpose of CBIR is to present an image
conceptually, with a set of low-level visual features such as
color, texture, and shape. These conventional approaches for
image retrieval are based on the computation of the
similarity between the users query and images via a query
by example (QBE) system [21]. Despite the power of the
search strategies, it is very difficult to optimize the retrieval
quality of CBIR within only one query process. The hidden
problem is that the extracted visual features are too diverse
to capture the concept of the users query. To solve such
problems, in the QBE system, the users can pick up some
preferred images to refine the image explorations iteratively.
The feedback procedure, called Relevance Feedback (RF),
repeats until the user is satisfied with the retrieval results.
Although a number of RF studies [1], [11], [12], [16] have
been made on interactive CBIR, they still incur some
common problems, namely redundant browsing and explora-
tion convergence. First, in terms of redundant browsing, most
existing RF methods focus on how to earn the users
satisfaction in one query process. That is, existing methods
refine the query again and again by analyzing the specific
relevant images picked up by the users. Especially for the
compound and complex images, the users might go
through a long series of feedbacks to obtain the desired
360 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
. J.-H. Su, W.-J. Huang, and V.S. Tseng are with the Department of
Computer Science and Information Engineering, National Cheng Kung
University, No. 1, Ta-Hsueh Road, Tainan, Taiwan, R.O.C.
E-mail: bb0820@ms22.hinet.net, honki@idb.csie.ncku.edu.tw,
tsengsm@mail.ncku.edu.tw.
. P.S. Yu is with the Department of Computer Science (MC 152), University
of Illinois at Chicago, 851 South Morgan Street, Chicago, IL 60607-7053.
E-mail: psyu@cs.uic.edu.
Manuscript received 17 Feb. 2009; revised 26 Aug. 2009; accepted 7 Nov.
2009; published online 27 July 2010.
Recommended for acceptance by X. Zhou.
For information on obtaining reprints of this article, please send e-mail to:
tkde@computer.org, and reference IEEECS Log Number TKDE-2009-02-0075.
Digital Object Identifier no. 10.1109/TKDE.2010.124.
1041-4347/11/$26.00 2011 IEEE Published by the IEEE Computer Society
images using current RF approaches. In fact, it is not
practical in real applications like online image retrieval in a
large-scale image database. Second, Fig. 1 illustrates the
problem of exploration convergence. In Fig. 1, suppose that
two users query with the same image whose concept
consists of car and sunset. In this example, however,
the aimed concept for Query 1 and Query 2 is car and
sunset, respectively. After a set of feedbacks for Query 1
and Query 2, two different moving paths will be produced
since they will lead to images of aimed concepts, respec-
tively. The involved problem, so-called visual diversity, is
shown in Fig. 2. In this case, if the compound concept to aim
at consists of car, sunset, and sunset and car, it is not
easy for traditional CBIR methods to capture the users
intention. Especially for query point movement methods,
this problem will result in that the features would converge
toward the specific point in the feature space during the
query session. Hence, it is still hard to cover the concepts of
car, sunset, and sunset and car even by performing
the weighted K-Nearest Neighbors (KNNs) search.
To resolve the aforementioned problems, we propose a
novel method named Navigation-Pattern-based Relevance
Feedback (NPRF) to achieve the high retrieval quality of
CBIR with RF by using the discovered navigation patterns.
The expected scenario for effectiveness versus efficiency is
clearly illustrated in Fig. 3. It shows that our intent is to
reach high precision efficiently and effectively. In terms of
efficiency, the navigation patterns mined from the user
query log can be viewed as the shortest paths to the users
interested space. According to the discovered patterns, the
users can obtain a set of relevant images in an online query
refinement process. Thus, the problem of redundant
browsing is successfully solved. In terms of effectiveness,
the proposed navigation-pattern-based search algorithm
(NPRFSearch) merges three query refinement strategies,
including Query Point Movement (QPM), Query Reweighting
(QR), and Query Expansion (QEX), to deal with the problem
of exploration convergence. In short, the discovered
navigation pattern in NPRFSearch can be regarded as an
optimized search path to converge the search space toward
the users intention effectively. As a whole, through NPRF,
the optimal results can be attained in very few feedbacks.
The rest of the paper is organized as follows: A review of
past studies is briefly described in Section 2. In Section 3, we
describe the details of our proposed method for query
refinement and relevance feedback. Empirical evaluations
of the proposed method are expressed in Section 4. Finally,
we conclude the paper in Section 5.
2 RELATED WORK
Relevance feedback [5], [17], [25], in principle, refers to a set
of approaches learning from an assortment of users
browsing behaviors on image retrieval [10]. Some earlier
studies for RF make use of existing machine learning
techniques to achieve semantic image retrieval, including
Statistics, EM, KNN, etc. Although these forerunners were
devoted to formulating the special semantic features for
image retrieval, e.g., Photobook [11], QBIC [1], VisualSEEK
[16], there still have not been perfect descriptions for
semantic features. This is because of the diversity of visual
features, which widely exists in real applications of image
retrieval. Therefore, active query refinement, based on the
analysis of usage logs, attracts researchers attention in this
area of RF.
2.1 Query Reweighting
Some previous work keeps an eye on investigating what
visual features are important for those images (positive
examples) picked up by the users at each feedback (also
called iteration in this paper). The notion behind QR is
that, if the ith feature )
i
exists in positive examples
frequently, the system assigns the higher degree to )
i
. QR-
like approaches were first proposed by Rui et al. [14],
which convert image feature vectors to weighted-term
vectors in early version of Multimedia Analysis and Retrieval
System (MARS). Furthermore, Rui et al. [15] provided
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 361
Fig. 1. Motivating example for the problem of exploration convergence.
Fig. 2. Example of visual diversity.
Fig. 3. The expected scenario for effectiveness versus efficiency.
another interactive approach that allows the user to submit
a coarse initial query to refine her/his need via a set of
relevance feedbacks. In this work, the feature weights are
dynamically updated to connect low-level visual features
and high-level human concepts. NNEW, developed by You
et al. [24], learns the users query from positive and
negative examples by weighting the important features.
For this kind of approach, no matter how the weighted or
generalized distance function is adapted, the diverse visual
features extremely limit the effort of image retrieval. Fig. 4
illustrates this limitation that although the search area is
continuously updated by reweighting the features, some
targets could be lost.
2.2 Query Point Movement
Another solution for enhancing the accuracy of image
retrieval is moving the query point toward the contour of
the users preference in feature space. QPM regards multi-
ple positive examples as a new query point at each
feedback. After several forceful changes of location and
contour, the query point should be close to a convex region
of the users interest. The well-known space-vector formula
proposed by Rocchio [13] is as follows:
Q
i
Q
i1
c
X
ii
,1
1
,
ii
u
X
iii
,1
11
,
iii
. 1
where Q
i
is the vector of the ith query, 1
,
is the vector of the
,th relevant image, 11
,
is the vector of the ,th irrelevant
image, nr is the cardinality of relevant images, and nir is the
cardinality of irrelevant images. One of the QPM ap-
proaches is the modified version of MARS [9]. MARS
performs weighted euclidean distance to compute the
similarity between the query and the targets. Another
well-known study is MindReader [6], which took advantage
of a generalized euclidean distance to look for the targets in
well ellipsoids. However, diverse visual contents shared by
different kinds of images damage the retrieval quality very
much. Also, it is very difficult to derive an adaptive and
perfect measuring function. A specific measuring function
indeed cannot cover all target groups with various visual
contents. Moreover, the modified query point of each
feedback probably moves toward the local optimal centroid.
Thus, global optimal results are not easily touched in QPM-
like work.
2.3 Query EXpansion
Because QR and QPM cannot elevate the quality of RF, QEX
has been another hot technique in the solution space of RF
recently. That is, straightforward search strategies, such as
QR and QPM, cannot completely cover the users interest
spreading in the broad feature space. As a result, diverse
results for the same concept are difficult to obtain. For this
reason, the modified version of MARS [9] groups the
similar relevant points into several clusters, and selects
good representative points from these clusters to construct
the multipoint query. Wu et al. [22] proposed FALCON,
which is designed to handle disjunctive queries within
arbitrary metric spaces. Qcluster, developed by Kim and
Chung [8], intends to handle the disjunctive queries by
employing adaptive classification and cluster merging
methods. As experimented in earlier studies, the effective-
ness of QEX is better than those of QPM and QR.
Nevertheless, there are still some problems unsolved for
QEX. For MARS, inappropriate search regions cannot deal
with complex queries. For FALCON, the relevant query
points are too many to be efficient. Adjusting the disjunctive
queries causes the expensive search cost and the results
cannot escape from the restricted range (clusters) that the
users are able to specify. On the whole, QEX brings out
higher computation cost and more feedbacks in RF.
2.4 Hybrid RF
In addition to past studies already described, another type
of RF approach emphasizes the integration of various
search strategies [2], [3], [4], [7], [18]. However, this kind of
method is instinctive, and very little hybridized work
focuses on the accumulated information (long-term usage
log) coming from various users. Moreover, the greater
effectiveness of the multisystem requires a higher computa-
tion cost, due to multiple processings. One of the hybrid RF
strategies is IRRL. IRRL, proposed by Yin et al. [23],
addresses the important empirical question of how to
precisely capture the users interest at each feedback. In
IRRL, exploiting knowledge from the long-term experience
of users can facilitate the selection of multiple RF techniques
to get the best results. The derived problems from IRRL are:
the selection of optimal RF technique cannot avoid the
overhead of long iterations of feedback. Also, the visual
diversity existing in the global feature space cannot be
resolved with an optimal RF technique alone.
3 PROPOSED APPROACH
As elaborated above, the critical issue of RF can be chiefly
summarized thus: how to achieve effective and efficient
image retrieval. To deal with this issue, we describe how
our proposed approach NPRF integrates the discovered
navigation patterns and three RF techniques to achieve
efficient and effective exploration of images.
3.1 Overview of Navigation-Pattern-Based
Relevance Feedback
The major difference between our proposed approach and
other contemporary approaches is that we approximate an
optimal solution to resolve the problems existing in current
RF, such as redundant browsing and exploration convergence.
To this end, the approximated solution takes advantage of
exploited knowledge (navigation patterns) to assist the
proposed search strategy in efficiently hunting the desired
images. Generally, the task of the proposed approach can be
divided into two major operations, namely offline knowledge
discovery and online image retrieval. As depicted in Fig. 5,
each operational phase contains some critical components
362 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
Fig. 4. Relevance feedback with generalized QR technique [14].
for completing the specific process. For online operation,
once a query image is submitted to this system, the system
first finds the most similar images without considering any
search strategy, and then returns a set of the most similar
images. The first query process is called initial feedback.
Next, the good examples picked up by the user deliver the
valuable information to the image search phase, including
new feature weights, new query point, and the users
intention. Then, by using the navigation patterns, three
search strategies, with respect to QPM, QR, and QEX, are
hybridized to find the desired images. Overall, at each
feedback, the results are presented to the user and the
related browsing information is stored in the log database.
After accumulating long-term users browsing behaviors,
offline operation for knowledge discovery is triggered to
perform navigation pattern mining and pattern indexing.
The framework of the proposed approach is briefly
described as follows:
3.1.1 Online Image Retrieval
. Initial Query Processing Phase: Without considering
the feature weight, this phase extracts the visual
features from the original query image to find the
similar images. Afterward, the good examples (also
called positive examples in this paper) picked up by
the user are further analyzed at the first feedback
(also called iteration 0 in this paper).
. Image Search Phase: Behind the search phase, our
intent is to extend the one search point to multiple
search points by integrating the navigation patterns
and the proposed search algorithm NPRFSearch.
Thus, the diverse inclusion of the users interest can
be successfully implied. In this phase, a new query
point at each feedback is generated by the preceding
positive examples. Then, the /-nearest images to the
new query point can be found by expanding the
weighted query. The search procedure does not stop
unless the user is satisfied with the retrieval results.
3.1.2 Offline Knowledge Discovery
. Knowledge Discovery Phase: Learning from users
behaviors inimage retrieval canbe viewedas one type
of knowledge discovery. Consequently, this phase
primarily concerns the construction of the navigation
model by discovering the implicit navigation patterns
from users browsing behaviors. This navigation
model can provide image search with a good support
to predict optimal image browsing paths.
. Data Storage Phase: The databases in this phase can
be regarded as the knowledge marts of a knowledge
warehouse, which store integrated, time-variant, and
nonvolatile collection of useful data including
images, navigation patterns, log files, and image
features. The knowledge warehouse is very helpful
to improve the quality of image retrieval. Note that
the procedure of constructing rule base from the
image databases can be conducted periodically to
maintain the validity of the proposed approach.
3.2 Offline Knowledge Discovery
In fact, usage mining has been made on how to generate
users browsing patterns to facilitate the web pages
retrieval. Similarly, for web image retrieval, the user has
to submit a query term to the search engine, so-called
textual-based image search. Then the user can obtain a set
of most relevant web images according to the metadata or
the browsing log. However, if the result does not satisfy the
user, the query refinement can be easily incorporated into
the query procedure. This is why CBIR using RF has been
the focus of the researchers in the field of image retrieval.
As far as the usage log of CBIR is concerned, the challenge
mainly lies on: how to generate and utilize the discovered
patterns. In this paper, we develop a navigation-pattern-
based data structure permeated by the query point move-
ment aspect, which has never been proposed by past
studies. Through the special data structure, the users
intention can be caught more quickly and precisely.
In detail, the data structure can be viewed as a hierarchy,
including positive images, query points, and clusters. A
query session contains a set of iterative feedbacks (itera-
tions), which is referred to a navigation path. At each
feedback, the positive examples, which indicate the results
picked up by the user, are used to derive a referred visual
query point by averaging the positive visual features.
Finally, the query sessions, iterations, positive examples,
and visual query points are stored into the original log
database, as shown in Fig. 8. If the original log data are
ready, the next task is to discover navigation patterns from
the original log data. Basically, navigation pattern discovery
consists of two stages: data transformation and navigation
patterns mining. For data transformation, as shown in lines 1-
6 of Fig. 6, the visual query points of the ith iteration are
grouped into i clusters, where i is the maximum session
length and 0
<
i
<
i. Then, the visual query points in each
cluster are converted into a specific symbol, called item #.
The transformed log table is also divided into several
subtables at this time. For navigation patterns mining, as
shown in lines 8-21 of Fig. 6, the frequent itemsets are
mined from the navigation-transaction table.
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 363
Fig. 5. Workflow of NPRF.
3.2.1 Data Transformation
To date, very few significant studies have succeeded in
semantic image retrieval or image recognition because of the
complicated visual contents. To handle the vagueness in
image presentation, data transformation for visual content is
a fundamental and important operation because it can
simplify both the description of visual query points and the
discovery of navigation patterns. In other words, without the
data transformation, we have to consider all positive images
of each query session in the log database. If all positive
images are considered for navigation pattern mining, too
many items make the frequent itemsets (navigation patterns)
hard to find. Also, the mining cost is expensive.
As a result, the aim of data transformation is to generate
Query Point Dictionary (QPD) to reduce the kinds of items
on the transaction list. The scenario of QPD can be
presented in Fig. 7. In Fig. 7, each trail stands for a query
session (browsing path) and each point represents the
visual query point of the iteration in a query session. For
Fig. 7, given a set of iterations, 1T` fQ. itciotioi 0.
itciotioi 1. . . . .. itciotioi ig, where Q contains the set of
starting query images related to different navigation trails,
and iteration i contains the set of visual query points at the
ith iteration. From Q to iteration i, the visual query points
of each iteration are grouped into several clusters, with
respect to the circles encoded as C
11
, C
12
. . . . . and C
nm
in
Fig. 7, by utilizing k-means algorithm and visual similarity
calculations. For visual similarity calculation, the adopted
features are Color Layout, Color Structure, Edge Histo-
gram, Homogeneous Texture, and Region Shape. At last,
each query point of each cluster is assigned the specific
symbol referred to the cluster number. The symbolized
cluster number is employed as the item number in the
navigation-transaction table for the mining procedure. For
example, in Fig. 7, there are 2, 4, 3, 2, and 3 starting query
images in clusters C
11
. C
12
. C
13
. C
14
, and C
15
, respectively.
For C
11
, the related query transactions are fC
11
. C
21
. C
32
.
C
42
g and fC
11
. C
23.
C
32
. C
42
g. That is, C
11
. C
12.
C
13
. C
14
and
C
15
can be viewed as the starting items in these 14 query
transactions.
In this phase, the transformed log table is first generated
by the logged query sessions containing query session id,
iteration number, positive images, and visual query point
number. Once a query point is projected onto the QPD, the
referred item number is stored into the transformed log
table. As shown in Fig. 8, the transformed log table has to be
further partitioned into three tables for various needs in this
paper, including QP table, Navigation-transaction table,
and Partitioned Log table. Navigation-transaction table is
used for navigation patterns mining. QP table and Parti-
tioned Log table are necessary for image search discussed at
length in subsequent sections. In viewing the complete data,
the jointed table can be derived with connecting the joint
attributes of different tables.
3.2.2 Navigation Patterns Mining
This stage focuses on the discovery of relations among the
users browsing behaviors on RF. Basically, the frequent
patterns mined from the user logs are regarded as the
useful browsing paths to optimize the search direction on
RF. In our NPRF approach, the users common interests can
be represented by the discovered frequent patterns (also
called frequent itemsets). Through these navigation pat-
terns, the users intention can be precisely captured in a
shorter query process. In this phase, the Apriori-like
algorithm is performed to exploit navigation patterns using
the transformed data. The task for establishing the naviga-
tion model can be decomposed into two steps:
364 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
Fig. 6. Procedure for offline knowledge discovery.
Fig. 7. The query point dictionary of the proposed approach.
Step 1: Construction of the navigation transaction table.
From Fig. 7, let us select five query sessions as an example
showninTable1. InTable 1, aquerysessioncanbeconsidered
a transaction. In this case, the transaction is composed of a
query item C
1i
j C
1i
2 Q f g and several iteration items
fC
1i
j C
1i
2 iteration#4, where 2
<
i
<
4g. Toexploit valuable
navigation patterns, all query sessions in the transformed log
table are collected as the navigation-transaction table.
From Table 1, we can discover some interesting
phenomena. First, though passing through different itera-
tions, the paths starting with the same query item lead to the
same destination, e.g., Session 001 and Session 002. Second,
the paths starting with different query items lead to the
same destination, e.g., Session 002 and Session 004. Third,
the paths starting with the same query item lead to different
destinations, e.g., Session 003 and Session 004. Even the
special path, Session 005, may be the other important trail
for image hunting. Such evidence implicates the main
aspect we want to demonstrate in this paper.
Step 2: Generation of navigation patterns. This operation
concentrates on mining valuable navigation patterns to
facilitate online image retrieval. As shown in lines 8-21 of
Fig. 6, the frequent itemsets A whose supports support(A)
exceed the presetting minimum support minsup are mined
by Apriori-like algorithm. In fact, the generated frequent
itemsets can be regarded as sequential navigation patterns
directly since temporal continuities have been considered in
creating QPD. For example, as shown in Table 2, the
sequential navigation pattern C
11
! C
32
! C
42
f g derived
from frequent itemset C
11
. C
32
. C
42
f g under the minsup is 2.
3.2.3 Pattern Indexing
In this stage, we describe how to build the navigation
pattern tree with the discovered navigation patterns. As
shown in Fig. 9, the navigation patterns can be regarded as
the branches of the navigation pattern tree. Once the
navigation patterns are generated, the query item C
1m
2 Q
in each navigation pattern is used as a seed (called query
seed) to plant the navigation pattern tree. Therefore, if the
cardinality of the clusters is 7, there are seven navigation
trees generated in this stage. A tree contains a number of
navigation paths, and each node of the paths stands for an
item consisting of several visual query points. A visual
query point indicates a set of positive images. In particular,
to decrease the complexities of both pattern search and
pattern storage, the redundant navigation patterns have to
be pruned further. The observed redundancy between two
patterns can be defined as follows:
Definition 1 (Pattern Redundancy). Consider two navigation
patterns, namely Fitemset1 Cab. . . . . Cij f g and Fitemset
2 Cpg. . . . . Cxy f g. I f Cab Cpg, Cij Cxy, a n d
jFitemset1j

jFitemset2j, Fitemset1 is called redundant

navigation pattern.
After eliminating the redundant patterns, the trimmed
navigation pattern tree reduces the search cost significantly.
Based on the navigation pattern tree, the desired images can
be captured more promptly without repeating the scan of
the whole image database at each feedback, especially for
the large-scale image data.
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 365
TABLE 1
Example of Navigation-Transaction Table
TABLE 2
Example of Navigation Patterns
Fig. 9. Example of navigation pattern trees.
Fig. 8. The entity-relationship data model for partitioning the log data.
3.3 Online Image Search
3.3.1 Basic Idea
As we can recall from previous explanations, the aim of the
search strategy is to attack the weakness of the traditional
approaches, including redundant browsing and exploration
convergence. Indeed, these unsolved problems result in large
limitation in RF. Perhaps, the aged hybrid systems fusing
the results generated by multiple query refinement systems
can look for the better results than individual systems.
Nevertheless, the expensive computation cost makes it
impractical in real applications. Instead, we attempt to
approximate the optimal solution, namely NPRFSearch, to
resolve such problems by using the generated navigation
patterns. For the problem of exploration convergence, our
proposed approach extends the search range from a query
point to a number of relevant navigation paths. As a result,
each iterative search can escape from the local optimal
space and further move toward the global optimal space for
the users interest. For the problem of redundant browsing,
the discovered navigation patterns are adopted as the
shortest paths to derive the superior results in a shorter
feedback process. Additionally, the expensive navigation
cost is saved further, especially for the massive image data.
In general, the NPRFSearch algorithm can be recognized as a
very important part of our proposed iterative solution to
RF, which merges QEX, QP, and QR strategies.
3.3.2 Algorithm NPRFSearch
As alreadydescribedabove, NPRFSearch is proposedto reach
the high precision of image retrieval in a shorter query
process by using the valuable navigation patterns. In this
section, we explain the details of NPRFSearch. As illustrated
inFig. 10, the NPRFSearch algorithmis triggeredby receiving:
1) a set of positive examples G and negative examples N
determined by the user at the preceding feedback, 2) a set of
navigation patterns TR fti
1
. ti
2
. . . . . ti
/
g, where each ti
/
contains a query seed it
/
and several patterns (e.g.,
fC
11
. C
32
. C
42
g referred to Table 2), and 3) an accuracy
threshold thrd. In brief, the iterative search procedure can
be decomposed into several steps as follows:
1. Generate a new query point by averaging the visual-
features of positive examples.
2. Find the matching navigation pattern trees by
determining the nearest query seeds (root).
3. Find the nearest leaf nodes (terminations of a path)
from the matching navigation pattern trees.
4. Find the top : relevant visual query points from the
set of the nearest leaf nodes.
5. Finally, the top k relevant images are returned to
the user.
From the aspect of NPRFSearch, step 1 can be regarded as
QPM and steps 2-5 can be regarded as QEX. For QR, the
feature weights are iteratively updated based on the positive
examples at each feedback. As a whole, the proposed
NPRFSearch takes advantage of QPM, QEX, QR, and
navigation patterns to make RF more efficiently and
effectively. Without navigation patterns, NPRFSearch cannot
reach the high quality of RF. From the viewpoint of
applicability, the goal of our approach is to satisfy each
query efficiently instead of providing personalized functions
for eachuser. Hence, irrelevant queries froma user will not be
a problem. By collecting a large number of query transac-
tions, most queries can be well answered for matching users
interests by NPRFSearch. The details of the NPRFSearch
algorithm are described as follows:
Query point generation. The basic idea of this operationis
to find the images not only with the specific similarity
function. By recursively modifying the query point, the
search direction can move toward the targets gradually.
Assume that a set of images is found by the query point j
old
at the preceding feedback. Next, the visual features of the
positive examples G picked up by the user are first averaged
into a new query point j
new
. As shown in lines 1-3 of Fig. 10,
consider a set of positive examples G fq
1
. q
2
. . . . . q
/
g and d
dimensions of the ith feature 1
i
f)
r
1
. )
r
2
. . . . . )
r
d
g extracted
366 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
Fig. 10. Algorithm for NPRFSearch.
from the rth positive example. Thereupon, the new query
point j
new
implied by G can be defined as
j
new
1
1
. 1
2
. . . . . 1
/

. 2
where 1
<
i
<
/,
1
i
)
1
. )
2
. . . . . )
d

.
and
)
t

P
1r/.)
r
t
21
i
)
r
t
/
.
Meanwhile, j
new
andthepositive examples are storedinto
the log database to enhance the knowledge database. Next,
the negative examples are appended to the accumulated
negative set NIMG. At each feedback, eliminating MING
from the targets can increase the precision of image retrieval
significantly. In addition to generating j
new
and MING, the
weight of each feature has to be calculated to keep searching
the images similar to j
new
. In this paper, the feature weight
for similarity computation is normalized as follows:
Feature reweighting. Consider a set of positive examples
G fq
1
. q
2
. . . . . q
/
g found by the preceding query point
j
old
. Given that a set of o
1
. o
2
. . . . . o
/
f g is referred to
1
1
. 1
2
. . . . . 1
/
f g. The new weight of the ith feature 1
i
is
defined as
n
i

P
/
y1
o
y
o
i
P
/
:1
P
/
y1
o
y
o
:
. 3
where
o
X
/
r1

P
d
,1

)
r
,
)
j
o|d
,

2
q
d
and 1
<
i
<
b.
Query expansion. To keep an eye on the problem of
exploration convergence, the attempt of this stage is to cover
all possible results by the relevant patterns discovered. Then,
by performing a weighted KNN search, QEX-like procedure
first determines the nearest query seed to each of G, called
positive query seed, and the nearest query seed to each of N,
callednegative query seed. Lines 5-8 of Fig. 10 showhowto find
the positive and negative query seed sets. As a result, a set of
positive query seeds is selected to be the start of potential
search paths. Additionally, the slight loss of the information
embeddedin the negative examples is also deliberatedin this
paper. In theory, if the negative query seeds are all dropped
at each feedback, the desired results could be captured more
precisely. However, there exist some query seeds belonging
to both of the positive query seed set and the negative query
seed set at each feedback. Dropping the negative query seeds
would lead to the loss of positive query seeds. That is, these
dropped negative seeds may be the start of good search
paths. To take account of both positive and negative
information simultaneously, every seed has its own token
it
/
.chk. If the seed owns the maximum number of negative
examples or owns no positive example, it will be tokenizedas
a bad manner, i.e., it/.c// 0, as shown in lines 4 and 15 of
Fig. 10. Otherwise, rth.chk is 1 for any good manner. Yet,
considering both positive and negative information elevates
the computation cost. To reduce the computation cost, at
each feedback, the algorithm assigns a bad manner to the
seed that owns maximum number of negative examples if
andonly if the satisfaction rate (jGj,jG[ `j) cannot reach the
presetting threshold thrd, as shown in lines 9-16 of Fig. 10.
Finally, the good manners are the starts of the referred
navigation paths to find the relevant leaf nodes (terminations
of the relevant paths).
For example, consider that seven navigation pattern trees
are seeded by seven query seeds, referred to Figs. 7 and 9.
Assume that thrd is 20 percent. Fig. 11 is an example showing
that 10 resulting images are returned at a feedback. After the
users picking, six are positive examples and four are
negative examples. For each example, the proposed system
then performs the weighted KNN search to determine the
nearest query seed (root) of the pattern tree. Thus, the good
manner set is {Tree 2, Tree 5, Tree 6, Tree 7}. In this case,
Tree 2 owns two positive examples, Tree 5 owns two positive
examples, Tree 6 owns one positive example and two
negative examples, and Tree 7 owns one positive example
and one negative example. The potential bad manner is
{Tree 6} because it owns maximum number of negative
examples (that is, two negative examples). Because the
precision (jGj,jG[ `j 60%) in this case exceeds thrd, the
bad manner does not need to be eliminated. Otherwise, Tree
6 has to be dropped from the good manner set.
Soon after the relevant query seeds (good manners) are
determined, as shown in line 18 of Fig. 10, a set of matching
leaf nodes is found by traversing the navigation pattern
tree. Then the crucial search procedure starts looking for the
desired images, using the new feature weights derived by
(3). In detail, the crucial search procedure can be decom-
posed into two steps.
Step 1: Determination of the relevant visual query
points. As shown in lines 17-22 of Fig. 10, from the
collection of the relevant navigation paths, the leaf node of
each relevant navigation path is used to find the relevant
query points by scanning QP table shown in Fig. 8. In other
words, the last item of each pattern represents the
destination of each search path. Fig. 9 is an appropriate
example to show the aspect. Thus, the users intention can
be repaid quickly and comprehensively through determin-
ing the relevant visual query points contained in the found
destinations. At last, the most similar visual query points
to j
new
are acquired by executing a weighted KNN search,
as shown in line 23 of Fig. 10.
Step 2: Determination of the relevant images. To reduce
the computation cost, top : similar visual query points are
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 367
Fig. 11. Example of determining the relevant query seeds.
determined in step 1. Hence, the search space is narrowed
to the images referred to the top : similar visual query
points by scanning the partitioned log table shown in Fig. 8.
Finally, top / images with the shorter distances to j
new
can
be found by performing a weighted KNN search. Note that
as shown in line 28 of Fig. 10, the negative examples
existing at each feedback are all skipped in this paper.
Observation and discussion. In our proposed method,
the data structure can be regarded as a three-layer hierarchy
referred to Fig. 9, with respect to images, query points, and
patterns (also called items or clusters). At the beginning of
implementing NPRF, we tried to look for the desired images
directly from the positive images in the relevant items (leaf
nodes), but in vain. This search strategy can be basically
regarded as a Breadth-First Search (BFS)-based KNN. The
major purpose of using BFS is to speed up the search without
considering the relevant visual query points in the relevant
items (leaf nodes). However, an irreconcilable conflict exists
between Depth-First-Search (DFS)-based KNN and BFS-
based KNN search strategies. Traditional KNN takes
advantage of BFS-based search strategy to look for top
/-nearest neighbors to the query in the feature space.
Comparatively, we adopt DFS-based search to hierarchically
find the desired images from query point level to image
level. The experimental results show that a vertical search is
better than a horizontal search in this case. This is due to the
fact that the correct results do not spread across the query
points. In other words, a query point represents a topic in the
users mind, containing a set of similar positive images. If
performing BFS KNN, the results will spread across the
query points/topics. Thus, the precision is inferior.
The above viewpoint can be clarified in Fig. 12. Given that
three candidate query points {query Point 1, query Point 2,
and query Point 3} are found in the relevant items, and
1
1
. 1
2
. . . . . 1
14
f g indicates the related image set. Assume that
the number of relevant query points is 1 and the number of
relevant images is 5. If the correct results 1
4
. 1
5
. 1
6
. 1
7
. 1
8
f g are
all contained in the range of query Point 2, there exists a
large gap of results between DFS-based KNN and BFS-based
KNN. For BFS-based KNN, the nearest neighbors to j
new
contain the images 1
1
. 1
2
. 1
3
. 1
4
. 1
5
f g spreading on the ranges
of three query points. Thus, the precision of BFS-based KNN
is 2/5. In contrast, the irrelevant query points, with respect
to query Point 1 and query Point 3, are dropped using DFS-
based KNN. Thereby, the most relevant query Point 2 is
identified, and the actual desired images 1
4
. 1
5
. 1
6
. 1
7
. 1
8
f g
limited in the range of query Point 2 can be found.
Accordingly, the precision of DFS-based KNN is 5/5. In
this paper, DFS-based KNNbrings out the better results than
BFS-based KNN.
4 EMPIRICAL EVALUATIONS
So far, we have presented our proposed approach NPRF in
the preceding section in great detail. In this section, we
evaluate the effectiveness of NPRF.
4.1 Experimental Data
The experimental data came from the collection of the Corel
image database and the web images. We prepared seven
data sets composed of different kinds of categories, as
shown in Table 3. Each category contains 200 images. In our
experimental logs, we initially performed QPM to collect the
log on the queries. Then, the navigation patterns are
obtained by adopting our pattern discovery mechanism.
Incrementally, the knowledge discovered from the naviga-
tion patterns can be enhanced once the new query is
submitted to NPRF. Indeed, it does need time to gather the
usage logs. However, a larger log that needs longer
collection time can help achieve higher retrieval quality.
An alternative way to reduce the whole collection time is to
increase the size of collected logs incrementally such that the
precision will also be enhanced gradually. All the experi-
ments were implemented in C++, running on a personal
computer with Intel Dual Core Xeon 3050 2.13 GHz
processor and 1 G MB RAM.
To analyze the effectiveness of our proposed approach,
two major criteria, namely precision and coverage, are used to
measure the related experimental evaluations. They are
defined as
368 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
Fig. 12. Example of the search space for BFS-based KNN and DFS-
based KNN.
TABLE 3
The Experimental Data Sets
jicci:ioi
jcoiicctj
jictiic.cdj
100%.
Co.cioqc
joc coiicctj
jic|c.oitj
100%.
where correct is the positive image set to the query image at
each feedback, retrieved is the resulting image set exploited
by the proposed approach at each feedback, ac_correct is the
union set of all correct during a query session, and relevant is
the ground truth. The criterion precision delivers the ability
for hunting the desired images in users mind and the
coverage represents the ability for finding the accumulated
positive images in a query session.
4.2 Experimental Results
In practice, the primary intentions behind our experiments
are: 1) evaluations for parameter settings, 2) comparisons
between NPRF and other existing RF approaches in terms of
effectiveness and efficiency, and 3) the promise of the
performance for the scale-up data.
4.2.1 Evaluations for Parameter Settings
Before evaluating NPRF, we have to elicit the appropriate
parameter settings, as shown in Table 4, to set up the related
experiments. Based on data set 3, the evaluation measure of
this experiment is the average precision of iterations 0-6. The
first parameter we concerned is the number of clusters cl.
Fig. 13 reveals that the precision slightly degrades as the
number of clusters increases. However, the impact of varied
settings on cl is not significant. The second parameter we
testedis the number of logs andFig. 14 depicts that, the larger
the number of logs, the higher the average precision and the
higher the retrieval cost. To keep balance between effective-
ness and efficiency, 3,600 is adopted as our default setting for
log size in the experiments. The third parameter tested is the
minimum support minsup for navigation patterns mining.
Fig. 15 shows that 0.01 is the best setting. The proper
explanation is that the smaller the minsup, the more the rules
and the more the noises. In other words, a smaller minsup
results in more noises, and thus, lower precision. The fourth
parameter testedis the most relevant seed:. Fig. 16 illustrates
that a larger number of seeds (query points) could result in
lower precision since more irrelevant images would be
produced. The fifth parameter tested is thrd. In our experi-
mental results, the best setting is 0.1 (the figure is eliminated
due to the space limitation). However, the difference of
resulted precision is within 0.00185 with thrd varied from0 to
1. That is, the impact of thrd is not obvious. The last parameter
we tested is / (number of returned images) with the value set
as 20 and 30. The experimental results show that higher
precision is produced when / 30. The potential reason is
that more returned images may provide more useful
information to support RF. However, few users would like
to do feedback on a large number of results in practice.
Hence, / is set as 20 in our further experiments. Overall, the
most important parameters are minimum support, the
number of logs, and most relevant seeds.
4.2.2 Comparisons between NPRF and
Other Approaches
Evaluation of effectiveness. The first experimental evalua-
tion shows the effectiveness of the proposed approach. In
this experiment, we utilized the data set 7 as the experi-
mental data set. Fig. 17 shows: First, the precision of NPRF
performs better than other existing RF approaches signifi-
cantly whatever the iteration number is. Second, for our
proposed approach, the best improvement (steep slope)
exists between iteration 0 and iteration 2. This is the major
point we want to demonstrate, as depicted in Fig. 3. Third,
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 369
TABLE 4
The Experimental Parameter Settings
Fig. 15. The average precisions of different minsup for data set 3.
Fig. 16. The average precisions of different : for data set 3. Fig. 13. The average precisions of different cl for data set 3.
Fig. 14. The average precisions of different numbers of log-transaction
for data set 3.
the na ve approaches, such as Na ve QR and Na ve QPM,
execute worse than the advanced approaches, such as QR
and QPM, because the advanced approaches skip all the
negative examples occurring at each feedback. The similar
operation in our proposed approach is shown in line 28 of
Fig. 10. Fourth, the hybrid approaches outperform the
individual approaches. Overall, QPM is the best individual
approach and NPRF is the best of all.
Moreover, we also evaluate the strength of our proposed
approach for solving the exploration convergence problem
in terms of coverage. Fig. 18 exhibits that NPRF can find
more diverse images than other approaches. It indicates
that the navigation patterns can effectively help the users
get away from the local optimal results and capture the
desired images in the global search space, as already
described in Section 1.
Evaluation of efficiency. In addition to the effectiveness,
another issue we are interested in is the efficiency of our
proposed approach. Through this experiment, we can
realize whether the redundant browsing problem can be
really avoided or not. Table 5 reveals the comparison of the
efficiency among different approaches. The result shows
that NPRF reaches the specific precision 80 percent only
needing two iterations. That is, according to the high quality
of search strategy, the users can obtain the desired images
quickly without a needless exploration program. In contrast
to the other compared approaches, our proposed approach
can be viewed as the best solution to resolve the redundant
browsing problem.
4.2.3 Scale-Up Evaluation
In most former approaches, an important limitation for
image retrieval is that the explosive growth of images leads
to poor and unstable performance. This leaves us with the
motivation of approximating a good solution to cope with
the largeness of the data. To address this point, we enlarged
the data set to evaluate the performance of our proposed
approach. Obviously, Table 6 illustrates that the precision is
substantially stable even when enlarging the data set. This
further proves that our approach is very robust in the
success of RF for the large-scale image data. In addition to
precision, the execution time of NPRF is shown in Table 7. It
reveals that the computation time increases slightly from 0.9
to 1.5 seconds, with the number of categories increased
from 30 to 50 (i.e., the number of images increased from
6,000 to 10,000). This shows that NPRF is scalable in terms
of the execution time.
4.2.4 Illustrative Examples
Now let us consider some testing examples, as shown in
Figs. 19, 20, and 21. In each example, top 20 images are
returned at three specialized iterations. In these illustrative
examples, the images marked in box are called the correct
set. Fig. 19 shows that even though the query image is a
complex image (jack-o-lantern), NPRF can exactly capture
the concept in users mind at iteration 1. That is, through only
one feedback, the precision can reach 12/20 at iteration 1.
Furthermore, after six iterative feedbacks, all 20 resulting
images canbe includedinthe correct set as the user desires. In
370 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011
Fig. 17. The precisions of different approaches for data set 7.
Fig. 18. The coverage of different approaches for data set 7.
TABLE 5
The Minimum Number of Feedbacks for Different Approaches to
Reach the Specific Precision 80 Percent
TABLE 6
The Precisions for Different Amounts of Data by NPRF
TABLE 7
The Execution Time for Different Amounts of Data by NPRF
this case, fromiterations 0 to 6, the improvement of precision
by NPRF is
Improvement

precision of iteration 6 precision of iteration 0

precision of iteration 0

20
20

2
20
2
20
100% 900%.
As shown in Figs. 20 and 21, other traditional approaches
cannot bring out the same effectiveness. The results reveal
that our proposed approach outperforms other well-known
approaches in terms of effectiveness and efficiency.
5 CONCLUSION
To deal with the long iteration problem of CBIR with RF,
we have presented a new approach named NPRF by
integrating the navigation pattern mining and a naviga-
tion-pattern-based search approach named NPSearch.
In summary, the main feature of NPRF is to efficiently
optimize the retrieval quality of interactive CBIR. On one
hand, the navigation patterns derived from the users long-
term browsing behaviors are used as a good support for
minimizingthe number of user feedbacks. Onthe other hand,
the proposed algorithm NPRFSearch performs the naviga-
tion-pattern-based search to match the users intention by
merging three query refinement strategies. As a result,
traditional problems such as visual diversity and exploration
convergence are solved. For navigation-pattern-based
search, the hierarchical BFS-based KNN is employed to
narrowthe gap between visual features and human concepts
effectively. Inaddition, the involvedmethods for special data
partition and pattern pruning also speed up the image
exploration. The experimental results reveal that the pro-
posed approach NPRF is very effective in terms of precision
andcoverage. Withina veryshort termof relevance feedback,
the navigation patterns can assist the users in obtaining the
global optimal results. Moreover, the new search algorithm
NPRFSearch can bring out more accurate results than other
well-known approaches.
In the future, there are some remaining issues to
investigate. First, in view of very large data sets, we will
scale our proposed method by utilizing parallel and dis-
tributed computing techniques. Second, we will integrate
users profile into NPRF to further increase the retrieval
quality. Third, we will apply the NPRF approach to more
kinds of applications on multimedia retrieval or multimedia
recommendation.
ACKNOWLEDGMENTS
This research was supported by the National Science
Council, Taiwan, R.O.C., under grant no. NSC 98-2631-H-
006-001, and by the Ministry of Economic Affairs, Taiwan,
R.O.C., under grant no. 97-EC-17-A-02-S1-024.
REFERENCES
[1] M.D. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang, B.
Dom, M. Gorkani, J. Hafner, D. Lee, D. Steele, and P. Yanker,
Query by Image and Video Content: The QBIC System,
Computer, vol. 28, no. 9, pp. 23-32, Sept. 1995.
[2] R. Fagin, Combining Fuzzy Information from Multiple Systems,
Proc. Symp. Principles of Database Systems (PODS), pp. 216-226, June
1996.
[3] R. Fagin, Fuzzy Queries in Multimedia Database Systems, Proc.
Symp. Principles of Database Systems (PODS), pp. 1-10, June 1998.
[4] J. French and X-Y. Jin, An Empirical Investigation of the
Scalability of a Multiple Viewpoint CBIR System, Proc. Intl Conf.
Image and Video Retrieval (CIVR), pp. 252-260, July 2004.
SU ET AL.: EFFICIENT RELEVANCE FEEDBACK FOR CONTENT-BASED IMAGE RETRIEVAL BY MINING USER NAVIGATION PATTERNS 371
Fig. 19. The resulting example for NPRF.
Fig. 21. The resulting example for QPM.
Fig. 20. The resulting example for QR.
[5] D. Harman, Relevance Feedback Revisited, Proc. 15th Ann. Intl
ACM SIGIR Conf. Research and Development in Information Retrieval,
pp. 1-10, 1992.
[6] Y. Ishikawa, R. Subramanya, and C. Faloutsos, MindReader:
Querying Databases through Multiple Examples, Proc. 24th Intl
Conf. Very Large Data Bases (VLDB), pp. 218-227, 1998.
[7] X. Jin and J.C. French, Improving Image Retrieval Effectiveness
via Multiple Queries, Multimedia Tools and Applications, vol. 26,
pp. 221-245, June 2005.
[8] D.H. Kim and C.W. Chung, Qcluster: Relevance Feedback Using
Adaptive Clustering for Content-Based Image Retrieval, Proc.
ACM SIGMOD, pp. 599-610, 2003.
[9] K. Porkaew, K. Chakrabarti, and S. Mehrotra, Query Refinement
for Multimedia Similarity Retrieval in MARS, Proc. ACM Intl
Multimedia Conf. (ACMMM), pp. 235-238, 1999.
[10] J. Liu, Z. Li, M. Li, H. Lu, and S. Ma, Human Behaviour
Consistent Relevance Feedback Model for Image Retrieval, Proc.
15th Intl Conf. Multimedia, pp. 269-272, Sept. 2007.
[11] A. Pentalnd, R.W. Picard, and S. Sclaroff, Photobook: Content-
Based Manipulation of Image Databases, Intl J. Computer Vision
(IJCV), vol. 18, no. 3, pp. 233-254, June 1996.
[12] T. Qin, X.D. Zhang, T.Y. Liu, D.S. Wang, W.Y. Ma, and H.J. Zhang,
An Active Feedback Framework for Image Retrieval, Pattern
Recognition Letters, vol. 29, pp. 637-646, Apr. 2008.
[13] J.J. Rocchio, Relevance Feedback in Information Retrieval, The
SMART Retrieval SystemExperiments in Automatic Document
Processing, pp. 313-323, Prentice Hall, 1971.
[14] Y. Rui, T. Huang, and S. Mehrotra, Content-Based Image
Retrieval with Relevance Feedback in MARS, Proc. IEEE Intl
Conf. Image Processing, pp. 815-818, Oct. 1997.
[15] Y. Rui, T. Huang, M. Ortega, and S. Mehrotra, Relevance
Feedback: A Power Tool for Interactive Content-Based Image
Retrieval, IEEE Trans. Circuits and Systems for Video Technology,
vol. 8, no. 5, pp. 644-655, Sept. 1998.
[16] J.R. Smith and S.F. Chang, VisualSEEK: A Fully Automated
Content-Based Image Query System, Proc. ACM Multimedia Conf.,
Nov. 1996.
[17] G. Salton and C. Buckley, Improving Retrieval Performance by
Relevance Feedback, J. Am. Soc. Information Science, vol. 41, no. 4,
pp. 288-297, 1990.
[18] H.T. Shen, S. Jiang, K.L. Tan, Z. Huang, and X. Zhou, Speed up
Interactive Image Retrieval, VLDB J., vol. 18, no. 1, pp. 329-343,
Jan. 2009.
[19] V.S. Tseng, J.H. Su, J.H. Huang, and C.J. Chen, Integrated
Mining of Visual Features, Speech Features and Frequent
Patterns for Semantic Video Annotation, IEEE Trans. Multimedia,
vol. 10, no. 2, pp. 260-267, Feb. 2008.
[20] V.S. Tseng, J.H. Su, B.W. Wang, and Y.M. Lin, Web Image
Annotation by Fusing Visual Features and Textual Information,
Proc. 22nd ACM Symp. Applied Computing, Mar. 2007.
[21] K. Vu, K.A. Hua, and N. Jiang, Improving Image Retrieval
Effectiveness in Query-by-Example Environment, Proc. 2003
ACM Symp. Applied Computing, pp. 774-781, 2003.
[22] L. Wu, C. Faloutsos, K. Sycara, and T.R. Payne, FALCON:
Feedback Adaptive Loop for Content-Based Retrieval, Proc. 26th
Intl Conf. Very Large Data Bases (VLDB), pp. 297-306, 2000.
[23] P.Y. Yin, B. Bhanu, K.C. Chang, and A. Dong, Integrating
Relevance Feedback Techniques for Image Retrieval Using
Reinforcement Learning, IEEE Trans. Pattern Analysis and Machine
Intelligence, vol. 27, no. 10, pp. 1536-1551, Oct. 2005.
[24] H. You, E. Chang, and B. Li, NNEW: Nearest Neighbor
Expansion by Weighting in Image Database Retrieval, Proc. IEEE
Intl Conf. Multimedia and Expo, pp. 245-248, Aug. 2001.
[25] X.S. Zhou and T.S. Huang, Relevance Feedback for Image
Retrieval: A Comprehensive Review, Multimedia Systems, vol. 8,
no. 6, pp. 536-544, Apr. 2003.
Ja-Hwung Su received the BS and MS degrees
from the Department of Information Manage-
ment at I-Shou University in 2000 and 2002,
respectively. He is currently working toward the
PhD degree in the Department of Computer
Science and Information Engineering at National
Cheng Kung University. His research interests
include data mining and data warehousing.
Wei-Jyun Huang received the BS and MS
degrees from the Department of Computer
Science and Information Engineering at National
Chengchi University and National Cheng Kung
University, respectively. Her research interest
includes data mining.
Philip S. Yu received the BS degree in electrical
engineering from National Taiwan University, the
MS and PhD degrees in electrical engineering
from Stanford University, and the MBA degree
from New York University. He is a professor in
the Department of Computer Science at the
University of Illinois at Chicago and also holds
the Wexler chair of information technology. He
spent most of his career at IBM Thomas J.
Watson Research Center and was the manager
of the Software Tools and Techniques Group. His research interests
include data mining, database systems, and privacy. He has published
more than 560 papers in refereed journals and conferences. He holds or
has applied for more than 300 US patents. He is an associate editor of
theACM Transactions on the Internet Technology and ACM Transac-
tions on Knowledge Discovery from Data. He is on the steering
committee of the IEEE Conference on Data Mining and was a member
of the IEEE Data Engineering steering committee. He was the editor-in-
chief of the IEEE Transactions on Knowledge and Data Engineering
(2001-2004). He had received several IBM honors including two IBM
Outstanding Innovation Awards, an Outstanding Technical Achievement
Award, two Research Division Awards, and the 94th plateau of Invention
Achievement Awards. He was an IBM master inventor. He received a
Research Contributions Award from the IEEE International Conference
on Data Mining in 2003 and also an IEEE Region 1 Award for promoting
and perpetuating numerous new electrical engineering concepts in
1999. He is a fellow of the ACM and the IEEE.
Vincent S. Tseng received the PhD degree
from National Chiao Tung University, Taiwan,
R.O.C., in 1997, with major in computer science.
He is currently a professor in the Department of
Computer Science and Information Engineering
at National Cheng Kung University (NCKU),
Taiwan. Before this, he was a postdoctoral
research fellow in the Computer Science Divi-
sion at the University of California at Berkeley,
during January 1998 and July 1999. He has
acted as the director of the Institute of Medical Informatics of NCKU
since August 2008. During February 2004 and July 2007, he had also
served as the director of the Informatics Center at National Cheng Kung
University Hospital, Taiwan. He is on the editorial board of the
International Journal of Data Mining and Bioinformatics and served as
a board member for various societies like Taiwan Association for
Artificial Intelligence, Taiwan HL-7 Society, and Taiwan Bioinformatics
Society. His research interests include data mining, biomedical
informatics, mobile web technology, and multimedia databases. He
has published more than 170 research papers in referred journals and
international conferences. He has also held (or filed) more than
15 patents in the US and R.O.C. He has also served as the chair or
program committee member for various premier conferences related to
data mining and biomedical informatics, including the 2009 IEEE
International Conference on Data Mining, 2009 SIAM International
Conference on Data Mining, the 2009 IEEE International Conference on
Mobile Data Management, 2009 ACM CIKM, PAKDD 2009 (PC member
and tutorial chair), PAKDD 2010, DASFAA 2010, the IEEE BIBM 2008
and 2009, the IEEE CBMS 2006-2009, etc. He is a member of the IEEE,
the ACM, and an honorary member of Phi Tau Phi Society.
> For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.
372 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 23, NO. 3, MARCH 2011