A Concept-Based Image Retrieval with Concept Pattern Predicates

J. D. Yang
In this paper, we propose a concept-based image retrieval performed by concept pattern
predicates which can capture the pattern of concepts buried in iconic images (or simply
images)[1][2]. Concepts may be viewed as an aggregation of constituent concepts or primitive
objects in an image arranged according to specific patterns. Primitive objects are objects labeled
by indexing. For example, in Figure 1, the composite pattern of a concept, ‘office room’ may be
described by its configuration involving a ‘study room’ and a ‘reception set, each of which is in
turn a concept. ‘Table’, ‘sofa’, ‘bookshelf’, etc. are primitive objects labeled manually or
automatically.

computer
StudyRoom
DeskChair

ReceptionSet sofa
table
sofa

OfficeRoom

StudyRoom

ReceptionSet
StudyRoom bookshelf
DeskChair

ReceptionSet
sofa table sofa

Figure 1 Two possible patterns of an office room

A query may be given as a concept such as ‘office room’ or ‘study room’ <south> ‘reception
set’. An image is therefore an answer if its configuration satisfies the pre-defined patterns of the
query.

1. Pattern predicates
Definition 1 Let D={east, west, south, north, southeast, …, northeast} and let IDB be a set of
images, p be an image in IDB and Op be a set of primitive objects in p. Then p is indexed by a
set of triples p.T of the following form.
p.T ={ t | t = <o1, r12, o2> where r12 ∈ D and o1, o2 ∈ Op }.
Definition 2 Let c be a concept. Then the components and configuration of c is referenced by
c.comp and c.config respectively.
Definition 3 Given a set of concepts Cc, a concept pattern predicate CP is defined as follows.
1. CP : Cc × IDB → {true, false} is a concept pattern predicate.
2. If CP1 , CP2 , …, CPn are concept pattern predicates, then

1
 CP(C, P)=CP1 (C1, P)<r12 >CP2 (C2 , P)<r23> CP3 ⋅ ⋅ ⋅ CPn-1 (Cn-1, P ) <r(n-1) n> CPn (Cn, P)
is a concept pattern predicate where r12, … r(n-1) n ∈D’= D ∪ { Above, Below, Around,
Leftside, RightSide}..
3. Nothing else is a concept pattern predicate.
Definition 4 A special concept pattern predicate called index pattern IX : Cc× IDB → {true,
false} is defined as follows.
IX(c, p) is true if c.config ⊆ p.T.
Definition 5 Let CP and CP1 are concept pattern predicates. Then a concept pattern rule is given
by
CP(C, P) ← CP1 (C1, P) :[action(C, C1, P)]
where action is a procedure to perform semantic actions associated with CP(C,P).
Definition 6 An image information Retrieval system IIR is defined as follows.
IIR = < IDB, IKB, IE, Q >
where IDB is a set of images, IKB is a set of concept pattern rules, IE is an inference engine
processing a query and Q is a set of queries. Queries are given as concept pattern predicates.

Definition 7 An answer for a query Q, || Q || is defined as follows.

|| Q || = { p | IDB ∪ IKB |−IE Q(*, p)}

Example 1

IKB: OfficeRoom(or,P) ← Study Room(sr,P) south ReceptionSet(rs, P)

| Study Room(sr, P) east ReceptionSet(rs, P)
:[ dir(sr, rs, P) = east; or.comp={sr, rs}; or.config = {<sr, east, rs>}]
StudyRoom(sr, P) ← DeskChair(dc, P) Above computer

DeskChair(dc, P) Above bookshelf

:[ dir(dc, bookshelf, P) = Above.value; dc.comp={dc, bookshelf};

dc.config = {<dc, Above.value, bookshelf>};]

ReceptionSet (rs, P)← IX(rs , P)

:[rs.comp={table, sofa}; rs.config={<table, Around.value, sofa>}; rs.config ⊆ P.T]
DeskChair (dc, P)← IX(dc, P)
:[ dc.comp= {desk, chair};dc.config= {<desk south chair>}; dc.config ⊆ P.T]

In the above concept pattern rules, the corresponding semantic actions are coded in :[]. They are
performed whenever the rule is executed. Dir(sr, rs, P) is a function to determine a new direction
between ‘StudyRoom’ and ‘ReceptionSet’ by using sr and rs together with P. A similar
algorithm to implement the function can be found in [4]. Figure 2 shows an image p1, followed
by Figure 3 which depicts a query processing to include it in an answer set.

2
 or
sr b
rs

d
t s
c
dc

OR: OfficeRoom, SR: StudyRoom, DC:DeskChair, RS:ReceptionSet,

Figure 2 An answer image p1

OR(or, p1)
:[dir(sr, rs, p1 ) = east ;
or. comp = {sr, rs}
or. config = {<sr, east, rs>}]

SR(sr, p1) east RS(rs, p1)

:[dir(dc, b, p1) = north; :[rs .comp ={t,s};
sr.comp = {dc, b}; rs .config = {t2}]
sr.config= {<dc, north,b>};]

DC(dc, p1) north b t Around sofas

:[dc.comp= {d,c};
dc.config= {t1} ]

east s
<d south c> Around.value=east
t2 = <t, east, s>
t1 = <d, south, c>
IX(dc, p1), t1 ⊆ p1.T IX(rs, p1), t2 ⊆ p1.T

Figure 3. The processing of the query ‘office room’ for obtaining p1 as an answer

References

1. C. C. Chang and S. Y. Lee, “Retrieval of Similar Pictures on Pictorial Databases, ” Pattern

Recognition, 1991.
2. S. K. Chang et al., Intelligent Image Database Systems, Vol. 5, World Scientific, 1996.
3. H.J. Yang, H. Y. Kim and J. D. Yang, “Design and Implementation of COIRS(a Concept-
Based Image Retrieval System,” Visual’99, The Netherlands, June, pp. 391-398, 1999.
4. J. D. Yang and H. J. Yang, “A Formal Framework for Image Indexing with Triples: Towards
a Concept-Based Image Retrieval,” Inter. Journ. of Intell. Sys., Vol. 14, No. 6, 1999.
5. J. D. Yang, “An Image Retrieval Model Based on Fuzzy Triples,” Revised for Fuzzy Sets
and Systems, “http://jiri.chonbuk.ac.kr/~jdyang,” 1999.