Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11

OLEX: RULE BASED LEARNING FOR TEXT CATEGORIZATION

1
N.poongavanam
2
R.Giritharan, M.C.A., M.E.
1
II-Year \M.E. Computer Science and Engineering
2
Asst.Prof/CSE
Department of Computer science & Engineering
Thiruvalluvar College of Engineering &Tech.,
Vandavasi.

ABSTRACT—This paper describes Olex, a novel method for the automatic induction of rule-based text
classifiers. Olex supports a hypothesis language of the form “if T1 or… or Tn occurs in document d, and none of
Tn+1, . . . Tn+m occurs in d, then classify d under category c,” where each Ti is a conjunction of terms. The
proposed method is simple and elegant. Despite this, the results of a systematic experimentation performed on
the REUTERS-21578, the OHSUMED, and the ODP data collections show that Olex provides classifiers that
are accurate, compact, and comprehensible.

Index Terms—Data mining, text mining, clustering, classification, and association rules, mining methods and
algorithms.

1. INTRODUCTION Olex achieves high performance on all data

A text classifier (or simply “classifier”) is
a program capable of assigning natural language
texts to one or more thematic categories on the
basis of their contents. A number of machine
learning methods like k-nearest neighbors (k-NN),
probabilistic Bayesian, neural networks,and SVMs.
Rule-based classifiers provide the
desirable property of being interpretable and, thus,
easily modifiable based on the user’s a priori
knowledge. In this paper, we propose Olex, a novel
method for the automatic induction of rule-based
text classifiers. Here, a classifier is a set of
propositional rules, each characterized by one
positive literal and (zero or) more negative literals.
A positive (respectively, negative) literal is of the
form T 2 where T is a conjunction of terms t1 ^ …
^ tn (a term ti being a n-gram) and d a document.
Rule induction is based on a greedy optimization
heuristics whereby a set of high-quality rules is
generated for the category being learned. Unlike
other (either direct or indirect) rule induction
algorithms, e.g., Ripper and C4.5, Olex is a one-
step process, i.e., it directly mines the final rule set,
without the need of any postinduction optimization.
sets and induces classifiers that are compact and
comprehensible. The comparative analysis
performed against some state-of the-art learning
methods, notably, Naive Bayes (NB), Ripper, C4.5,
polynomial SVM, and Linear Logistic Regression
(LLR) demonstrates that Olex is more than
competitive in terms of both predictive accuracy
and efficiency. A previous version of Olex, where
rules with just “simple” terms (i.e., terms of the
form t 2 d and: t 2 d) are learned, is described in
[8]. In the remainder of this paper, after providing
an overview of Olex and giving some preliminary
definitions and notation , we state the optimization
problem of selecting a best set of discriminating
terms (which is the heart of our method) and prove
that this task is computationally difficult . Thus, we
propose a heuristic approach to solve it and give a
description of the whole learning process.
2. OLEX OVERVIEW
Olex is an inductive rule learning method for Text
Categorization (TC). Informally, the TC induction
problem can be stated as follows: Given .

309
 Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11
• a background knowledge B as a set of
ground logical facts of the form t 2 d, meaning that
term t occurs in document d (other ground
predicates may occur in B as well) and .
• a set P of positive examples consisting of
ground logical facts of the form d 2 c, meaning that
document d belongs to category c (ideal
classification); given P, the set N of negative
examples consists of the facts d 2 c that are not in P
constructs a hypothesis Hc (the classifier of c) that,
combined with the background knowledge B, is
(possibly) consistent with all positive and negative
examples, i.e., B^Hc _ P and B^Hc 6_ N. The
induced rules will allow prediction about the
belonging of a document to a category on the basis
of the presence or absence of some terms in that
document.
The above induction problem is essentially an
instance of Inductive Logic Programming (ILP),
which deals with the general problem of inducing
logic programs from examples in the presence of
background knowledge. It is well known that ILP
problems are computationally intractable, so that a
main topic is that of identifying classes of
programs that are efficiently learnable. The theory
of PAC-learnability [9] provides a model of
approximated polynomial learning where the
polynomially bound amount of resources (both
number of examples and computational time) is
traded off against the accuracy of the induced
hypothesis. It is well known that, if a learning
problem is PAC-learnable, the related consistency
problem is in the randomized polynomial
complexity class RP [10]. In [9], Valiant identifies
a subset of propositional logic which is PAC-
learnable. More recently, several authors
investigated the identification of PAC-learnable
subclasses of first-order Horn clauses, providing a
number of both positive and negative results. For
an instance, in [11], Dzeroski et al. show that a
restricted class of function-free clauses, namely k-
iscriminative nonrecursive ij-determinate predicate
definitions, is PAC-learnable. In [19], Kietz and
Deroski prove that relaxing some syntactic
restriction to the above class of programs implies
the loss of PAC-learn ability (unless RP ¼ NP).
Finally, in [13], Gottlob et al. show that learning a
function-free Horn clause with no more than k
literals from a set of both positive and negative
examples expressed as function-free ground Horn
clauses is NP-hard (notably, the respective ILP-
consistency problem is _P2 ) and, thus, not PAC-
learnable (unless RP ¼ NP). An overview of some
important techniques for deriving complexity
310
results has been given by Cohen and Page in [8].
However, while in ILP it is assumed that the input
sample is consistent with some hypothesis in the
hypothesis space, this is not necessarily true in TC;
indeed, the relationship between terms and
categories is nondeterministic, i.e., it is not
possible, in general, to correctly categorize a
document under a category only on the basis of the
terms occurring in it. For this reason, the expected
induced hypothesis is in general one which
maximally satisfies (both positive and negative)
examples.
3. NOTATION AND PRELIMINARY
DEFINITIONS
Throughout this paper, we assume the existence of
1. a finite set C of categories, called classification
scheme, 2. a finite set D of documents (i.e.,
sequences of words), called corpus; D is partitioned
into a training set TS, a validation set and a test set;
the training set along with the validation
etrepresent the so-called seen data, used to induce
the model, while the test set represents the unseen
data, used to assess the performance of the induced
model, and 3. a relation I _ C_D (ideal
classification) which assigns each document d 2 D
to a number of categories in C. We denote by TSc
the subset of the raining set TS whose documents
belong to category c according to I (the training set
of c). For this reason, in the following, we will
concentrate on a single category c 2 C. Once a
classifier for category c has been constructed, its
capability to take the right categorization decision
is tested by applying it to the documents of the test
set and then comparing the resulting classification
to the ideal one. The effectiveness of the predicted
classification is measured in terms of the classical
notions of Precision, Recall, and F-measure.
4. SELECTION OF DISCRIMINATING
TERMS:PROBLEM DEFINITION AND
COMPLEXITY
In this section, we provide a description of the
optimization problem aimed at generating a best set
of discriminating terms (d-terms, for short) for
category c 2 C. In particular, we give a formal
statement of the problem and show its complexity.
To this end, some preliminary definitions are
needed (Table 1). A term (or n-gram) is a sequence
of one or more words, or variants obtained by using
word stems, consecutively occurring within a
document. A scoring function _ (or feature
 Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11
selection function—often simply “function”,
hereafter), such as Information Gain and Chi
Square (see, e.g., [12] and [33]), assigns to a term t
a value _ðt; cÞ expressing the goodness” of t w.r.t.
category c. Scoring functions are used in TC for
dimensionality reduction: noninformative words
are removed from documents in order to improve
both learning effectiveness and time efficiency.
Definition 1 (vocabulary). Given a scoring
function _ and a nonnegative integer _, let Vc(Ø,v)
denote the set consisting of the _ terms t, occurring
in the documents of TSc, having the highest value
of Ø(t,c). The vocabulary V(Ø,v), for the given Ø
and v, is the set consisting of the best v terms,
according to Ø, of each category c € C.
Definition 2 (coterms). Let us fix a vocabulary
V(Ø,v). A coterm (conjunctive term) T over
V(Ø,v) of degree k is a conjunction of terms t1
^….. ^ t k, with ti € V(Ø,v), 1< i< k. A coterm of
degree 1 is a simple term.
Definition 3 (d-terms). Let us fix a vocabulary
V(Ø,v). A d-term (discriminating term) for c over
V(Ø,v). is a pair < T,s > , where T is a coterm over
V(Ø,v). and s € {+,-) the sign of T. We will
represent< T, s > as Ts. A d-term with sign “+”
(respectively, “_”) is called positive
(respectively,negative) d-term. We say that 1) Ts
occurs in a document d if T occurs in d, 2) Ts has
degree k if T has degree k, and 3) Ts1and Ts 2 are
independent if T1 and T2
TABLE 1
List of the Main Symbols Used in This Paper
311
Definition 4 (eligible documents). Given a set of
(independent) d-terms Xc={T+1,….T+n, T-n+1, T-
n+m} for c, we say that a document d 2 TS is eligible
for classification under c according to Xc if any of
the positive d-terms T+1,…, T+n occurs in d and
none of the negative d-terms T-n+1,…,Tn+m occurs
in d4.
PROBLEM DT-GEN (d-term generation).
Given a vocabulary V(Ø,v), find a set Xc of
(independent) d-terms over V(Ø,v) such that, by
classifying under c the set E(Xc)of the documents
eligible for classification under c according to Xc,
the resulting Fα (Xc)is maximum. Intuitively, a
solution of problem DT-GEN is a set of d-terms Xc
such that E(Xc)best “fits” the training set
TSc.Next, we show that DT-GEN is a
computationally difficult problem. To this end, we
need the following preliminary result.
Lemma 1. Let A be a set of independent coterms
over a given vocabulary V(Ø,v). Then, the size of
A has no polynomial bound on the size of V(Ø,v).
Proposition 1. Problem DT-GEN requires
exponential time in the size of the vocabulary
V(Ø,v)
Proof. Even restricting problem DT-GEN to admit
solutions Xc consisting solely of independent d-
terms, Xc has no polynomial bound on the size of
V(Ø,v) (this immediately follows from Lemma 1).
Thus, an exponential amount of time is needed to
generate a solution. Tu Next, we show that DT-
GEN remains difficult even if we restrict Xc to
consists of d-terms of degree 1.
5. AHEURISTICS FOR DEALING WITH
PROBLEM DT-GEN
To deal with the complexity of problem 1DT-GEN,
we devised the greedy heuristics sketched in Fig.1.
The input consists of the vocabulary V(Ø,v), along
with the parameter α. The do-while-loop of lines 4-
9 generates the sequence of sets of d-terms
X0=Ø,…,Xi=GNS(X i-1),…,Xm=GNS(X m-1)
by invoking, at each round, function Generate-
New-Set (lines p1-p11). The ith call to this function
“extends” the current solution Xi_i by picking
(lines p3-p10) the term t € V(Ø,v), which, joined to
a d-term Ts in the current solution (line p5), yields
the greatest increase of the objective function Fα
 Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11
(notice that the current maximum value of Fα is
stored into Fmax, a global variable which is
modified by Generate-New-Set at line p8). The
symbol _ at line p4 represents the logical constant
true (taken both positively and negatively). This
formal device is needed in order for the algorithm
to capture also d-terms of degree 1. The evaluation
of F_ is based on (2). Once the best term t has been
selected, it is returned to the main program through
the global variable topt (which is updated at line
p8) and then removed from V(Ø,v), (line 8). The
do-while-loop iterates as long as the vocabulary
V(Ø,v), is not empty and a new d-term is
generated. It is straightforward to realize that the
size of Xc.
6. THE LEARNING PROCESS
While algorithm Greedy-Olex is the search of a
“best” classifier over the training set, for given
values of the input parameters, the learning process
is the search of a “best” classifier over the
validation set, for all input parameters values. Let
Hc(Ø,v,α) denote the classifier of c associated to a
312
solution Xc of problem DT-GEN generated by
algorithm Greedy-Olex, for given values of Ø, v,
and α. Now, to learn a “best” classifier Hc of c,
essentially Olex repeatedly induces Hc(Ø,v,α), for
different input vocabularies, each time validating it
over the validation set. Here, the input consists of a
number of vocabularies (obtained by setting
different values of Ø and v), along with a value for
(which depends on the needs of the application at
hand; usually, α=0.5).Whenever a classifier
Hc(€,v,α)is generated by the induction step (lines
4-5), it is validated over the validation set (line 7).
Finally, the classifier Hc with the maximum value
of the F-measure over the validation set is output
(line 9). Hc is assumed to be the “best classifier” of
c. Notice that, of the three model parameters €,v,
and α, only the former two are actually used for
“driving” the search of Hc.
6.1 Benchmark Corpora
The OHSUMED test collection is a subset of the
MEDLINE database, which is a bibliographic
database of important, peer-reviewed medical
literature maintained by the National Library of
Medicine. The subset we considered is the
collection consisting of some documents from the
research abstracts .The ODP data set consists of
millions of Websites classified according to a set of
hierarchically organized categories. For our
experimentation, we decided to use the subset of
some documents classified under the categories of
the Top/Science subtree, which has a first level
with 5 categories (hereafter, we will refer to this
subset as ODPS25). We performed our experiments
using a flat structure, where all documents below
level one were considered as
belonging to the respective root . As for
documents, we extracted, from the ODP data set
stored in Resource Description Framework (RDF)
format, the title and the description (we left out the
URL) for each Web page in the catalog. Thus, an
ODP document consists of the title of the Web
page plus its description.
6.2 Document Preprocessing
Preliminarily, all corpora were subjected to the
following preprocessing steps. First, we removed
from documents all words occurring in a list of
common stop words, as well as punctuation marks
and numbers. Then, we generated the stem of each
of the remaining words, so that documents were
 Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11
represented as sets of word stems. Second, we
proceeded to the partitioning of the training
corpora:
As far as REUTERS-21578 and OHSUMED are
concerned, we segmented each corpus into five
equalsized partitions for cross validation. During
each run, four partitions will be used for training,
and one for validation (note that validation and test
sets coincide in this case). Each of the five
combinations of one training set and one validation
set is a fold.
Concerning ODP-S25 (for which the holdout
method was used), we segmented the corpus into
two partitions: the seen data (70 percent of the
corpus documents) and the unseen data (the
remaining 30 percent). The former is used to
induce the model (according to the learning process
of Fig. 2), while the latter is used for testing. The
seen data were then randomly split into a training
set (70 percent), on which to run algorithm Greedy-
Olex, and a validation set, on which tuning the
model parameters. We performed both the above
splits in such a way that each category was
proportionally represented in both sets (stratified
holdout).
Finally, for every corpus and training set, we
scored all terms occurring in the documents of the
training set TSc of c, for each c € C.
6.3 Performance Metrics
Classification effectiveness was measured in terms
of the classical notions of Precision, Recall, and F-
measure, as defined in Section 3. To obtain global
estimates relative to experiments performed over a
set of categories, the standard definition of micro
averaged Precision and Recall was used.
6.4 Results with REUTERS (Cross Validation)
The first data set we considered is the REUTERS-
21578 and the task was to assign documents to one
or more categories of R90. As already mentioned,
performance evaluation was based on fivefold
cross validation. At each fold, we conducted a
number of experiments for the induction of the best
classifier of each category in R90 according to the
algorithm sketched in Fig. 2. In particular, the
algorithm was executed with input vocabularies.
6.4.1 Performance
313
F-measure and BEP obtained at each of the five
folds, and the respective means are given Table 3,
in turn, provides a picture of the results for the 10
most frequent categories of R90 (hereafter referred
to as R10), averaged over the five folds. In
particular, besides F-measure and BEP, for each
category we report the average characteristics of
the respective best classifiers (i.e., number of rules
and number of negative literals occurring in each
rule—recall that the sum of these two
values equals the number of induced d-terms).
6.4.2 Effect of Category Size on Performance
We partitioned the categories in R90 with more
than seven documents into four intervals, based on
their size. Then, we evaluated the mean F-measure
over the categories of each group, averaged over
the five folds. Results are summarized in Table 5.
As we can see, the F-measure values indicate that
performances are substantially constant on the
various subsets, i.e., there is no correlation between
category size and predictive accuracy (this is not
the case of other machine learning techniques, e.g.,
decision tree induction classifiers, which are biased
toward frequent classes [15]).
6.5 Results with OHSUMED (Cross Validation)
The second data set we considered is OHSUMED
and the task was to assign documents to one or
more categories of the 23 MeSH diseases. As for
the REUTERS-21578, we performed a fivefold
cross validation by running, at each fold, Algorithm
2 with input vocabularies V(CHI,v), with
v€{10,20,30,..100} (also in this case, _ was set to
0.5). Performance results are reported in Table 6.
As we can see, the average _-F and _-BEP are
equal to 66.08 and 66.31, respectively.
In Table 7, we summarize the performance values
for the five most frequent MeSH categories, along
with the characteristics of the respective best
classifiers, averaged over the five folds. Again, we
notice the compactness of classifiers (at most 40
rules for category C23).
TABLE 7
OHSUMED—Cross-Validation Results for the
Five Most Frequent MeSH Categories
 Proceedings of the National Conference on Recent Trends in Engineering Sciences 2k11
6.6 Results with ODP-S25 (Holdout Method)
ODP-S25 is a data set one order of magnitude
larger than both REUTERS-21578 and
OHSUMED. In particular, the most frequent
category “biology” owns 28,286 documents, and
there are four categories with more than 10,000
documents. These characteristics make ODP-S25
very suitable for testing both the accuracy and the
efficiency of the learning procedure of Olex on
large categories. Because of the large size of this
data set, experiments were conducted based on the
holdout method. In particular, according to the
learning procedure of Fig. 2, we learned classifiers
over the training set and selected the best ones over
the validation set. Then, we assessed their
performance over the test set. The learning
procedure was run with the same input
vocabularies used for the previous two data sets
(and α=0.5).
7. CONCLUSION
We have presented Olex, a novel approach to the
automatic induction of rule-based text classifiers.
In the first part of this paper, we provided a formal
description of the method. In particular, we
described the problem of determining a best set of
discriminating terms for a category and proved its
intractability. Then, we showed how rules are
derived from a given set of discriminating terms.
The Olex’s hypothesis language consists of rules
with one positive conjunction of terms and (zero
or) more negative ones. Thus, Olex predictions
require testing the simultaneous presence of several
terms (forming the positive conjunction) along with
the simultaneous absence of several sets of terms
(each forming a negative conjunction). While there
is in the literature a wide range of rule learning
algorithms, one contribution of this paper is the
form of the hypothesis language. We performed
experiments on three data collections, namely,
314
REUTERS-21578, OHSUMED, and ODP. We also
conducted a thorough comparative study against
NB, polynomial SVM, Ripper, C4.5, and LR. We
found that Olex consistently achieves
comparatively high-performance results,
significantly outperforming most of the other
approaches. In addition, it showed to be very
efficient (by far the fastest method, apart from NB).
Results on ODP demonstrate that Olex behaves
well also on large data sets (from both predictive
accuracy and efficiency viewpoints).
REFERENCES
[1] D.D. Lewis and P.J. Hayes, “Guest Editors’
Introduction to the Special Issue on Text
Categorization,” ACM Trans. Information Systems,
[2] F. Sebastiani, “Machine Learning in Automated
Text Categorization,” ACM Computing Surveys,
[3] W.W. Cohen and Y. Singer, “Context-Sensitive
Learning Methods for Text Categorization,” ACM
Trans. Information Systems,
[4] D.E. Johnson, F.J. Oles, T. Zhang, and T.
Goetz, “A Decision-Tree- Based Symbolic Rule
Induction System for Text Categorization,” [5] W.
Li, J. Han, and J. Pei, “Cmar: Accurate and
Efficient Classification Based on Multiple-Class
Association Rule,” Proc. First IEEE Int’l Conf.
Data Mining (ICDM), 2001.
[6] J.R. Quinlan, “Generating Production Rules
from Decision Trees,” Proc. 10th Int’l Joint Conf.
[7] C. Apte´, F.J. Damerau, and S.M. Weiss,
“Automated Learning of Decision Rules for Text
Categorization,” ACM Trans. Information Systems,
[8] P. Rullo, C. Cumbo, and V.L. Policicchio,
“Learning Rules with Negation for Text
Categorization,” Proc. 22nd Ann. ACM Symp.
Applied Computing (SAC ’07),
[9] L.G. Valiant, “A Theory of the Learnable,”
Proc. 16th Ann. ACM Symp. Theory of Computing
(STOC ’84),
[10] M. Anthony and N. Biggs, Computational
Learning Theory. Cambridge Univ.