A method to estimate Grape Phenolic Maturity based on seed images

Felipe Avila
a,,1
, Marco Mora
a,,1
, Claudio Fredes
b,1
a
Department of Computer Science, Universidad Catlica del Maule, Talca, Chile
b
Department Agricultural Science, Universidad Catlica del Maule, Curic, Chile
a r t i c l e i n f o
Article history:
Received 10 September 2013
Received in revised form 7 December 2013
Accepted 16 December 2013
Keywords:
Phenolic Maturity
Seed images
Sequential Forward Selection
Neural networks
a b s t r a c t
The timing of the grape harvest has a strong impact on wine quality. A recent line of studies proposes
visual seed inspection by a trained expert to determine Phenolic Maturity. In this paper a method is pre-
sented to estimate Grape Phenolic Maturity based on seed images. The acquired images present problems
such as shadows, highlights and low contrast. Two classes of seed are dened (mature and immature) by
the expert (enologist) involved in the research. The method consists of three stages: segmentation, fea-
ture extraction and classication. Segmentation was performed by a hybrid method combining super-
vised and unsupervised learning, feature extraction by the Sequential Forward Selection algorithm, and
classication by a Simple Perceptron. The results for each stage are presented. The method as a whole
proved to be simple and effective in the classication of seeds. Therefore, it is possible to visualize the
implementation of the method in real conditions.
2013 Elsevier B.V. All rights reserved.
1. Introduction
Wine is a highly valued drink due to its natural origin and it has
proven antioxidant benets. In wine-producing countries,
improvements in the production process have a strong economic
impact. One of the relevant factors to obtain quality wines is to cor-
rectly determine the harvest timing for the grapes.
Traditionally, Phenolic Maturity estimation is done by a human
expert (enologist) through organoleptic inspection of samples, or
by laboratory chemical analysis. A recent approach to determine
the Phenolic Maturity involves visual inspection of the seed. A pio-
neering work in this eld is found in Ristic and Iland (2005) where
a high correlation between visual seed appearance indicators (col-
or, texture and shape) and the state of Phenolic Maturity of the
grapes is reported. In a recent paper (Fredes et al., 2010), the evo-
lution of wine astringency is studied, a color scale of the seed is
developed, and the colors associated with minimal astringency
are established.
Visual inspection of the seed has the advantage of being simple
to perform, and does not require equipment. However, visual
inspection requires a highly trained expert, and it is inherently
subjective and imprecise, since color perception varies from one
person to another. Furthermore, for the study to be representative,
a large number of analyses must be performed, which is not feasi-
ble in real conditions.
The images involved in this work are those acquired in Fredes
et al. (2010). These images have various problems, such as shad-
ows, highlights and low contrast between pixels of the seed and
the shadow.
Shadows are caused by light direction in image acquisition.
Shadows produce serious problems in the segmentation, analysis
and tracking of objects. Studies that describe and classify the great
variety of techniques for the treatment of shadows can be found in
Al-Najdawi et al. (2012) and Sanin et al. (2012). Shadows are di-
vided into cast shadows and self shadows (Xu et al., 2006a); the
cast shadows are external to the object, while self shadows are
on the object itself. It is noted that the shadows present in the
images of this work correspond to the cast shadows type.
To detect cast shadows, there is an established line of research
based on color models that are invariant to illumination (hereinaf-
ter invariant model or simply invariant). These models are called
invariant because the color conguration obtained from them re-
mains approximately the same in the presence of changes in image
capture conditions (changes in the viewing angle of the object, in
the orientation of the surface, and in lighting conditions). A repre-
sentative work in this area is Salvador et al. (2001), in which invari-
ant models are used to detect and classify shadows in a static
image, and edge detection is used to segment regions with shadow.
A method to eliminate shadows based on representing the color
image using a gray-scale invariant model is proposed in Finlayson
et al. (2006). In this model, the edge of the shadow is dened as the
pixels which are present in the edges of the original image but
which are not present in the invariant. The problem with this
0168-1699/$ - see front matter 2013 Elsevier B.V. All rights reserved.
http://dx.doi.org/10.1016/j.compag.2013.12.006

Corresponding authors. Tel.: +56 712203530; fax: +56 712413650 (F. Avila).
E-mail addresses: favila@litrp.cl (F. Avila), mora@spock.ucm.cl, marcomoracofre@
gmail.com (M. Mora), cfredes@ucm.cl (C. Fredes).
1
Address: Laboratory of Technological Research on Pattern Recognition, Universidad
Catlica del Maule, Avenida San Miguel 3605, Talca, Chile. www.litrp.cl
Computers and Electronics in Agriculture 101 (2014) 7683
Contents lists available at ScienceDirect
Computers and Electronics in Agriculture
j our nal homepage: www. el sevi er . com/ l ocat e/ compag
method is that to generate the invariant, it is necessary to identify
the angle of projection of the shadow, which can be very difcult. A
technique that combines two invariant models is proposed in Xu
et al. (2006b), the rst based on a normalization of the RGB model,
and the second based on the model proposed in Finlayson et al.
(2006). The HSI color model and the c1c2c3 model are combined
in Sun and Li (2010) to obtain a more robust shadow detection.
These methods based on the invariant are of great interest because
they are relatively simple to implement. However, in complex
images (such as grape seed) the invariant models do not produce
good segmentation results, since they are based on the detection
of edges, which are difcult to determine in processed images,
even for the human eye.
To address both problems of visual inspection and poor image
quality, this paper considers the visual inspection of the seed as
a problem of pattern recognition in digital color images. First, a
segmentation method robust to common image problems (shad-
ows, highlights and low contrast) is developed. A hybrid method
for detecting shadows is proposed, which combines neural net-
works of supervised learning with invariant color models. Second,
two classes of maturity are dened, immature and mature. A large
set of color, texture and shape features are computed and the rel-
evance of these descriptors is studied, using the Sequential For-
ward Selection algorithm according to the methodology proposed
in Mery and Soto (2008). The cited study allows to nd a represen-
tative and reduced set of descriptors to separate the patterns into
the categories dened for the problem. Finally, due to the high per-
formance of the identied descriptors, classication is performed
by means of a Simple Perceptron, which allows to distinguish clas-
ses of linearly separable data. Overall, the ability to sort seeds as
mature and immature provides the enologist with objective infor-
mation in order to make a better decision about the timing of the
harvest.
The rest of the paper is organized as follows: Section 2 provides
a review of the shadow detection method in complex images. Sec-
tion 3 presents the details of the classier employed to determine
the maturity of the grapes. Experiments and results are provided in
Section 4. Finally, Section 5 presents the conclusions.
2. Hybrid method of object segmentation robust to shadows
The focus of this section is to describe in detail the hybrid meth-
od of segmentation robust to shadows. This method combines an
unsupervised approach based on invariant color models and a
supervised approach based on neural networks.
2.1. Determining relevant features for classication
A rst approach to separate the seed from its shadow was the
study of color characteristics. For this purpose, samples of pixels
were taken both from seed and shadow in different images. The
values of these pixels were transformed to various color models
(RGB, HSV, YIQ, YCbCr, XYZ, CMYK, Lab, Luv, among others), includ-
ing the invariant models l1l2l3 and c1c2c3 proposed in Gevers
(1999). To determine the channels that allow greater distance be-
tween seed and shadow, the Sequential Forward Selection algo-
rithm (SFS) was used (Jain et al., 1999). This algorithm provides a
ranking of the features according to their contribution to the sep-
aration of the classes (in this case, seed and shadow). The high-
est-ranked features were the invariant model channels; however,
the performance function obtained by the SFS algorithm yielded
a value near zero. Fig. 1 shows that the SFS algorithm selected
three color channels; adding a fourth channel did not signicantly
increase the classication performance. The channels selected by
the SFS were S (from the HSV model), l1 and c3. It is observed that
the performance function provides a very small value, close to 1%.
This shows that color characteristics are insufcient to separate the
seed from the shadow.
Therefore, the use of texture features is considered to separate
the classes, adopting the Haralick (Haralick, 1979) descriptors for
this task. It is noted that these descriptors were chosen over other
texture features, such as LBP and Gabor, due to the better perfor-
mance of the Haralick descriptors in the analysis performed by SFS.
After the selection of descriptors, the extraction of features was
performed as follows: 500 samples of seeds and 500 samples of
shadows were taken from different images. Each sample corre-
sponds to a window of 41 41 pixels. Fig. 2 shows the samples
from one seed image. Thus, the training data set corresponds to a
matrix of 1000 rows and 28 columns, where each column is a tex-
ture feature. For each sample, Haralick descriptors where com-
puted. Initially, 28 texture descriptors were computed, and
application of the SFS algorithm established that nine features
yielded a good separation of the classes.
2.2. Segmentation using a neural classier
A multilayer perceptron (MLP) was considered as classier be-
cause this architecture corresponds to a universal function estima-
tor, and it also allows to group patterns which are not linearly
separable. For training, the Bayesian Regularization method was
adopted (Forensee and Hagan, 1997). The advantage of this method
is that it provides a criterion for determining the number of neu-
rons in the hidden layer, based on the effective parameters of the
network. Briey, the procedure for determining the number of
neurons of the hidden layer is as follows: gradually increase the
number of neurons of the hidden layer until the effective parame-
ters of the network stabilize.
The neural classier has 9 inputs (one for each texture descrip-
tor) and 1 output to differentiate between the two classes (seed
and non-seed). To segment the seed by means of the neural net-
work, a grid with cells of 41 41 pixels is set in the image. For each
cell, the 9 selected Haralick texture descriptors are computed,
which are then assigned as inputs to the neural network. The seg-
mentation process is shown in Fig. 3. Fig. 3(a) presents an example
of a seed image (Fredes et al., 2010), in which very little difference
is observed between the edge of the seed and the shadow. Fig. 3(b)
shows a cell on the original image, illustrating that the size of the
Fig. 1. Color channel selection using SFS algorithm.
F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683 77
cell permits a good approximation to the ideal segmentation.
Fig. 3(c) shows the binary image produced by segmentation, and
Fig. 3(d) shows the edges of the segmented area on the original im-
age. It is observed that a large quantity of shadow has been re-
moved, but the method introduces errors in the border areas of
seed and shadow.
2.3. Segmentation based on an invariant color model
Invariant color models have been developed to segment objects
in the presence of shadows and highlights. The c1c2c3 invariant
model used in Salvador et al. (2001) was adopted for this work,
due to its good performance with shadows and highlights. The
expression of the model is as follows:
c1
i;j
arctan
R
i;j
maxG
i;j
; B
i;j

1
c2
i;j
arctan
G
i;j
maxR
i;j
; B
i;j

2
c3
i;j
arctan
B
i;j
maxG
i;j
; R
i;j

3
The results of applying this model to an image are presented in
Fig. 4. Fig. 4(a) shows the original image in the RGB model. Fig. 4(b)
shows the image corresponding to channel c3 of the invariant
model (this channel was chosen because it obtain the highest rank-
ing in the application of the SFS algorithm). Finally, Fig. 4(c) shows
the automatic segmentation of the c3 channel through the well-
known Otsu method. In the last image it is observed that the
invariant segmentation completely eliminates the shadow and
highlights from the orange.
The above gure illustrates the effectiveness of the invariant to
segment objects in the presence of shadow. However, in more
complex images the results are decient. Fig. 5 shows the segmen-
tation for an image of a grape seed. As mentioned above, these
images have very little contrast between the object and the sha-
dow, and they also contain pixels affected by highlights. Fig. 5(a)
shows the original image of the seed in the RGB model. Fig. 5(b)
shows the image of channel c3. Finally, Fig. 5(c) shows the auto-
matic segmentation of channel c3 by means of the Otsu method.
In the last image it is observed that the invariant segmentation
does not completely eliminate the shadow, and it contains signi-
cant errors inside and outside the seed.
2.4. Hybrid method of segmentation
To improve the performance of the invariant in complex
images, we propose the combination of supervised and unsuper-
vised segmentation, by means of segmentation with a neural net-
work and segmentation with invariant color models, respectively.
For the particular case of the models involved in this work we pro-
pose the following equation:
I
fusion
i; j
1 If I
seg-inv
i; j 1 y and I
seg-net
i; j 1
0 Otherwise

4
where I
seg-net
is the segmentation obtained by the neural network,
I
seg-inv
is the image generated from segmentation in channel c3,
Fig. 2. Sample of the construction of training data set.
Fig. 3. Segmentation performed with MLP.
78 F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683
I
fusion
and is the image resulting from the combination of both
methods.
In order to improve the results obtained with the combination
of supervised and unsupervised methods, a sequence of binary
morphological operations is implemented to remove the noise
present in the image. To remove salt-and-pepper noise present
in the image, a morphological openingclosing sequence is pro-
posed. To obtain a solid object without changing the shape, hole
lling is implemented. Fig. 6 shows the above mentioned stages
of improvement. Fig. 6(a) shows fusion image, Fig. 6(b) the mor-
phological opening, Fig. 6(c) the morphological closing, Fig. 6(d)
the hole lling, and nally Fig. 6(e) the edge of the segmented area
overlaid onto the original image. In the last image it is observed
that the result is closer to the ideal segmentation than the fusion
image.
It is noted that the irregular shape of the contour has no rele-
vance to the problem. The important condition is that the pixels
of the area dened by the contour always belong to the seed,
allowing to implement a subsequent stage of analysis of the seed
surface to determine its level of maturity.
3. Determining grape maturity using neural classier
After the segmentation of the seed, a classier is constructed to
determine the maturity of the seed. The classes of interest used in
this work are mature and immature.
2
In Fig. 7, images of these two
classes can be observed. Fig. 7(a) and (b) correspond to images of the
immature class, and Fig. 7(c) and (d) to the mature class.
3.1. Feature extraction
Following the principle of The More The Better (Mery and
Soto, 2008), a large number of descriptors were computed, in order
to select the most signicant descriptors. In the feature extraction
process, 379 different descriptors were computed of the following
types: Haralick descriptors, intensity descriptors, local binary pat-
terns, Gabor features, crossing line proles, Fourier descriptors,
and contrast descriptors.
To determine the most signicant descriptors the SFS algo-
rithm was used. The study of descriptor relevance showed that
it was possible to separate the two classes using only two
descriptors. In this case the most relevant descriptors for sep-
arating the classes were the Haralick texture descriptor, and
the Gabor texture descriptor. Fig. 8 shows the result provided
by the SFS algorithm. It is observed that the performance ob-
tained is very high, which indicates that the two classes pos-
sess a high degree of separability when the above descriptors
are used.
3.2. Neural classier design
Due to the high separability of the two classes by means of the
identied descriptors, a Simple Perceptron was adopted for classi-
cation purposes. The Simple Perceptron is a widely studied neural
network that allows correctly classify linearly separable data. Per-
ceptron output is binary; thus it allows to differentiate between
two classes of data. The fact that the problem can be solved with
only two descriptors and a simple classier implies that the pro-
posed solution is feasible to be implemented in real conditions.
The output of a Simple Perceptron neuron is given by the following
expression:
Fig. 4. Segmentation performed with invariant model. Image of orange was obtained from (Salvador et al., 2001).
Fig. 5. Segmentation performed on seed image with invariant model.
2
Two images were acquired between March and May 2008. Images corresponding
to the immature class were obtained in March and April 2008, whereas images
corresponding to the mature class were obtained in May 2008 (Fredes et al., 2010).
F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683 79
y
i
g
X
M
j0
w
j
x
j
!
5
where M corresponds to the dimension of the input patterns x; w
j
are the weights of the neuron, and g is the activation function of
the perceptron. The expression of the activation function is as follows:
ga
1 when a < 0
1 when a P0

6
The training set of the perceptron considers a dataset
D fx
i
; t
i
g
N
i1
where x
i
is the input pattern i; t
i
corresponds to
the pattern class i, and N to the number of data to be classied.
Traditional training of the Simple Perceptron is a nonlinear optimi-
zation problem without constraints, where the cost function to be
optimized is the mean squared error:
Fig. 6. Improvement of the hybrid method.
Fig. 7. Images of the classes of interest: (a and b) immature class and (c and d)
mature class.
Fig. 8. Performance of classication into mature class and immature class using SFS
algorithm.
80 F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683
E
1
2
X
N
i1
y
i
t
i

2
7
The perceptron training algorithm is the Gradient Descent,
which has the following expression:
w
s1
j
w
s
j
gy
i
t
i
x
i;j
8
where w
s
j
is the present value of the weights, the learning rate g; t
j
the objective value for each entry, y
i
the value estimated by the net-
work for each input and nally x
i;j
the value of the inputs. It is
important to note that this training is iterative and batch, meaning
that it is performed simultaneously for all inputs.
Fig. 9 shows the architecture of the classier. Firstly, the seed
was segmented, then the descriptors (F
1
and F
2
) were computed
from the segmented area, and these descriptors were presented
as input to the Simple Perceptron in order to be classied. The per-
ceptron output allows to assign the class of seed.
The training set was formed from 80 images, of which 40 corre-
spond to immature seed images and 40 to mature seed images.
Training was carried out as indicated above.
4. Analysis of results
In this section we present the results obtained in the segmenta-
tion stage and the classication stage.
4.1. Segmentation results
To quantitatively evaluate the segmentation method, we com-
pared the results with manual segmentation of the seed (ground
truth). A set of 20 seeds, selected by the human expert (enologist)
including samples of both classes (immature and mature) were
considered in the evaluation. The assessment consisted of compar-
ing the area obtained by the segmentation method and the area of
the ground truth. A sample image of the ground truth is shown in
Fig. 10.
In the evaluation, a performance function was adopted that is
widely used in the literature (Grigorescu et al., 2003). This function
comprises the true positive hits, the false positives and the false
negatives. The expression of the performance function is as
follows:
q
cardTP
cardTP cardFP cardFN
9
where q is the performance function, card (X) is the cardinality of a
set X, TP is the set of true positives, FP is the set of false positives,
and FN is the set of false negatives. Interestingly, this function is
normalized between 0 and 1, which allows to compare the perfor-
mance of different methods.
In particular, the comparison was made between the segmenta-
tion provided by the hybrid method (fusion), by the invariant
alone, and by the neural network alone. Fig. 11 presents the perfor-
mance function for the 20 images, displaying for each image the
value of the function as obtained by the three methods. The rst
bar corresponds to the hybrid method, the second bar to the neural
network method, and the third bar to the invariant. The average
performance function is 0.8 for the hybrid method , 0.75 for the
invariant, and 0.7 for the neural network. The hybrid method is
superior in 75% of the cases, the invariant in 25%, and the neural
network in 0%. The above results numerically show the advantages
of the combination of both a supervised and an unsupervised
method for the segmentation problem addressed.
Fig. 12 shows the overall results of the method for additional
images; it can be observed that the segmented area corresponds
to regions that belong to the seed. It is noted that even though
the edge of the segmented region does not perfectly follow the
real shape of the seed, the dened area is representative of the
target area. From the adequately dened area, descriptors which
may be related to grape maturity (color and texture) can be
extracted.
4.2. Classication results
The image database comprises a total of 120 seeds, which were
divided into a training set and a test set (80 images for training and
40 images for testing). To determine the performance of the con-
structed classier, the percentage of correct classication was
established for the training set and for the test set. Fig. 13 shows
scatter plots for both sets. Fig. 13(a) corresponds to the training
set and 13(b) to the test set. The gure also shows the line that sep-
arates mature and immature classes in both sets. The circles corre-
spond to the descriptor values of the immature class, while the
asterisks correspond to the mature class. It can be observed that
the line found in training correctly separates the classes. In the test
set, it can be seen that a misclassication error exists in one ele-
ment (the error may be a problem of seed classication as per-
formed by the expert).
The classication results are quantied in Table 1. In the train-
ing set a 100% effectiveness for both classes was obtained. In the
test set a 100% effectiveness was obtained for the immature class
and a 93% effectiveness for the mature class. The numerical results
show that the classier has been correctly designed, i.e. a suitable
architecture for the problem has been chosen, and it has been
trained properly. Fig. 9. Classier architecture.
Fig. 10. Manual segmentation: (a) original image and (b) ground truth.
F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683 81
Fig. 11. Evaluation of the segmentation.
Fig. 12. Overall results of hybrid method.
82 F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683
5. Conclusions
In this paper a method was presented to estimate Grape Pheno-
lic Maturity based on seed images. Two classes of seed were de-
ned, mature and immature, following the suggestion of the
expert (enologist) involved in the research. The acquired images
presented problems such as shadows, highlights and low contrast.
We proposed a segmentation method that combines supervised
and unsupervised learning to adequately address the complexity of
the acquired images.
Further, we implemented a robust methodology to determine
descriptors that represent the surface of the seed, in order to accu-
rately separate the classes dened for the problem.
Finally, we designed a simple and effective classier to separate
the classes; this classier is adapted to the separability character-
istics of the data.
The method as a whole proved to be simple and effective in the
classication of seeds (see Table 1). Therefore, it is possible to visu-
alize the implementation of the method in real conditions, due to
the hit rates of the classication method.
Acknowledgements
This work has been fully funded by IDeA FONDEF CA12i10236:
Estimation of Phenolic Maturity based Grape Seed Image. Marco
Mora, Claudio Fredes, Sergio Hernandez. National Commission for
Scientic and Technological Research (CONICYT), Government of
Chile.
References
Al-Najdawi, N., Bez, H.E., Singhai, J., Edirisinghe, E., 2012. A survey of cast shadow
detection algorithms. Pattern Recogn. Lett. 33, 752764.
Finlayson, G.D., Hordley, S.D., Lu, C., Drew, M.S., 2006. On the removal of shadows
from images. IEEE Trans. Pattern Anal. Mach. Intell. 28 (1), 5968.
Forensee, F., Hagan, M., 1997. GaussNewton approximation to bayesian learning.
In: Proceedings of the International Joint Conference on Neuronal Networks,
vol. 3, 1997, pp. 19301935.
Fredes, C., Bennewitz, E.V., Holzapfel, E., Saavedra, F., 2010. Relation between seed
appearance and phenolic maturity: a case study using grapes cv. carmenere.
Chilean J. Agric. Res. 70 (3), 381389.
Gevers, T., 1999. Color-based object recognition. Pattern Recogn. 32 (3), 453464.
Grigorescu, C., Petkov, N., Westenberg, M., 2003. Contour detection based on
nonclassical receptive eld inhibition. IEEE Trans. Image Process. 12 (7), 729
739.
Haralick, R., 1979. Statistical and structural approaches to texture. Proc. IEEE 67 (5),
786804.
Jain, A., Duin, R.P.W., Mao, J., 1999. Statistical pattern recognition: a review. Intell.
Data Anal. 22 (1), 434.
Mery, D., Soto, A., 2008. Features: the more the better. In: The 7th WSEAS
International Conference on Signal Processing, Computational Geometry and
Articial Vision (ISCGAV-2008), Rodos Island, Greece, 2008, pp. 4651.
Ristic, R., Iland, P., 2005. Relationships between seed and berries development of
vitis vinifera L. cv shiraz: developmental changes in seed morphology and
phenolic composition. Aust. J. Grape Wine Res. 11 (1), 4358.
Salvador, E., Cavallaro, A., Ebrahimi, T., 2001. Shadow identication and
classication using invariant color models. In: Proceedings of IEEE
International Conference on Acoustics, Speech, and Signal Processing, vol. 3,
2001, pp. 15451548.
Sanin, A., Sanderson, C., Lovell, B.C., 2012. Shadow detection: a survey and
comparative evaluation of recent methods. Pattern Recogn. 45, 16841695.
Sun, B., Li, S., 2010. Moving cast shadow detection of vehicle using combined color
models. In: 2010 Chinese Conference on Pattern Recognition (CCPR), 2010, pp.
15.
Xu, L., Qi, F., Jiang, R., Hao, Y., Wu, G., 2006a. Shadow Detection and Removal in Real
Images: A Survey. Shanghai JiaoTong University, PR China.
Xu, L., Qi, F., Jiang, R., 2006b. Shadow removal from a single image. In: Proceedings
of International Conference on Intelligent Systems Design and Applications, vol.
2, 2006b, pp. 10491054.
Fig. 13. Scatter plot of seed classication.
Table 1
Classication performance results.
Hits Hits
Training set Test set
Immature type 100 100
Mature type 100 93
Average 100 98
F. Avila et al. / Computers and Electronics in Agriculture 101 (2014) 7683 83