Evaluation of Resources

LRMDS
Assessment and Evaluation

(A&E)
Evaluating Learning
Resources
LRMDS Assessment and Evaluation: Evaluation of Resources
LRMDS
lrmds pub 1.pub
ro nga
ib
l
a
g
n
n
a
m
Ania
g nga
in
b
u
i
it
y
a
g
makagaya agbalin daytoy a
o
agbasa, tapn a, ken makatulong
n
paset ti biag panagdur-as na,
daytoy ti
bro
li
a
y
o
t
y
a
d
t
ket nasayaa
This session is intended to provide

the basic information and
instructions relating to criteriabased evaluation of resources.
Participants will develop
understanding of the Assessment
and Evaluation guidelines,
standards and specifications as
they relate to the evaluation of
learning, teaching and professional
development resources.
Enabling Objectives
Explain why having specifications and standards are
essential to doing an evaluation of learning/teaching
resources
Give a brief description of the contents of the following
areas for which specifications have been developed:
IPR management
Educational Soundness
Educational Quality
Social content
Technical
Terminal Performance
Objective
To evaluate at least one resource and
prepare a Request Brief for the action to be
taken on the said resource.
Apply the appropriate evaluative criteria and
instruments in the evaluation of specific
resources.
Enabling Objectives
Explain the purpose of the Request Brief and
identify its key contents
Define the key terms/concepts in the LRMDS
A & E system Guidelines
Content
The importance of standards and
specifications for evaluation
The LRMDS Assessment & Evaluation
System at the regions, divisions and lead
school levels
Processes, workflow, personnel
Evaluation specs, guidelines and
instruments
Content
Preparation of the Request
Brief
Preparation of the Evaluation
Plan
Evaluation of resources using
appropriate criteria and
instruments
Introduction to the concepts of Evaluation,

Standards, Specifications
Introduction to the concepts of Evaluation,

Standards, Specifications
Activity 1
Aka Mirror, mirror on the wall, which
goody is the tastiest of all?
(Refer to activity sheet #1)
10
Activity 1
Instructions:
1.Divide the participants into three groups.
2.Give each team a bag of goodies, e.g.,
(banana cake, hopia and other snack
food, at least 3-4 kinds, plastic spoons
and forks, small paper plates)
11
Activity 1
The team should evaluate the food items and agree on a
recommendation for each item.
Which of the items should be recommended for inclusion in
the school cafeteria menu?
Are there snack items which may need reformulation
before being considered for cafeteria distribution?
Each team must have a consensus on their
recommendations. The recommendations of all the
groups will be consolidated.
Each team should have a convener and a documentation
officer.
12
Activity 1
Document your process and write your recommendations.
The assigned presenter will present the report of the group
to the plenary.
The report must contain the following information:
(1)A description of the processes that the team used in
evaluating the food items and in agreeing on its
recommendations;
(2)The problems, issues which the group encountered in
the process of evaluation;
(3)The group recommendations based on the evaluation.
13
Processing
Discuss the processes that the groups employed.
How did each group conduct the evaluation?
Intrinsic to evaluation are standards and the use of criteria
to measure standards.
What were the criteria and standards used by the
groups?
Did the group try to evaluate without agreeing on the
standards and criteria first?
Why was it important to agree on the standards and
criteria first?
14
Processing
Would your work have been easier if the
processes and persons responsible for the
processes were defined, if the areas for
which standards or specifications were
defined?
How did you resolve your differences?
15
Evaluation, Standards and

Specifications
What is evaluation?
Evaluation is the systematic determination
of the merit, worth, and significance of
something or someone; fit for purpose;
using criteria against a set of standards
16
Evaluation, Standards and

Specifications
A standard is an established norm or
requirement.
Standards are developed or formulated by
professional groups, associations, unions,
guilds who have a stake in the product or
service being evaluated.
17
Standards
Sometimes the standards are formulated by
private groups through formal consensus.
In many instances, especially when public welfare
is concerned, government through its
instrumentalities formulate the standards for the
evaluation of a product or service and the
standardization process is by edict, that is it is
established by law or an executive issuance.
18
Specifications
A specification is an explicit set of
requirements to be satisfied by a material,
product, or service.
A specification is a type of a standard. It
provides the necessary details about the
specific standards required.
19
LRMDS Assessment and

Evaluation System
The Assessment and Evaluation System is for
identification of materials to be procured,
redeveloped or reproduced is designed to
support schools, divisions, regions and
central office:
in identifying learning, teaching and
professional development resource needs
reviewing and recommending resources for
acquisition and procurement
20
LRMDS Assessment and

Evaluation System
identifying existing resources for
reproduction and or redevelopment
evaluating teacher developed
materials for local use and shared
distribution via LRMDS
21
Activity 2
(Refer to Activity Sheet #2)
Activity Instructions:
The activity is to level understanding of basic
terms, concepts, processes and purposes
for evaluation by the group before the
evaluation of actual resources is
undertaken.
Modified Jeopardy game where the
answer is not really the answer and the
question is really the answer.
22
Jeopardy
Read the Guidelines and the Handouts to
get the definitions of terms and or
concepts that you will encounter in
evaluation of resources.
Work in pairs. Each pair should write on one
side of the meta strip the definition of the
term; on the other side, write the question
to which the definition is the answer.
23
Jeopardy
For example:
Is an opening, or rupture, in a planet's surface
or crust, which allows for the release of hot ash,
and gases.
The answer is A VOLCANO
Curriculum, subject matter, skills and
processes, age and stage of schooling, cultural
inclusivity and language of instruction.
The answer is Educational Specs
24
Define/explain
Accessibility Specifications
Intellectual Property Rights Specifications
Storage and Distribution
Publication and Use
Technical Specifications
Assessment and Evaluation
QA Framework
25
Jeopardy/ Matching Game

Work as a group.
You are given meta strips with definitions, terms
and explanations.
Read and discuss with your group as to where the
description belong or best describe the terms
posted on the board.
40 minutes
26
Terms
Accessibility
Specification
5
6
Description
Geospatial, physical and intellectual ability,

special needs: physical, intellectual differences,
flexible operation and presentation.
Intellectual
Author, contributors, copyright, access and
Property Rights
distribution, equitable: negotiation is balanced
specification
between rights owner and user.
Storage and
Version control for management and
distribution
maintenance, meta data, version control,
discovery, publish in varied formats.
Publication and use File types, formats for reproduction and usage
permissions, communicated information
contains, educational purpose, format.
Technical
Format, file size, authoring software, user plugSpecifications
ins, players, software interoperability.
Assessment and
Identifying learning, teaching and professional
Evaluation
development resource needs, reviewing and
recommending resources for acquisition and
procurement.
QA framework
Supports LRMDS personnel to identify and
consistently implement required actions to
LRMDS Assessment
andagreed
Evaluation: standards
Evaluation of Resources
achieve
and specifications.
27
Quick Processing of the activity

Purpose of the activity: assist the
participant manage his learning through
a collaborative activity.
Establish a common
understanding/working definition of terms
to be used in subsequent activities.
28
Introduction to QA and Criteria

based evaluation
29
What is Quality Assurance?

Quality
refers to the totality of characteristics of a product
or service that bears on its ability to satisfy stated
and implied needs.
Principles
provide the frame of reference for achievement of
quality.
Measures & requirements
base level descriptors.
Criteria
operational requirements for quality.
Indicators
30
concrete objectives for criteria.
Ensuring Quality
LRMDS Quality Assurance Framework describes the
parameters to ensure that all learning, teaching and
professional development resources that are catalogued
and accessed or located via the LTMDS web portal are
of high educational and technical quality.
31
Ensuring Quality
QA is a coordinated process for deliberately bringing about
the attainment of the desired quality. It is a systematic
process of monitoring all management, organizational,
productivity and services of the LRMD systems against
standards of quality in order to raise performance.
32
Ensuring Quality
LRMDS Quality Assurance Framework describes the processes that ensure
that learning, teaching and profession development resources are:
Relevant to the needs and expectations of the users;

Accessible by the range of technology available to the users;
Usable and functional, and;
Pedagogically sound and cohesive.
33
Purpose
The LRMDS Quality Assurance Framework has been
developed to assist:
LRMDS Managers to manage, maintain and
continuously improve access to quality resources,
LRMDS development and production teams to comply
with agreed standards and specifications for educational
resources.
Key LRMDS personnel within Regions and Divisions to
implement QA processes to ensure the continued
provision of quality services and products
Schools and teachers to be able to identify quality
resources in relation to educational need
34
Quality Framework
Area of Quality:
Principles:
Characteristics
Criteria &
indicators
Specifications
and/or
Requirements
Guidelines
Procedures
Instruments
(tools/templates)
Review
Application
A process of continuous improvement

35
Measures & Requirements

LRMDS Quality Assurance measures &
requirements:
Clearly documented and transparent processes
Well defined standards, specifications and
evaluation criteria
Well trained personnel
Capacity of the LRMDS system to respond to
user need.
High quality products and services
36
Principles
The main principles underlying the LRMDS quality assurance processes
are:
User centred
There is active and integrated involvement of LRMDS users at all

levels school, division and region.
Responsive
LRMDS product and service delivery is informed by user

requirements and feedback for each of the LRMDS sub-systems.
LRMDS employs the use of the most appropriate media for the
content, taking into account the profiles of schools, teachers, learners
and access to technologies.
Standards based
LRMDS inputs, processes and outputs conform to agreed quality

standards and specifications.
37
Under the microscope:

LRMDS QA
38
LRMDS Assessment & Evaluation QA
39
Activity 3
(Refer to activity sheets #3)
Evaluating Resources
Purpose of the Evaluation: To select resources to be:
A. Catalogued in the LRMDS portal
B. Uploaded to the portal and catalogued
Types of Resources:
Print
Digital print
Online websites
Video based
Interactive resources
40
Instruments used and their purposes

Refer to the instruments and point out to
the mandatory requirements stipulated in
the instrument.
Ask participants to read them out aloud.
41
Evaluation Streams
Explain to the participants that in LRMDS, the
instruments will be used in various streams of
evaluation, namely,
Evaluation of existing resources in existing
format for reproduction and cataloguing: digital
and non digital
Evaluation of existing resources for
redevelopment: digital and nondigital
Evaluation of teacher-developed resources
Evaluation of On-line resources and sites
42
Activity 3
Determine which of the Evaluation instruments
should be used.
Refer to Section 6 Guidelines and Processes for
LRMDS Assessment and Evaluation.
Use the available copies of the instruments for
the actual conduct of the Evaluation
Each group evaluates at least 1 resource using
the instruments.
43
Activity 3
All resource evaluations should commence with the IPR
review. When many resources belonging to a series are
to be evaluated then the IPR can be completed once for
all.
Any DepED owned and copyright resources for which
content errors, accuracy issues, editorial issues or layout
issues are identified can be fixed and revised and then
uploaded. If the IPR allows. The requirement to revise
and correct the issues would form part of the Evaluation
recommendation.
44
Activity 3
Be prepared to make a specific
recommendation at the end of your
evaluation.
As you apply the instruments to specific
resources, note all criteria and instructions
which were problematic.
45
Why do we need specifications?

To determine:
Quality and educational value
Fit for purpose
Access and Use
Management, maintenance, storage
Publication, discovery & distribution
46
Specifications
Educational
curriculum, subject matter, skills and

processes, age and stage of schooling,
cultural inclusivity
Accessibility
geospatial, physical and intellectual

ability
Technical
format, file size, authoring software,

user plug-ins, players, software
interoperability
IP & rights
management
author, contributors, copyrights, access

and distribution, use
Storage & distribution version control for management and

maintenance, access
Publication & use
file types, formats for reproduction and

use.
47
Achieving quality
No LR, TR, PDM resource is likely to encompass
all quality principles in its own right as many of
these are realised in the relationships between
teacher, learner, resource and learning
environment.
Usually, the principles and associated criteria will
consist of quantitative and qualitative elements.
Thus a concept of 'reasonableness' on balance is
used in the application of the criteria and applied
throughout the quality assurance process.
48
49
50
Criteria
Criteria are focused requirements for
quality that are close to the operational
level though they do not necessarily
contain a specific scale.
Testable statements formulated on the
basis of accepted statements
They can express minimum requirements
(statutory conditions) and best practice
(requirements that must be fulfilled)
51
Indicators
Indicators are:
Concrete and measurable
Qualitative or quantitative
Express central characteristics (e.g. of
resource to evaluated by the criteria)
Indicators should altogether provide a
picture of whether the resource is in
compliance with the criteria.
52
Criteria & Indicators Examples

Non print: Criteria
Indicators
Content stimulates and promotes critical

thinking
4. More than 80% of the content

stimulates and promotes critical
thinking
3. 60-80%
2. 40-59%
1. Less than 40%
Graphics/colors/sounds are used for the

appropriate instructional reasons
4. 90-100% is appropriate
3. 80-89%
2. 70-79%
1. Less than 70%
Content is logically developed and

organized
4. 90-100% is logically developed

and organized
3. 80-89%
2. 70-79%
1. Less than 70%
5353
Criteria & Indicators Examples

Non print: Criteria
Indicators
Content stimulates and promotes critical thinking

This might occur in many ways, but the main point is that the activity requires a
cognitive effort, not simply a chance selection.
Graphics/colors/sounds are used for the appropriate instructional reasons.
Does the graphic style, use of color and sound match the content and the target
audience?
Content is logically developed and organized

The underlying question is: Can you understand the apparent structure of the
knowledge domain and how it represented to the user? In random sequence
structures, the logic is determined by the user and it is difficult to say that it is not
appropriate from just a review.
54
QA Elements A & E example

Structures
Inputs
Processes
Outcomes &
Products
School Learning Resource

Committee
School Report Card
Application of LRMDS
specifications &
criteria/checklists to:
Evaluated & recommended

resources for cataloguing
LRMDS Division Quality

Assurance Team
Annual Implementation
Plan and School
Implementation Plan
Region LRMDS Quality

Assurance Team
LRMDS online catalogue
Trained evaluators
LRMDS Trainers
Existing resources
identify learning,
teaching and
professional
development
resource needs
evaluate teacher developed
materials for local use and
shared distribution via
LRMDS
Reproduction
Redevelopment
Access to quality resources
via LRMDS online catalogue
Analysis of identified school

needs and completion of LR
Plan, production of DEPD
and REDP
55
Ensuring quality- Example

LRMDS Assessment & Evaluation system
Process
Core Quality
objective
Identifying the Identifying

responsible
methods or
actors
instruments
that can be
used
to assure
quality
Measuring the
success of the
quality
objective.
Assessment &
evaluation system
effectively &
efficiently
assesses the
needs of target
users
There are LR
plans at the
region, division
& school levels
that respond to
the needs of
target users
School LR
Teams
LR Assistance
Plans of
Reg/Div reflect
consolidated
needs of
School LR
plans
Division
LRMDS Team
& Trainers
Convergence
table to
compare LR
needs identified
with Reg/Div
LR Assistance
Plans
Division
LRMDS
Manager
56
Evaluation planning
What is the purpose of the evaluation?
Need for science resources to support teaching and
learning in grade/year.
Science resources exemplifying constructivist learning
principles.
What is the source of the resources?

Within DepED?
External?
How are they to be reproduced & distributed?

Offline?
Online?
Online and offline?
57
Evaluation planning
Answers to these questions should determine how
the criteria and outcomes are weighted.
E.G.
A very sound resource that is highly instructivist and
didactic will not fulfill the need for a resource based on
constructivist principles but it can still be a good
resource.
Or a good resource with typographical errors may be able
to be fixed.
Or a good resource in an older version of flash may be able
to be redeveloped.
58
Criteria and indicators

Remember to determine that the purpose of
the evaluation is understood by all
evaluators.
Are there options for the outcomes of the
evaluation? For example issues can be
fixed?
59
LRMDS A & E Evaluation

Instruments
Intellectual Property Rights
Educational Soundness
Educational Quality Evaluation for:
- Print
- Charts, posters, drill & flash cards
- Non-print
- General reference
Technical: usability, interoperability, offline digital
Accessibility Evaluation (SPED)
Online site evaluation
60
Evaluation Planning: Which instruments? What

criteria, if any, is mandatory?
Evaluation Planning
Possible scenarios
What is the purpose of the Evaluation?
To procure science learning resources for

Grade V to be distributed to all schools in
Region
To procure professional development materials
for elementary science teachers
What resources are to be evaluated?
DepED bureau produced.

Foreign Assisted project developed
Commercial
What is the format of the resources
Print
Non-print (digital offline or online)
Online site
What funds are available for procurement?
School MOOE; FAP e.g. SPHERE

Regional funds
What expertise is required to conduct the

evaluation?
Science, elementary education, evaluation,

technical
Application of Evaluation Instruments

Evaluation Scenario
Purpose: To procure science learning resources
for Grade V
Source: Foreign Assisted project developed &
Commercial
Format: print
Funds: SPHERE & Regional funds
Evaluator expertise: Elementary science and
evaluation experience.
LRMDS Instruments:
Evaluation Areas
Intellectual Property Rights
Educational quality
Intellectual Property Rights checklist:

Verify ownership, Copyright, Conditions of
Use.
Educational Soundness General checklist.
Determine the general qualities in relation to
the purpose and whether to proceed with a
more extensive evaluation.
Educational Quality: Rating sheet for Print
resources. Determine suitability for use in
schools, accuracy and currency of all content
coverage, format, presentation & organization

Evaluation
Using the instrument

criteria

Verify ownership, Copyright, Conditions
of Use.
Finding: DepED copyright, but contains

some third party material.
What should be recommended?
Educational Soundness General

checklist. Determine the general
qualities in relation to the purpose and
whether to proceed with a more
extensive evaluation.
Finding: Identified some content

inaccuracies in Criteria 1 and 5.
Educational Quality: Rating sheet for

Print resources.
Determine suitability for use in schools,
accuracy and currency of all content
coverage, format, presentation &
organization
Finding: Passed all factors but reported

errors of fact, grammatical,
typographical

Evaluation
Using the instrument criteria

DepED copyright, but contains some
Verify ownership, Copyright, Conditions of Use. acknowledged third party material (diagrams,
pictures, excerpts of text).
Recommendation: Reproduce only in existing
format.
Educational Soundness General checklist.
Determine the general qualities in relation to
the purpose and whether to proceed with a
more extensive evaluation.
Identified some content inaccuracies in
Educational Quality: Rating sheet for Print

resources. Determine suitability for use in
schools, accuracy and currency of all content
coverage, format, presentation & organization
Passed all factors but reported errors of fact,

grammatical, typographical
Recommend: All errors are fixed, digitize and
reproduce and distribute.
Criteria 1 and 5.*

Recommendation: Do not distribute or
reproduce in current format. If the content
errors are not in 3rd party material then
conduct Ed Quality review, do not reproduce
unless all content inaccuracies have been
fixed.
* If commercial resource contains errors or inaccuracies then report errors but do not
proceed with further reviews.
Activity 4
What is the Request Brief?

What purpose does it serve?
Who prepares the request brief/
What are the major parts or components
of the request brief?
65
Request Brief
A Request Brief is a written explanation given to the development and production
team by a requesting party or client to
describe the specific service required and
the general design requirements of the
resource to be developed or produced.
66
Request Brief
The purpose of this document is to describe
in detail the service required and the
design requirements of a client/user for an
existing resource that is to be reproduced,
digitized, or redeveloped and for the
design of new resources.
67
Prepare a Request Brief

Instructions:
In pairs/or triads prepare the request brief for the
materials you have evaluated.
All the resources evaluated in all the streams of
evaluation must have a corresponding Request Brief.
After the pairs have completed the Request Brief, they
should exchange Request Briefs with another pair.
The task of the pair or triad at this stage is to REVIEW
the completed Request Brief of their peers for the
following:
68
Check the Request Brief

Completeness
Correctness
Clarity
Exchange comments, feedback on each
others Request Briefs.
Make the necessary adjustments in the
briefs.
Compile all the accomplished instruments
and request briefs
69
Processing
Ask the participants to probe the following:
Is the purpose for which a Request Brief is
prepared understood?
What areas/parts of the Request Brief did the
participants have difficulty with?
Determine whether the difficulty can be
addressed by presenting the materials for the
second time
70
71
Whatever the
cost of our
libraries, the price
is cheap
compared to that
of an ignorant
nation.
Walter Cronkite
72
73

Evaluation of Resources

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Evaluation of Resources

Uploaded by

Copyright:

Available Formats

LRMDS

Assessment and Evaluation

LRMDS Assessment and Evaluation: Evaluation of Resources

lrmds pub 1.pub

LRMDS Assessment and Evaluation: Evaluation of Resources

This session is intended to provide

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

Introduction to the concepts of Evaluation,

LRMDS Assessment and Evaluation: Evaluation of Resources

Introduction to the concepts of Evaluation,

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

Evaluation, Standards and

LRMDS Assessment and Evaluation: Evaluation of Resources

Evaluation, Standards and

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and

LRMDS Assessment and

LRMDS Assessment and Evaluation: Evaluation of Resources

Jeopardy/ Matching Game

Geospatial, physical and intellectual ability,

Quick Processing of the activity

LRMDS Assessment and Evaluation: Evaluation of Resources

Introduction to QA and Criteria

LRMDS Assessment and Evaluation: Evaluation of Resources

What is Quality Assurance?

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

Relevant to the needs and expectations of the users;

LRMDS Assessment and Evaluation: Evaluation of Resources

A process of continuous improvement

Measures & Requirements

There is active and integrated involvement of LRMDS users at all

LRMDS product and service delivery is informed by user

LRMDS inputs, processes and outputs conform to agreed quality

Under the microscope:

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment & Evaluation QA

LRMDS Assessment and Evaluation: Evaluation of Resources

Instruments used and their purposes

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

Why do we need specifications?

LRMDS Assessment and Evaluation: Evaluation of Resources

curriculum, subject matter, skills and

geospatial, physical and intellectual

format, file size, authoring software,

author, contributors, copyrights, access

Storage & distribution version control for management and

file types, formats for reproduction and

LRMDS Assessment and Evaluation: Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources

Criteria & Indicators Examples

Content stimulates and promotes critical

4. More than 80% of the content

Graphics/colors/sounds are used for the