Formative Evaluation:
What, Why, When, and How
Abstract
This paper is going to describe the formative evaluation by comparing two
books of Tessmer and Flagg. One can employ Tessmer stages of doing
formative evaluation (experts reviews, one-to-one, small group, and field test)
in instructional design or Flaggs steps (pre-production, production, and
implementation) in instructional technologies. This paper recommends
Tessmers book to be used for the beginners because its detail explanation of
how to do, plan, collect data, and analyze it.
A. Background
Having no methods of its own, evaluation has always borrowed strategies from all the social
sciences (Krathwohl, 1997). Evaluation can evaluate almost anything, such as a person, a
curriculum, a student, a process, a product, or a program. Every evaluation conducted by
some experts has its own name such as personal evaluation, program evaluation, program
evaluation, or teaching evaluation. Evaluation also differs from research not by its methods
but by other aspects, such as decision-driven (to failitate making decision) and utilization
(usefulness of the process). The intent of evaluation is to reduce the uncertainty and to
provide an imformation-rich decision making environment. Comparing to research,
evaluation gives a better information basis for action (decision).
Evaluation is variously conceived as a tool for more effective program management or means
of empowerment for those affected programs, or an effort to be responsive to the concerns of
stakeholders, or directed by a highly competent professional opinion, or a means to
conclusions and recommendation, or a process of negotiation among stakeholders to produce
an agenda for further negotiation. Some experts see evaluation as the responsibility of
connoissearial judgements by an areas experts, best be done as naturalistic descriptive
qualitative research or embedded in measurement and experimentation (Krathwohl, 1997).
Evaluation is oriented primarily toward gathering information that will facilitate improving a
person, a curriculum, a student, a process, a program or a product (formative) or that will help
determining its value or worth (summative). Many experts have analyzed the difference
between formative and summative evaluation. Markle (1989, in Tessmer, 1993) mentioned
that summative evaluation is an evaluation to prove but formative evaluation is an evaluation
to improve the programs or the products. Baker and Alkin (1973, in Tessmer, 1993) stated
that the difference between the two evaluations as between evaluation for validation
(summative) versus evaluation for revision (formative). Schriven (1991, in Krathwohl, 1997)
quoted Robert Stake on the formative and summative distinction, "When the cook tastes the
soup, that is formative evaluation; when the guest tastes it, that is summative evaluation".
This article is going to describe thoroughly formative evaluation as a tool of improving
instructional programs, products, and materials. By comparing two books of formative
evaluation, i.e. "Planning and conducting formative evaluation" by Martin Tessmer (1993)
and "Formative evaluation for educational technologies" by Barbara N. Flagg (1990), this
paper will summarize and analyze the following questions:
1. What is formative evaluation?
2. When is evaluation conducted, why and why not?
3. What stages of formative evaluation are there?
4. How to plan formative evaluation?
5. What are the remarks and critical issues in conducting formative evaluation?
This paper will give recommendation for using the two formative evaluation books. This
article will focus on answering those questions and describe the strengths and weaknesses by
analyzing the books critically and by giving opinions and reasons whether the books valuable
or useable.
A. Definition
Tessmer (1993) explicitly defined formative evaluation as a judgement (of the
strengths and weaknesses of instruction in its developing stages) for purposes of
revising the instruction to improve its effectiveness and appeal. Defining solely that
evaluation is a process of gathering data to determine the worth or value of
instruction, of its strengths and weaknesses. The evaluation is conducted by collecting
data about the instruction from variety of sources, using a variety of data gathering
methods and tools. The readers should understand that the process of gathering data is
very important since the formative evaluation is a judgement for improving the
effectiveness of the instruction (products, programs, or materials).
Flagg (1990) does not give an explicit definition of formative evaluation. She refers
formative evaluation as the process of gathering information to advise design,
production, and implementation. Discussing three case studies ("Sesame Street", "The
Business Disc", and "Puppet Theater"), the book explained the phases (design,
production, and implementation) of formative evaluation in each case. Explicitly the
book also mentioned that formative evaluation is valuable in decision-making process
during the design of computer software and videodisc, is working in production
settings, and is facilitating decision in implementation process.
These two definitions describe that formative evaluation is a process of collecting data
to be used to judge the strengths and weaknesses of instructional in order to revise and
improve the programs, products, and materials. This judgement is a guideline for the
researcher to improve the quality, effectiveness and efficiency of the programs,
products, and materials. It also can be used to make decision whether the programs,
products, and materials should be continued or cancelled, revised or changed,
improved or destroyed. Both books consider formative evaluation as an important step
and one should understand that continuation of the programs, products, and materials
depends on the result of formative evaluation.
Scriven (1967, in Tessmer, 1993 and Flagg, 1990) attached the name "formative
evaluation" to a revision process that referred to an outcome evaluation of an
intermediate stage in the development of the teaching instrument. Using the same
name given by Scriven, Tessmer (1993) mentioned other names of formative
evaluation from other experts, i.e. "try out", "developmental testing", "pilot study",
"formative assessment", "dry run", "alpha/beta testing", "quality control", and "learner
verification and revision". Tessmer preferred to use "quality control" as formative
evaluation but since "quality control" does not describe and represent the actual
meaning of formative evaluation and the actual target who is going to judge the
effectiveness and quality of the products, "learner verification and revision" is a better
name for formative evaluation. But what is in a name? For formative evaluation, the
process of conducting it is the most important thing to be planned and conducted
thoroughly.
Flagg (1990) gave no specific name for formative evaluation and mentioned no reason
why the name is included in each phase of evaluation, such as pre-production
formative evaluation, production formative evaluation, and implementation
formative evaluation. The names are referred to the collection of information to
guide decisions during the design, production, and implementation phase respectively.
B. When, why and why not
Flagg (1990) mentioned the only reason for performing formative evaluation is that to
inform the decision-making process during the design, production and implementation
process. To understand the content, attitudes toward the content, interests in the
content, and learners experience with the medium in design phase, to reduce
expensive mistakes and improve user friendliness in production phase, to restructure
the products for different settings in implementation phase are the main reasons why
formative evaluation needed. But particularly, formative evaluation is warranted for
the novice designers, for the implementation of new content, for the application of the
new technologies, for the different target learners, for the unfamiliar strategies, for the
accurateness of critical performance, for the large quantity of dissemination, and for
the little chance of revisions given (Tessmer, 1993).
Considering the importance of formative evaluation and by analyzing several studies,
Tessmer stated that there were three main results of doing formative evaluation in
instruction. First, using formative evaluation in all types of instruction (computerbased, simulation, games, texts, and multi media) can improve the learning
effectiveness of materials. Second, even though there is not enough evidence of
whether the instruction is more interesting or motivating, formative evaluation can be
used to obtain criticism and suggestions on interest or motivation of instruction to its
users. Third, since practitioners used some types of formative evaluation (no specific
names) in their projects, formative evaluation appears to be part of the "real world" of
instructional design.
By considering and explaining three case studies, Flagg (1990) perform the need of
formative evaluation to inform the decision-making process during their design,
production, and implementation stages of educational program with the purpose of
improving the program. The Sesame Street staff conduct research on childrens
conceptual understanding of death in order to provide information useful in scripting
the program. In production phase, user observation gave producers of The Business
Disc feedback to improve user friendliness before the instructional program reached a
stage where changes were cost prohibitive. Formative evaluation gives feedback to
developer of Puppet Theater to reconfigure a program for different context and users.
They reworked the program and added tools in response to formative data from other
users.
Eventhough formative evaluation is frequently used by practitioners, most of
organizations did not accept the formative evaluation as a part of their program. They
do not understand the purpose or utility of formative evaluation because they think
that formative evaluation is only for the finishing product evaluation, for
incompetent/inexperience designers, and for the insufficient personal evaluators.
Flagg also mentioned that there are six reasons why formative evaluation is not given
in the development of educational materials in electronic technologies. The major
excuses are ones of time (under pressure to produce by certain deadlines), money
(small percentage of production budget), human nature (constraint on creativity),
unmet expectations (unrealistic expectations), measurement difficulties (difficult to
measure long term objectives), and lack of knowledge (unaware of the philosophy and
methods). In addition, the formative evaluation is not worth value if those in control
of the project disagree with the philosophy of it, if developers can not agree on the
goals of the program and the intended audience, and if there is no chance for change.
C. Stages
Tessmer (1993) suggested four classically recognized types of formative evaluation:
1. Expert review: experts reviews the instruction with or without the evaluator
2. One-to-one: one learner at a time reviews the instructional with evaluator and give
comments upon it
3. Small-group: the evaluator tries out the instruction with a group of learners and
records their performances and comments
4. Field test: the evaluator observes the instruction being tried out in a realistic situation
with a group of learners.
In order to understand the concept of formative evaluation, Tessmer drew two figures of the
stages on formative evaluation. The figure below represents the conclusion of both figures.
Within few explanation about how to apply self-evaluation, Tessmer suggested to conduct
expert reviews and one-to-one together after self-evaluation, revise the instruction, conduct a
small group, revise the instruction, hold the field test and revise and improve the instruction
for the last time. One can use variation of those types applied in the four steps such as expert
panels, (team of experts and evaluator), two-to-one (two learners review the instruction with
the evaluator), and rapid prototyping (immediate field-test evaluation).
Unfortunately, Tessmer did not mention anything about how to or whether we can combine
those types in each step of evaluation plan. Another problem that Tessmer did not give
explanation is that when we conduct expert reviews and one-to-one together, what should be
done to revise the prototype if there is no agreement between the expert and one-to-one
evaluation about any factors/aspects of the prototype. In order to reduce this kind of
confusion, this paper suggests to conduct each step carefully, one step by one step.
Nevertheless whenever there is an opportunity to go back to previous step (for instance: time,
money, and manpower), one can do so. The book of Tessmer did not mention this step. He
assumed that whenever one can follow the steps precisely by considering the suggestions in
the book thoroughly than one will not have any difficulties or make unreasonable errors in
doing formative evaluation. This is the reason why he mentions that doing formative
evaluation depends on the thoroughness of the plan. Another possible way of combining the
steps of Tessmer is that to decide earlier what kind of information one wants to gain and in
what step or from whom the information should be gained. Designing this type of
information would be advantageous for combine the steps together as long as there have the
availability of the resources (time, money, and manpower), the capability of the resources to
be used effectively and efficiently, and the accomplishment of the implementation of the
steps.
Flagg (1990) explains the stages of formative evaluation by considering the development of
the program and the evaluation steps itself. The following is the description of the formative
evaluation for television, software, and videodisc.
Program Development
Evaluation
Planning
Needs Assessment
Design
Pre-production Formative Evaluation
Production
Production Formative Evaluation
Implementation
Implementation Formative Evaluation
By considering other experts, Flagg (1993) describes that the first evaluation phase is need
assessment or front-end analysis where to obtain the reason of the program, content, and
feasibility of the delivery system. The second phase (the pre-production formative
evaluation) conceptualize the planning phase to guide the pre-production of the program
called preliminary scripts. The third phase (production formative evaluation) revise the early
program versions with the target group. The implementation formative evaluation phase
operates the target learners in the environment for which it was designed. These phases can
be found and described explicitly in chapter 4 7, except chapter 8 that is only for
implementation formative evaluation.
These two similar but different stages of formative evaluation describe each step in detail but
in different purpose. Tessmer explained each step in purpose of conducting formative
evaluation on instructional design in general but Flagg described it on the purpose on
evaluating the instructional technology format. In general these two stages used similar
approach, questions, attention, and measurement tools used in conducting formative
evaluation.
A. Plan
Tessmer suggested that the formative evaluation process can be done by the following
steps, i.e. determining the goals and the resources for and constraints upon the
evaluation, conducting task analysis, describing the learning environment,
determining the media characteristics, outlining the information sought, selecting data
gathering methods and tools, and implementing the evaluation. Tessmer described
each step by not only giving questions that should be considered in doing formative
evaluation, but also constructing some answers for them. For instance, the answer for
"what do you want to find out from the evaluation?" are learning effectiveness,
learner interest/motivation, content quality, technical quality, and implementability.
In conducting formative evaluation in instructional technologies, Flagg gave four
criteria, i.e. usability (usable for decision making), practicality (gain answer within
the time limits & money available), importance (relevant to the objectives and
situation), and uncertainty (uncertain to those expectation to be answered). Methods
to be used in the formative evaluation can be hypothetico-deductive paradigm (topdown approach or theory-based hypothesis) to confirm or explore causal relationship
between or among variables. Another method is inductive paradigm (bottom-up
approach) which begins with collection of qualitative and quantitative data directed
by the evaluation question.
Giving much attention on each step of formative evaluation (22 pages for expert
reviews, 29 for one-to-one, 35 for small group, and 16 for field test), Tessmer
described in detail how to conduct each step of formative evaluation. In expert
reviews is conducted to evaluate the clarity of objectives and content, practicality
(technical quality) of the prototype, and validity of the materials by using
connoisseurial or expert judgements reviews. Considering many aspects, Tessmer
stated that in expert reviews there are many types of experts and many types of
questions that should or should not be asked of each one, depending upon each
experts strengths and the goals of the evaluation. To be more thorough in doing
expert reviews, Tessmer gave an example at the end of each chapter.
The advantages of one-to-one are its interactive and highly productive, easy, quick
and inexpensive, its sources of revision information, the clearness of instruction and
direction, the completeness of the instruction, and the adequate quality of the
materials. Small group evaluation evaluates the effectiveness, appeal and
implementability of the approach. It also gives the study many advantages such as
inexpensive, easy to conduct, more accurate measures of teachers performance, and
more improvement in the instruction prototype. Field test evaluation is conducted to
describe the teacher acceptance, implementability, and organizational acceptance of
the prototype approach. It can be used to confirm the revisions made in previous
formative evaluations, to generate final revision suggestions, and to investigate the
effectiveness of the prototype instruction and also to gain the polished version of the
products and programs. Tessmer also provided an example in each chapter of one-toone, small group, and field test.
By considering other experts, Flagg (1993) describes that the first evaluation phase is
need assessment or front-end analysis to obtain the reason of the need of the program,
the content, and the feasibility of the delivery system. The gathering data information
entails reviews of existing studies, test and curricula, experts reviews, and
measurement of target audience characteristics. The second phase is called the preproduction formative evaluation where the conceptualization of the planning phase is
guided the pre-production of the program called preliminary scripts or writers
notebook. In electronic learning project researchers include the target audience and
teachers in the process of making design decisions about content, objectives, and
production formats but the expert reviews (content and design) will be used to guide
the creativity of the designers and reduce uncertainty of some critical decisions. The
third phase is called production formative evaluation where the program is revised
considering the feedback from tryouts of early program versions with the target
group. Information of user-friendliness, comprehensibility, appeal, and persuasiveness
can give the production team confidence of success in their revisions and decisions.
The subject matter specialists, the designers, and other experts can work together to
improve the versions. The implementation formative evaluation phase is concerned
with how the program operates with target learners in the environment for which it
was designed. Field-testing helps designers see how program managers will really use
their final products with target learners and the feedback aids the development of the
support materials and the future programs. This phase differs from the summative
evaluation since the later measures the learners who have not been yet exposed by the
program.
Meanwhile Tessmer gave specific method used in doing formative evaluation along
with guidelines for collecting data and analyzing them in each step, Flagg described
many alternatives of measurements, such as self-report (respondent responds on
questionnaires of interviews), observation (renders an obtrusive and objective record),
tests, records (collection of data). Even though there was not elaboration of the "how"
question to conduct these measures, Flagg, by quoting Mielke (1973) stated that
superiority or inferiority of a research method cannot be established an inherent
quality, but it can be established in terms of performance in answering questions.
In order to design sophisticated software, the user-friendliness (accessibility,
responsibility, flexibility, and memory), the reception (attention, appeal and
excitement), and the outcome effectiveness (motor skills, cognitive abilities, or
attitude that the learners have to learn). The methods such as observation and
mechanical recording devices will yield valuable information relevant to the
accessibility, responsiveness, and flexibility of computer-based educational programs
as well as self-report (think aloud, escorted trial, and diary) and tests. Giving much
elaboration only on visual attention, the book elaborates each of these self-report
methods especially in program evaluation analysis computer (PEAK system). The
book also did not explain why only self-report methods are suitable for measuring
these variables.
B. Miscellaneous
In general these two books gave different names but similar stages in doing formative
evaluation. Tessmer gave attention on the general instructional design and Flagg
focused on the instructional technologies. Tessmer described each stage in detail and
gave an example in each stage, but Flagg gave five examples and described each stage
in each example explicitly. The following table describes more general difference
between the books. The statistic of the two books:
Characteristics
Flaggs Book
Tessmers Book
Pages
259
159
Chapters
12
Examples
5 ex. in the chapter 4-8
1 ex. In the chapter 36
3 ex. in
the
chapter
2
ISBN
0-8058-01278
0-7494-08014
Fl. 30
16.95
Lawrence Erlbaum
Kogan Page
Year of issues
1990
1993
Index of subject
Exists
Exists
Index of authors
Exists
No
Glossary
No
Exists
Reference
In each chapter
At the end
Definition
Implicitly
Explicitly
Summary
3 in each chapters except chapter 2,
4-8
1 in each chapter
History of
Formative
Evaluation
Exists
Exists
Price
Publisher
C. Recommendation
The first weakness of Flaggs book is that the structure of the content of the book itself. Since
there is no suggestion in using or how to read the book, one should read the introduction
(chapter 1) carefully in order to have big view of the content of the book. Since the book is
meant for students, practitioners, researchers, designers, and developers in order to give them
in-depth process of formative evaluation by providing many examples, they should spent
more time in the first chapter and go to the summary of each chapter (chapter 5, 6 & 8 have
no summary) then begin to read the book in detail. The book is difficult for the beginners to
understand but is easy to read by the experienced person. The reason behind this could be
because there was no formative evaluation conducted for the book before the book is issued.
For whom who are beginners, it is very useful and understandable to read Tessmers book
because it is very structured. It is not because of only 6 chapters in the book but it could be
because the expert review and the field test have been conducted before it is issued. Even
without reading "A Note to my readers about this Book", one can understand what, how,
when, why, and why not to do formative evaluation in instructional design. The book can be
usefully applied not only on educational instructional design but also in instructional
technology. The book looks like a "cook book" where one can read, use, plan, create, and
implement the formative evaluation by considering all the steps, strategies, questions, and
suggestions included in the book.
For conducting a developmental research, these books can be useful resources. The books
described in detail how to do, how to collect data, and how to analyze them for improving
instructional design or instructional technologies. Using the books as guidelines for
conducting formative evaluation is advantageous but the most important thing is that the
individual should have a good and strong attitude toward conducting evaluation. It is not only
because it will motivate person to do it but also because it will help the person to understand,
collect, compare, and analyze the data virtuously and thoroughly in order to gain the best
result of the phenomena in the evaluation. Formative evaluation needs a fair, truth, and honest
attitude of a person in conducting it. Otherwise the result is only GIGO (Garbage In
Garbage Out).
References:
Flagg, Barbara N. 1990. Formative evaluation for educational technologies.
Hillsdale, New Jersey: Lawrence Erlbaum Associates, Publisher.
Krathwohl, David R. 1998. Methods of educational & social science research:
An integrated approach. New York: Addison-Wesley Longman, Inc.
Tessmer, Martin. 1993. Planning and conducting formative evaluation.
London: Kogan Page Limited.
