THE DELPHI METHOD

By H. Murat GÜNAYDIN, Ph.D.

Definition and Historical Background. The objective of most Delphi

applications is the reliable and creative exploration of ideas or the production of suitable
information for decision making. The Delphi Method is based on a structured process
for collecting and distilling knowledge from a group of experts by means of a series of
questionnaires interspersed with controlled opinion feedback (Adler and Ziglio, 1996).
According to Helmer (1977) Delphi represents a useful communication device among a
group of experts and thus facilitates the formation of a group judgement. Wissema
(1982) underlines the importance of the Delphi Method as a monovariable exploration
technique for technology forecasting. He further states that the Delphi method has been
developed in order to make discussion between experts possible without permitting a
certain social interactive behavior as happens during a normal group discussion and
hampers opinion forming. Baldwin (1975) asserts that lacking full scientific
knowledge, decision-makers have to rely on their own intuition or on expert opinion.
The Delphi method has been widely used to generate forecasts in technology, education,
and other fields (Cornish, 1977).

The technology forecasting studies which eventually led to the development of the
Delphi method started in 1944. At that time General Arnold asked Theodor von
Karman to prepare a forecast of future technological capabilities that might be of
interest to the military (Cornish, 1977). Arnold got the Douglas Aircraft company to
establish in 1946 a Project RAND (an acronym for Research and Development) to study
the “broad subject of inter-continental warfare other then surface.” In 1959 Helmer and
fellow RAND researcher Rescher published a paper on “The Epistemology of the
Inexact Sciences,” which provide a philosophical base for forecasting (Fowles, 1978).
The paper argued that in fields that have not yet developed to the point of having
scientific laws, the testimony of experts is permissible. The problem is how to use this
testimony and, specifically, how to combine the testimony of a number of experts into a
single useful statement. The Delphi method recognizes human judgement as legitimate
and useful inputs in generating forecasts. Single experts sometimes suffer biases; group
meetings suffer from “follow the leader” tendencies and reluctance to abandon
previously stated opinions (Gatewood and Gatewood, 1983, Fowles, 1978). In order to
overcome these shortcomings the basic notion of the Delphi method, theoretical
assumptions and methodological procedures developed in the 1950s and 1960s at the
RAND Corporation. Forecasts about various aspect of the future are often derived
through the collation of expert judgement. Dalkey and Helmer developed the method
for the collection of judgement for such studies (Gordon and Hayward, 1968).

Fowles (1978) asserts that the word Delphi refers to the hallowed site of the most
revered oracle in ancient Greece. Forecasts and advices from gods were sought through
intermediaries at this oracle. However Dalkey (1968) states that the name “Delphi” was
never a term with which either Helmer or Dalkey (the founders of the method) were
particularly happy. Dalkey (1968) acknowledged that it was rather unfortunate that the
set of procedures developed at the RAND Corporation, and designed to improve
methods of forecasting, came to be known as “Delphi”. He argued that the term implies
“something oracular, something smacking a little of the occult”, whereas, as a matter of
fact, precisely the opposite is involved; it is primarily concerned with making the best
you can of a less than perfect kind of information.

One of the very first applications of the Delphi method carried out at the RAND
Corporation is illustrated in the publication by Gordon and Helmer (1964). Its aim was
to assess the direction of long-range trends, with special emphasis on science and
technology, and their probable effects on society. The study covered six topics:
scientific breakthroughs; population control; automation; space progress; war
prevention; weapon systems (Gordon and Helmer, 1968). The first Delphi applications
were in the area of technological forecasting and aimed to forecast likely inventions,
new technologies and the social and economic impact of technological change (Adler
and Ziglio, 1996). In terms of technology forecasting, Levary and Han (1995) state the
objective of the Delphi method as to combine expert opinions concerning the likelihood
of realizing the proposed technology as well as expert opinions concerning the expected
development time into a single position. When the Delphi method was first applied to
long-range forecasting, potential future events were considered one at a time as though
they were to take place in isolation from one another. Later on, the notion of cross
impacts was introduced to overcome the shortcomings of this simplistic approach
(Helmer, 1977).

According to Wissema (1982), unfortunately the Delphi method is also sometimes

used for a normal inquiry among a number of experts. Delphi has found its way into
industry, government, and finally, academe. It has simultaneously expanded beyond
technological forecasting (Fowles, 1978). Since the 1950s several research studies have
used the Delphi method, particularly in public health issues (such as, policies for drug
use reduction and prevention of AIDS/HIV) and education areas (Adler and Ziglio,
1996; Cornish, 1977).
 The Basics of the Delphi Method. The Delphi method is an exercise in group
communication among a panel of geographically dispersed experts (Adler and Ziglio,
1996). The technique allows experts to deal systematically with a complex problem or
task. The essence of the technique is fairly straightforward. It comprises a series of
questionnaires sent either by mail or via computerized systems, to a pre-selected group
of experts. These questionnaires are designed to elicit and develop individual responses
to the problems posed and to enable the experts to refine their views as the group’s work
progresses in accordance with the assigned task. The main point behind the Delphi
method is to overcome the disadvantages of conventional committee action. According
to Fowles (1978) anonymity, controlled feedback, and statistical response characterize
Delphi. The group interaction in Delphi is anonymous, in the sense that comments,
forecasts, and the like are not identified as to their originator but are presented to the
group in such a way as to suppress any identification.

In the original Delphi process, the key elements were (1) structuring of information
flow, (2) feedback to the participants, and (3) anonymity for the participants. Clearly,
these characteristics may offer distinct advantages over the conventional face-to-face
conference as a communication tool. The interactions among panel members are
controlled by a panel director or monitor who filters out material not related to the
purpose of the group (Martino, 1978). The usual problems of group dynamics are thus
completely bypassed. Fowles (1978) describes the following ten steps for the Delphi
method:

1. Formation of a team to undertake and monitor a Delphi on a given subject.

2. Selection of one or more panels to participate in the exercise. Customarily, the
panelists are experts in the area to be investigated.
3. Development of the first round Delphi questionnaire
4. Testing the questionnaire for proper wording (e.g., ambiguities, vagueness)
5. Transmission of the first questionnaires to the panelists
6. Analysis of the first round responses
7. Preparation of the second round questionnaires (and possible testing)
8. Transmission of the second round questionnaires to the panelists
9. Analysis of the second round responses (Steps 7 to 9 are reiterated as long as desired
or necessary to achieve stability in the results.)
10. Preparation of a report by the analysis team to present the conclusions of the exercise

Delbecq et al., (1975) argue that the most important issue in this process is the
understanding of the aim of the Delphi exercise by all participants. Otherwise the
panelists may answer inappropriately or become frustrated and lose interest. The
respondents to the questionnaire should be well informed in the appropriate area (Hanson
and Ramani, 1988) but the literature (Armstrong, 1978; Welty, 1972) suggest that a high
degree of expertise is not necessary. The minimum number of participants to ensure a
good group performance is somewhat dependent on the study design. Experiments by
Brockhoff (1975) suggest that under ideal circumstances, groups as small as four can
perform well.
 Before deciding whether or not the Delphi method should be used, it is very important
to consider thoroughly the context within which the method is to be applied (Delbecq et
al. 1975). A number of questions need to be asked before making the decision of
selecting or ruling out the Delphi technique (Adler and Ziglio, 1996):

• What kind of group communication process is desirable in order to explore the

problem at hand?
• Who are the people with expertise on the problem and where are they located?
• What are the alternative techniques available and what results can reasonably be
expected from their application?

Only when the above questions are answered can one decide whether the Delphi
method is appropriate to the context in which it will be applied. Adler and Ziglio (1996)
further claim that failure to address the above questions may lead to inappropriate
applications of Delphi and discredit the whole creative effort.

The outcome of a Delphi sequence is nothing but opinion. The results of the sequence
are only as valid as the opinions of the experts who made up the panel (Martino, 1978).
The panel viewpoint is summarized statistically rather than in terms of a majority vote.

The Delphi method has got criticism as well as support. The most extensive critique
of the Delphi method was made by Sackman (1974) who criticizes the method as being
unscientific and Armstrong (1978) who has written critically of its accuracy. Martino
(1978) underlines the fact that Delphi is a method of last resort in dealing with extremely
complex problems for which there are no adequate models. Helmer (1977) states that
sometimes reliance on intuitive judgement is not just a temporary expedient but in fact a
mandatory requirement. Makridakis and Wheelright (1978) summarize the general
complaints against the Delphi method in terms of (a) a low level reliability of judgements
among experts and therefore dependency of forecasts on the particular judges selected;
(b) the sensitivity of results to ambiguity in the questionnaire that is used for data
collection in each round; and (c) the difficulty in assessing the degree of expertise
incorporated into the forecast. Martino (1978) lists major concerns about the Delphi
method:

• Discounting the future: Future (and past) happenings are not as important as the
current ones, therefore one may have a tendency to discount the future events.
• The simplification urge: Experts tend to judge the future of events in isolation from
other developments. A holistic view of future events where change has had a
pervasive influence cannot be visualized easily. At this point cross-impact analysis is
of some help.
• Illusory expertise: some of the experts may be poor forecasters. The expert tends to
be a specialist and thus views the forecast in a setting which is not the most
appropriate one.
• Sloppy execution: there are many ways to do a poor job. Execution of the Delphi
process may loose the required attention easily.
• Format bias: it should be recognized that the format of the questionnaire may be
unsuitable to some potential societal participants.
• Manipulation of Delphi: The responses can be altered by the monitors in the hope of
moving the next round responses in a desired direction.

Goldschmidt (1975) agrees that there have been many poorly conducted Delphi
projects. However, he warns that it is a fundamental mistake to equate the applications of
the Delphi method with the Delphi method itself, as too many critics do. There is, in fact,
an important conceptual distinction between evaluating a technique and evaluating an
application of a technique.

On the other hand there have been several studies (Ament, 1970; Wissema, 1982;
Helmer, 1983) supporting the Delphi method. A study conducted by Milkovich et al.
(1972) reports the use of the Delphi method in manpower forecasting. The results of the
comparison indicated high agreement between the Delphi estimate and the actual number
hired and less agreement between quantitative forecasts and the number hired. Another
study by Basu and Schroeder (1977) reports similar results in a general forecasting
problem. They compared Delphi forecasts of five-year sales with both unstructured,
subjective forecasts and quantitative forecasts that used regression analyses and
exponential smoothing. The Delphi forecasting consisted of three rounds using 23 key
organization members. When compared against actual sales for the first two years, errors
of 3-4% were reported for Delphi, 10-15% for the quantitative methods, and of
approximately 20% for the previously used unstructured, subjective forecasts.

In general, the Delphi method is useful in answering one, specific, single-dimension

question. There is less support for its use to determine complex forecasts concerning
multiple factors. Such complex model building is more appropriate for quantitative
models with Delphi results serving as inputs (Gatewood and Gatewood, 1983). This
point is supported by Gordon and Hayward (1968) who claim that the Delphi method,
based on the collation of expert judgement, suffers from the possibility that reactions
between forecasted items may not be fully considered. The need for the cross impact
matrix method of forecasting integrated with the Delphi method is pointed out by many
researchers (Gordon and Hayward, 1968; Gatewood and Gatewood, 1983; Adler and
Ziglio, 1996). An improvement in forecasting reliability over the Delphi method was
thought to be attainable by taking into consideration the possibility that the occurrence of
one event may cause an increase or decrease in the probability of occurrence of other
events included in the survey (Helmer, 1978). Therefore cross impact analysis has
developed as an extension of Delphi techniques.

BIBLIOGRAPHY

Adler, M., & Ziglio, E. (1996). Gazing into the oracle. Jessica Kingsley Publishers:
Bristol, PA.
Alarcon, L.F. (1992). Project performance modeling: a methodology for evaluating
project execution strategies. Ph.D. thesis, University of California, Berkeley.

Brochner, J. (1990). Impacts of information technology on the structure of construction.

Construction Management and Economics, 8, pp. 205-218.

Cornish, E., (1977). The study of the future. World Future Society: Washington, D.C.

Duval, A., Fontela, E., Gabus, A., (1975). Portraits of Complexity. (Ed. Baldwin, M.
M.) Battelle Memorial Institute: Columbus: Ohio.

Enzer, S., & Alter, S. (1978). Cross-impact analysis and classical probability: The
question of consistency. Futures, June 1978, 10(3), pp. 227-239.

Ewing, R.P. (1979). The uses of futurist techniques in issues management. Public
Relations Quarterly, 4, pp. 15-18.

Forgionne, G.A. (1997). HADTS: a decision technology system to support army

housing management. European Journal of Operational Research, pp. 363-379.

Fowles, J., (1978). Handbook of futures research. Greenwood Press: Connecticut.

Gatewood, R.D. and Gatewood, E.J., (1983). The use of expert data in human resource
planning: guidelines from strategic forecasting. Human Resource Planning, 5(1). Pp.83-
94.

Gordon, T.J. and Hayward, H. (1968). Initial experiments with the cross-impact matrix
method of forecasting. Futures, December, 1968, 1(2), pp. 100-116.

Hanson, W.H., and Ramani, N. (1988). Technology forecasting: a hydroelectric

company experience. Technology Management Publication, 1(3)., pp. 266-270.
Harvard Business School Press.

Helmer, O., (1977). Problems in futures research: Delphi and causal cross-impact
analysis. Futures, February 1977, pp. 17-31.

Helmer, O., (1981). Reassessment of cross-impact analysis. Futures, October 1981, pp.
389-400.

Huss, W.R. (1988). A move toward scenario analysis. International Journal of

Forecasting, 4, pp. 377-388.

Jensen, R.E. (1981). Scenario probability scaling: an eigenvector analysis of elicited

scenario odds ratios. Futures, December, 1981. pp. 489-498.
Jin, Y., Levitt, R.E., Kunz, J.C., and Christiansen, T.R. (1995). The virtual design team:
a computer simulation framework for studying organizational aspects of concurrent
design. Simulation, March, 1995, pp. 160-174.

Johnson, J.D. (1996). Information seeking: an organizational dilemma. Connecticut,

London: Quorum Books.

Kirkwood, C.W., and Pollock, S.M. (1982). Multiple attribute scenarios, bounded
probabilities, and threats of nuclear theft. Futures, December 1982, pp. 545-553.

Kwak, Y.H., (1997). A systematic approach to evaluate quantitative impacts of project

management. Ph.D. dissertation, University of California, Berkeley.

Levary, R.R. and Han, D. (1995). Choosing a technological forecasting method. IM,
pp. 14-18.

Novak, E., and Lorant, K., (1978). A method for the analysis of interrelationships
between mutually connected events: a cross-impact method. Technological Forecasting
and Social Change (12), pp.201-212.

Reuven, L., and Dongchui, H., (1995). Choosing a technological forecasting method.
Industrial Management, 37(1). pp. 14-22.

Sarin, R.K., (1978). A sequential approach to cross-impact analysis. Futures, February

1978, pp. 53-62.

Schuler, A., Thompson, W.A., Vertinsky, I., and Ziv, Y. (1991). Cross impact analysis
of technological innovation and development in the softwood lumber industry in Canada:
A structural modeling approach. IEEE Transactions on Engineering Management. 38(3),
pp. 224-236.

Wissema, J.G. (1982). Trends in technology forecasting. R & D Management, 12(1),

pp. 27-36.

Wissema, J.G., and Benes, J. (1980). A cross-impact case study: the Dutch construction
sector. Futures, October 1980. pp. 394-404.