Psychological Testing
& Assessment
Chapter 4
RELIABILITY
Things to be discussed:
Define what reliability is Discuss Spearman’s early studies
Describe the conceptualization of Know the different sources of
error error
Things to be discussed:
Discuss the reliability in behavioral
studies
Explain the different methods of
estimating reliability
Answer the question “how
reliable is reliable?”
Know how to calculate the
Discuss what to do about
reliability of a test
low reliability
What is reliability?
Reliability
It is the degree to
which a measurement
produces consistent
results.
Reliability coefficient
a measure of the proportion that indicates the ratio between
the true score variance and the observed score variance.
Spearman’s Early Studies
Charles Spearman
(1863 – 1945)
Conceptualization of Error
Measurement error
refers to all the factors associated with the process of measuring some
variable, other than the variable being measured.
Conceptualization of Error
Random error Systematic error
An error caused by unpredictable An error in measuring a variable
fluctuations of other variables in that is typically constant to what
the measurement process. is presumed to be the true value
of the variable being measured.
Sources of
Error Time Sampling
Item Sampling
Test Administration
Carryover effect
• performance in one condition is
Time affected by the condition that
precedes it
Sampling
Practice effect
• improvement due to repeated
practice
Item Sampling Variation among items within a
Item Sampling test as well as to variation
among items between tests.
Item Sampling
Test Administration
Room temperature, level of
lighting, the ventilation, and
surrounding noise, etc.
• Test-taker variables
• Examiner-related variables
Methods of Estimating Reliability
Test-Retest Method
Parallel Forms Method
Split-Half Method
Test-Retest Method
Test-Retest Method
an estimate of reliability
obtained by correlating
pairs of scores from the
same people on two
different administrations
of the same test.
Parallel Forms Method
Parallel Forms Method
an estimate of reliability
that compares two
equivalent forms of a test
that measure
the same attribute.
Split-Half Method
Split-Half Method
an estimate of reliability
obtained by correlating two
pairs of scores obtained odd-even system
from equivalent halves of a
single test administered
once.
Calculating
the reliability of a test
Correlation coefficient
• a mathematical index that
describes the direction and
magnitude of a relationship.
Pearson product moment
correlation
r
bbb
Test-Retest Method
Parallel Forms Method
r
Where:
n = number of pairs of scores
∑xy = sum of the products of paired scores
∑x = sum of x scores (scores on the 1st ad)
∑y = sum of y scores (scores on the 2nd ad)
∑x 2 = sum of squared x scores
∑y2 = sum of squared y scores
X Y x2 y2 xy
(1 administration)
st
(2 administration)
nd
43 38 1,849 1,444 1,634
32 41 1,024 1,681 1,312
41 44 1,681 1,936 1,804
46 45 2,116 2,025 2,070
45 40 2,025 1,600 1,800
32 36 1,024 1,296 1,152
48 48 2,304 2,304 2,304
36 34 1,296 1,156 1,224
40 39 1,600 1,521 1,560
50 49 2,500 2,401 2,450
∑x = 413 ∑y = 414 ∑x2 = 17,419 ∑y2 = 17,364 ∑xy = 17,310
N = 10
∑x = 413 ∑y = 414 ∑x2 = 17,419 ∑y2 = 17,364 ∑xy = 17,310
r
r
r
r
r
r r
Spearman-Brown Formula
Split-Half Method
Where:
rSB = reliability of the entire test
rhh = correlation between two halves
Student Total Score Odd (x) Even (y)
1 42 20 22
2 33 15 18
3 44 21 23
4 45 24 21
5 30 16 14
6 26 11 15
7 45 24 21
8 35 18 17
9 40 23 17
10 32 16 16
11 34 15 19
12 45 22 23
N = 12
x = 225 y = 226 x= 4,413 y = 4,364 xy = 4,334
r
r
r
r
r
r
r
rSB
rSB
rSB = reliability of the entire test
rhh = correlation between two halves
rSB
rSB
KR20 Formula
Split-Half Method
Where:
KR20 = the reliability estimate (r)
N = the number of items on a test
S 2 = the variance of the total score
p = the prop of the people getting each item correct
q = the prop of the people getting each item incorrect
pq = the sum of products of p and q
for each item on a test
Polychotomous
has more than 2 possible outcomes
(Spearman-Brown Formula, Coefficient Alpha)
Dichotomous
has only 2 possible outcomes (KR20 Formula)
Math Problems
1. 5+3 2. 7+2 3. 9+1 4. 6+3 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
1 1 1 1 1 1 1 1 1 1 1
2 1 0 0 1 0 0 1 1 0 1
3 1 0 1 0 0 1 1 1 1 0
4 1 0 1 1 1 0 0 1 0 0
5 0 0 0 0 0 1 1 0 1 1
6 0 1 1 1 1 1 1 1 1 1
7 0 1 1 1 1 1 1 1 1 1
8 0 0 1 1 0 1 1 0 1 0
9 0 1 1 1 1 1 1 1 1 1
10 0 0 1 1 0 1 0 1 1 1
11 0 0 1 1 0 0 0 0 0 1
12 1 1 0 0 0 1 0 0 1 1
13 1 1 1 1 1 1 1 1 1 1
14 0 1 1 1 0 0 0 0 1 0
15 0 1 1 1 1 1 1 1 1 1
Math Problems
1. 5+3 2. 7+2 3. 9+1 4. 6+3 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
1 1 1 1 1 1 1 1 1 1 1
2 1 0 0 1 0 0 1 1 0 1
3 1 0 1 0 0 1 1 1 1 0
4 1 0 1 1 1 0 0 1 0 0
5 0 0 0 0 0 1 1 0 1 1
6 0 1 1 1 1 1 1 1 1 1
7 0 1 1 1 1 1 1 1 1 1
8 0 0 1 1 0 1 1 0 1 0
9 0 1 1 1 1 1 1 1 1 1
10 0 0 1 1 0 1 0 1 1 1
11 0 0 1 1 0 0 0 0 0 1
12 1 1 0 0 0 1 0 0 1 1
13 1 1 1 1 1 1 1 1 1 1
14 0 1 1 1 0 0 0 0 1 0
15 0 1 1 1 1 1 1 1 1 1
# of 1s 6 8 12 12 7 11 10 10 12 11
Pro. 0.40 0.53 0.80 0.80 0.47 0.73 0.67 0.67 0.80 0.73
Math Problems
1. 5+3 2. 7+2 3. 9+1 4. 6+3 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
1 1 1 1 1 1 1 1 1 1 1
2 1 0 0 1 0 0 1 1 0 1
3 1 0 1 0 0 1 1 1 1 0
4 1 0 1 1 1 0 0 1 0 0
5 0 0 0 0 0 1 1 0 1 1
6 0 1 1 1 1 1 1 1 1 1
7 0 1 1 1 1 1 1 1 1 1
8 0 0 1 1 0 1 1 0 1 0
9 0 1 1 1 1 1 1 1 1 1
10 0 0 1 1 0 1 0 1 1 1
11 0 0 1 1 0 0 0 0 0 1
12 1 1 0 0 0 1 0 0 1 1
13 1 1 1 1 1 1 1 1 1 1
14 0 1 1 1 0 0 0 0 1 0
15 0 1 1 1 1 1 1 1 1 1
# of 0s 9 7 3 3 8 4 5 5 3 4
Pro. 0.60 0.47 0.20 0.20 0.53 0.27 0.33 0.33 0.20 0.27
Math Problems
1. 5+3 2. 7+2 3. 9+1 4. 6+3 5. 8+6 6. 7+5 7. 4+7 8. 9+2 9. 8+4 10. 5+6
1 1 1 1 1 1 1 1 1 1 1
2 1 0 0 1 0 0 1 1 0 1
3 1 0 1 0 0 1 1 1 1 0
4 1 0 1 1 1 0 0 1 0 0
5 0 0 0 0 0 1 1 0 1 1
6 0 1 1 1 1 1 1 1 1 1
7 0 1 1 1 1 1 1 1 1 1
8 0 0 1 1 0 1 1 0 1 0
9 0 1 1 1 1 1 1 1 1 1
10 0 0 1 1 0 1 0 1 1 1
11 0 0 1 1 0 0 0 0 0 1
12 1 1 0 0 0 1 0 0 1 1
13 1 1 1 1 1 1 1 1 1 1
14 0 1 1 1 0 0 0 0 1 0
15 0 1 1 1 1 1 1 1 1 1
pxq 0.24 0.25 0.16 0.16 0.25 0.20 0.22 0.22 0.16 0.20
pq = 2.05
Math Problems
1. 2. 7+2 3. 4. 5. 6. 7. 4+7 8. 9. 8+4 10. Total
5+3 9+1 6+3 8+6 7+5 9+2 5+6 Score
1 1 1 1 1 1 1 1 1 1 1 10
2 1 0 0 1 0 0 1 1 0 1 5
3 1 0 1 0 0 1 1 1 1 0 6
4 1 0 1 1 1 0 0 1 0 0 5
5 0 0 0 0 0 1 1 0 1 1 4
6 0 1 1 1 1 1 1 1 1 1 9
7 0 1 1 1 1 1 1 1 1 1 9
8 0 0 1 1 0 1 1 0 1 0 5
9 0 1 1 1 1 1 1 1 1 1 9
10 0 0 1 1 0 1 0 1 1 1 6
11 0 0 1 1 0 0 0 0 0 1 3
12 1 1 0 0 0 1 0 0 1 1 5
13 1 1 1 1 1 1 1 1 1 1 10
14 0 1 1 1 0 0 0 0 1 0 4
15 0 1 1 1 1 1 1 1 1 1 9
= 99
Ẋ = 9.9
Total
Score
10
5
6
5
4
CALCULATE THE 9
VARIANCE USING
9
5 S 2 = 5.57
EXCEL: 9
6
3
5
10
4
9
KR20
KR20
KR20
KR20
KR20
Coefficient Alpha
Split-Half Method
Where:
r = the reliability estimate
N = the number of items on a test
S 2 = the variance of the total score
S 2i = the variance of the individual
items
Students Item 1 Item 2 Item 3 Total
1 6 6 8 20
2 5 5 6 16
3 9 8 6 23
4 3 2 4 9
5 2 3 2 7
6 1 1 2 4
7 5 4 6 15
Variance: S 2i = 7.29 S 2i = 5.81 S 2i = 5.14 S 2 = 48.95
S 2 = 7.29 + 5.18 + 5.14 = 18.24
i
S 2 = 48.95
S 2i = 18.24
r
r
r
r r
RELIABILITY
IN BEHAVIORAL OBSERVATION
STUDIES
inter-scorer reliability
Source of error: the degree of agreement or consistency
Observer differences between two or more observers regarding a
particular measure
Kappa statistics
How reliable is reliable?
reliability estimates in the range of .70 and .80 are
good enough for most purposes in basic research
in clinical settings, a test with a reliability of .90
might not be good enough, evaluators should
attempt to find a test with a reliability greater than .
95
What to do about low reliability?
Increase the number of items
Factor analysis
Chapter 5
VALIDITY
Things to be discussed:
Define what validity is Explain the 3 types of
evidences
Discuss the booklet Discuss the aspects of
published by a joint validity
committee
Things to be discussed:
Discuss validity coefficient Discuss the relationship
between reliability and
validity
What is validity?
Validity
the agreement between
a test score or measure
and the quality it is
believed to measure
American Educational Research Association
(AERA), American Psychological Association
(APA), and the National Council on
Measurement in Education (NCME)
Standards for Educational and
Psychological Testing
3 TYPES OF
EVIDENCES
Content-Related Evidence for Validity
Criterion-Related Evidence for Validity
Construct-Related Evidence for
Validity
Aspects of validity
Content-Related Evidence for Validity
• considers the adequacy of representation of
the conceptual domain the test is designed to
cover
Two
concepts:
Construct
underrepresentation
• describes the failure to capture important
components of a construct
Construct-irrelevant
variance
• occurs when scores are influenced by factors
irrelevant to the construct
Aspects of validity
Criterion-Related Evidence for Validity
• tells us just how well a test corresponds
with a particular criterion
Predictive validity
• is an index of the degree to which a test score predicts
some criterion measure
Concurrent validity
• is an index of the degree to which a test score is related to
some criterion measure obtained at the same time
(concurrently).
Validity coefficient
is a statistical index used to report evidence of
validity for intended interpretations of test scores
Aspects of validity
Construct-Related Evidence for
Validity
• demonstrates whether a test measures its
intended construct
Donald Campbell & John Fiske
Convergent evidence
• It is obtained when a measure correlates well with
other tests that are believed to measure the same
construct
Discriminant evidence
• It is obtained when a measure does not correlate
well with other tests that measures other construct
Relationship between
Reliability and Validity
• Reliability and validity are related concepts.
• Although different, they work together
fin
