Meta analysis
Ruswana Anwar, Herman Wibisono
The Logic of Meta-analysis
Traditional methods of review focus on
statistical significance testing
Meta-analysis changes the focus to the
direction and magnitude of the
effects across studies
Direction and magnitude represented by
the effect size
Forms of Research Findings Suitable to Meta-
analysis
Group contrasts
Experimentally created groups
Comparison of outcomes between treatment
and comparison groups
Central tendency research
Prevalence rates
When Can You Do Meta-analysis?
Meta-analysis is applicable to collections of
research that
Are empirical, rather than theoretical
Produce quantitative results, rather than qualitative
findings
Examine the same constructs and relationships
Have findings that can be configured in a
comparable statistical form (e.g., As effect sizes,
correlation coefficients, odds-ratios, proportions,
etc.)
Are “comparable” given the question at hand
Effect Size:
The Key to Meta-analysis
The effect size makes meta-analysis possible
It is the “dependent variable”
It standardizes findings across studies such that
they can be directly compared
Different meta-analyses may use different effect size
indices
Effect Size:
The Key to Meta-analysis
Any standardized index can be an “effect size” (e.G.,
Standardized mean difference, correlation coefficient,
odds-ratio) as long as it meets the following
Is comparable across studies (generally requires
standardization)
Represents the magnitude and direction of the
relationship of interest
Is independent of sample size
Which Studies to Include?
It is critical to have an explicit inclusion
and exclusion criteria
The broader the research domain, the
more detailed they tend to become
Refine criteria as you interact with the
literature
Which Studies to Include?
Components of a detailed criteria
distinguishing features
research respondents
key variables
research methods
cultural and linguistic range
time frame
publication types
Searching Far and Wide
The “we only included published studies
because they have been peer-reviewed”
argument
Significant findings are more likely to be
published than nonsignificant findings
Critical to try to identify and retrieve all
studies that meet your eligibility criteria
Searching Far and Wide
(continued)
Potential sources for identification of
documents
Computerized bibliographic databases
Authors working in the research domain
Conference programs
Dissertations
Review articles
Hand searching relevant journal
Government reports, bibliographies,
clearinghouses
Interpreting Effect Size
Results
Cohen’s “Rules-of-Thumb”
standardized mean difference effect size
small = 0.20
medium = 0.50
large = 0.80
correlation coefficient
small = 0.10
medium = 0.25
large = 0.40
odds-ratio
small = 1.50
medium = 2.50
large = 4.30
The Effect Size
The effect size (ES) makes meta-
analysis possible
The ES encodes the selected research
findings on a numeric scale
12
The Effect Size
There are many different types of ES
measures, each suited to different
research situations
Each ES type may also have multiple
methods of computation
Interpreting Effect Size
Results
These do not take into account the
context of the intervention
They do correspond to the distribution
of effects across meta-analyses
The Standardized Mean
Difference
X X G2 s12 n1 1 s22 n2 1
ES G1 s pooled
s pooled n1 n2 2
Represents a standardized group contrast on
an inherently continuous measure
Uses the pooled standard deviation (some
situations use control group standard
deviation)
Commonly called “d” or occasionally “g”
The Odds-Ratio
The odds-ratio is based on a 2 by 2
contingency table, such as the one below
Frequencies
Success Failure ad
Treatment Group a b ES
bc
Control Group c d
The Odds-Ratio is the odds of success in the treatment group
relative to the odds of success in the control group.
Confounding of Study
Features
Relative comparisons of effect sizes across
studies are inherently correlational!
Important study features are often
confounding, obscuring the interpretive
meaning of observed differences
If the confounding is not severe and you have
a sufficient number of studies, you can model
“out” the influence of method features to
clarify substantive differences
Strengths of Meta-analysis
Imposes a discipline on the process of
summing up research findings
Represents findings in a more
differentiated and sophisticated manner
than conventional reviews
Capable of finding relationships across
studies that are obscured in other
approaches
Strengths of Meta-analysis
Protects against over-interpreting
differences across studies
Can handle a large numbers of studies
(this would overwhelm traditional
approaches to review)
Weaknesses of Meta-analysis
Requires a good deal of effort
Mechanical aspects don’t lend themselves to
capturing more qualitative distinctions between
studies
“Apples and oranges” criticism
Most meta-analyses include “blemished” studies to
one degree or another (e.g., a randomized design
with attrition)
Weaknesses of Meta-analysis
Selection bias posses a continual threat
Negative and null finding studies that you were
unable to find
Outcomes for which there were negative or null
findings that were not reported
Analysis of between study differences is
fundamentally correlational
Final Comments
Meta-analysis is a replicable and
defensible method of synthesizing
findings across studies
Meta-analysis often points out gaps in
the research literature, providing a solid
foundation for the next generation of
research on that topic
Final Comments
Meta-analysis facilitates generalization
of the knowledge gain through
individual evaluations
Meta analysis
Ruswana Anwar, Herman Wibisono
