CS 4763 Fundamentals of Multimedia Systems
- Introduction to Image Processing
Qi Tian
Computer Science Department
University of Texas at San Antonio
[email protected]
http://www.cs.utsa.edu/~qitian/
Image Processing
Manipulation of multidimensional signals
image (photo)
video
CT, MRI
Fluid flow
f ( x, y )
f ( x, y , t )
f ( x, y , z , t )
v ( x, y , z , t )
A Typical Image Processing System
X-ary, radar imaging, infrared
imaging, ultrasound imaging,
medical imaging, geophysical
imaging
Imaging
systems
Sample and
quantize
Digital
storage
(disk)
Digital
computer
On-line
buffer
A/D
object
observe
digitize
Display
output
store
process
Refresh
/store
Record
Fundamentals of Image Processing
Representation
acquisition, digitization, and display to mathematical
characterization of images for subsequent processing
a prerequisite for an efficient processing techniques such as
enhancement, filtering, and restoration.
Processing Techniques
Image compression, image restoration, and image reconstruction
Statistical image processing techniques
Communications
Multimedia Processing Techniques
Coding/compression
Storage and communications
9 JPEG, JPEG2000
9 MPEG-1 (CD, mp3), MPEG-2 (HDTV, DVD)
9 H.261, H.263
Enhancement, restoration, reconstruction
9 feature extraction for image analysis and visual information display
9 removal of degradation in an image, LS, ML, Max entropy, MAP
9 2D -> 3D image MRI, CT, Radon transform
Analysis, detection, recognition, understanding
quantitative measurements from an image to produce a description on it
Visualization
Advanced Processing Techniques
Statistical processing techniques
Hidden Markov model (HMM)
Probabilistic graphical models
Bayesian networks
Markov random field
Many applications to speech recognition, pattern classification, data
compression, and channel coding, etc.
History of Image/Video Coding
Signal Processing
Based
1950
1960
1970
1980
1990
2000+
Math
PR, CV, CG
Fractal
3-D Model based
coding
PCM
DPCM
Transform Coding
VQ
Subband Coding
Wavelets
Reference:
F. Nebeker, Signal Processing: The Emergency of a Discipline,
1948-1998
IEEE History Center, 1998
Broadband TV (NTSC)
500 500 8 3
30 bits/sec
100 Mb/sec (compression is necessary!)
Modem: 56Kb/sec
Picture Element
Pixel
West coast people in USC
Pel
East people in MIT
Image/Video Compression
Signal-Processing Based:
Encoder
H
f ( x, y )
Representation
Signal
Proc.
g ( x, y )
Decoder
g ( x, y )
g ( x, y )
f ( x, y )
Image/Video Compression
3D Model-Based:
Encoder
f ( x, y )
H
Analysis
Model
Parameter P
Model
Representation P
Decoder
P
3D
Model
f ( x, y )
Image/Video Compression
Fractal-Based:
Encoder
f ( x, y )
System S
f ( x, y )
Find S for which f ( x , y ) is an Attractor.
Representation S
Iteration
Decoder
Any
signal
f ( x, y )
Image/Video Compression
Standard
Facsimile: Fax Group 1, 2, 3, 4
JBIG (Joint Bi-level Image Expert Group)
Images:
JPEG (http://www.jpeg.org/)
JPEG2000
Video:
Based on Wavelet
Transform
MPEG 7
P 64 Kb/s (P =1 ~ 30)
1.2 Mb/s Video, CD, MP3
1.2 20 Mb/s, sports, HDTV, DVD
1 kb/s 1Mb/s, very low speed video
coding, Multimedia
Multimedia description, audio/video
MPEG 21
Multimedia framework
H.261, H.263
MPEG 1
MPEG 2
MPEG 4
Lena
A de facto image for the past three decade for its rich texture
What are Challenging Problems in
Multimedia Processing?
Multimedia Processing is taken in a broad sense,
including:
Image/Video compression, enhancement, restoration,
reconstruction, analysis, recognition, understanding,
visualization, and synthesis/animation.
Examples
Face modeling, detection, and recognition
Emotion recognition
Gesture recognition
Gender/age/ethnicity recognition
Audio-visual speech recognition
Image/video superresolution
Image/video browsing, indexing, and retrieval
Biometrics
Face Related Research
Face modeling
Face detection
Face recognition
Facial expression recognition
Generic Face Model
Face model
Texture
mapping
morphing
Generic Face Model
The generic face model is generated from a MRI data set
Customize A Genetic Face on An
Individual
Polygon Mesh: 2240 Vertices + 3946 Triangles.
Non-Uniform Rational B-Splines (NURBS): 63 control points.
Avatar talking head
The iFACE system in a distributed collaborative environment. (a)
Avatar in the head mounted display, (b) avatar in the desk screen of
MIC3E, (c) avatar in the main screen of MIC3E
University of Illinois at Urbana-Champaign
Text-Driven Face Animation
We strive to make the meter on
animation production, and are
always looking for new technology
that will enable faster, more
appealing character creation,
said Joel Kransove, Digital Director of
Nickelodeon. (Source: Digital
Producer)
Speech-Driven Face Animation
Game characters have become
synthetic actors and dialogue is an
essential element of the effect we
create. The quality of the lipsynching can make or break the
sense of reality,
said Scott Cronce, vice president and CTO at
Electronic Art (Source: Gamepro)
Video-Driven Face Animation
Emotion Recognition
Emotion Recognition
Emotion Recognition
Face Detection Techniques
Face Detection Techniques
Face Recognition: Why it is easy?
Face Recognition: Why it is hard?
Beauty Check
What Are the Causes and Consequences of
Human Facial Attractiveness?
Babyfaceness
Symmetry
Social perception
Universities of Regensburg, Germany
Which is more attractive?
Universities of Regensburg, Germany
Babyfaceness
Large head
Large curved forehead
Facial elements (eyes,
nose, mouth) located
relatively low
Large, round eyes
Small, short nose
Round cheeks
4-year old girl
Kate Moss
Small chin
Include mature female features: high, prominent cheek bones and
concave cheeks
Which one is cuter?
Miss Germany (2002)
A selection of the 22 contestants of the final round of
the contest
Real vs. Virtual Miss Germany
Image Analysis
Texture synthesis and transfer
Image Super-resolution
Image Repairs
Illumination/Lighting changes and transfer
Texture Synthesis and Transfer
synthesis
+
transfer
SIGGRAPH01 Effros & Freeman, MIT, 2001
Texture Synthesis and Transfer
Image Superresolution
True
Sub-sampled
Intelligent guess about details of texture
Image Superresolution
Gaussian filter
Bicubic interpolation
Image Superresolution
Median filter
Wiener filter
Image Superresolution
Dynamic resolution
enhancement
Amos Storkey
True
Image Repairs
Image Repairs
Segmentation
Original Image
Curve connection
Image synthesis
based on Tensor
Voting
Result
Image Repairs
Illumination Effects on Images
Relighting Basic Algorithm
Step 1: Align image with generic 3D face model
Step 2: Approximate radiance environment map
Step 3: Synthesize novel appearance by adjusting
the 9 spherical harmonic coefficients
Lighting Transfer
input
target
results
Image/Video Retrieval
Image database
CBIR based on color, texture,
shape/structure
MARS: Multimedia Analysis and Retrieval System
C/C++
Feature
Extraction
User
Interface
Visual
C++
Color
Texture
structure
memory
Feature
weighting
Similarity
ranking
metadata
State-of-the-art
CBIR Systems
QBIC (IBM), PhotoBook (Media Lab), Netra (UCSB),
VisualSeek (Columbia), PicHunter (NEC-NJ), Amore (NECCA), EI Nio (Praja), MARS (UIUC), Virage (Virage Inc.),
CORE, PictoSeek, Piction, InfoScope
Research Communities
Computer Vision, Image/Video Processing, Library and
Information Science, Database and Management Systems
Leading Journals & Standard
PAMI, ACM Multimedia, IJCV, CVIU
MPEG-7
MARS using global features
Biometrics
Security Threats:
We now live in a global society of increasing desperate and dangerous
people whom we can no longer trust based on identification documents
which may have been compromised.
A challenging Pattern Recognition Problem
Enabling technology to make our society safer,
reduce fraud and offer user convenience.
Too many passwords to remember
Identification Problems
Identity Theft: Identity
thieves steal PIN (e.g., date
of birth) to open credit card
account, withdraw money
from accounts and take out
loans
3.3 million identity thefts in
U.S. in 2002; 6.7 million
victims of credit card fraud
Surrogate representations of identity such as password
and ID cards no longer suffice
Biometrics
Automatic recognition of people on their
distinctive anatomical (e.g., face, fingerprint, iris,
retina, hand geometry) and behavioral (e.g.,
signature, gait) characteristics.
Person identification is now an integral part of the
infrastructure needed for diverse business sectors
such as banking, border control, law
enforcement
Biometric Applications
Biometric Applications
There are ~500 million border
crossing/year (each way) in the US
Want to charge it?
Biometric Characteristics
Biometric Market Growth
International Biometric Group
State-of-the-art Error Rate
False accept rate
(FAR):
Proportion of
imposters
accepted
False reject rate
(FRR):
Proportions of
genuine users
rejected
Multibiometrics
Soft Biometrics
Privacy Concerns
Tracking
