Digital Camera Design

An Interesting Case Study

EE8205: Embedded Computer Systems
http://www.ee.ryerson.ca/~courses/ee8205/
Dr. Gul N. Khan
http://www.ee.ryerson.ca/~gnkhan
Electrical and Computer Engineering
Ryerson University

Overview
1. Introduction to a simple Digital Camera
2. Designer’s Perspective
3. Requirements and Specification
4. Designs and Implementations

Chapter 7, Embedded System Design by Vahid and Givargis 1

 Introduction
• Digital Camera Embedded System
 General-purpose processor
 Special-purpose processor
Custom or Standard
 Memory
 Interfacing
• Designing a simple digital camera
 General-purpose vs. single-purpose processors
 Partitioning of functionality among different types
of processor

Embedded Computer Systems: EE8205 Digital Camera Example 2

 A Simple Digital Camera
General Requirements
• Captures images
• Stores images in digital format
 No film
 Multiple images stored in camera
o Number depends on amount of memory and bits used per image
• Downloads images to Computer System (PC)
Only Recently Possible
• Systems-on-a-chip: Multiple processors & memories on an IC
• High-capacity flash memory
• Simple Description: Real Digital Camera has more features
 Variable size images, image deletion, digital stretching, zooming in/out, etc.

Embedded Computer Systems: EE8205 Digital Camera Example 3

 A Simple Digital Camera
Digital Camera Chip
CCD
CCD preprocessor Pixel coprocessor D2A
A2D

lens

JPEG Codec Microcontroller Multiplier/Accum

DMA controller Display ctrl

Memory controller ISA bus interface UART LCD ctrl

• Single-functioned -- always a digital camera

• Tightly-constrained -- Low cost, low power, small, fast
• Reactive and real-time -- only to a small extent
Embedded Computer Systems: EE8205 Digital Camera Example 4
 Design Challenges

Optimizing Design Metrics

• Obvious Design Goal
 Construct an implementation with desired functionality
• Key Design Challenge
 Simultaneously optimize numerous design metrics
• Design Metric
 A measurable feature of a system’s implementation
 Optimizing design metrics is a key challenge

Embedded Computer Systems: EE8205 Digital Camera Example 5

 Design Challenges
Common Design Metrics
• Unit cost: The monetary cost of manufacturing each copy of
the system, excluding NRE cost
• NRE cost (Non-Recurring Engineering cost): The one-
time monetary cost of designing the system
• Size: the physical space required by the system
• Performance: the execution time or throughput of the system
• Power: the amount of power consumed by the system
• Flexibility: the ability to change the functionality of the
system without incurring heavy NRE cost

Embedded Computer Systems: EE8205 Digital Camera Example 6

 Design Challenges

Common Design Metrics

• Time-to-prototype: the time needed to build a working
version of the system
• Time-to-market: the time required to develop a system
to the point that it can be released and sold to
customers
• Maintainability: the ability to modify the system after
its initial release
• Correctness, safety, many more

Embedded Computer Systems: EE8205 Digital Camera Example 7

 Design Metric
Improving one may worsen the others
Power • Expertise with both software
and hardware is needed to
Performance Size
optimize design metrics
 Not just a hardware or
software expert, as is common
NRE cost  A designer must be
comfortable with various
Digital camera chip
CCD technologies in order to choose
CCD preprocessor Pixel coprocessor D2A
A2D the best for a given application
lens
JPEG codec Microcontroller Multiplier/Accum
and constraints
DMA controller Display ctrl Hardware
Software
Memory controller ISA bus interface UART LCD ctrl

Embedded Computer Systems: EE8205 Digital Camera Example 8

 Time-to-Market
A demanding design metric
• Time required to develop a
product to the point it can be
sold to customers
• Market window
Revenues ($)

 Period during which the

product would have highest
sales
• Average time-to-market
Time (months) constraint is about 8 months
• Delays can be costly

Embedded Computer Systems: EE8205 Digital Camera Example 9

 Losses due to Delayed Market Entry

• Simplified revenue model

 Product life = 2W, peak at W
Peak revenue  Time of market entry defines a
Peak revenue from triangle, representing market
Revenues ($)

delayed entry penetration

On-time

Market rise Market fall  Triangle area equals revenue

Delayed • Loss
 The difference between the on-
time and delayed triangle areas
D W 2W
On-time Delayed Time
entry entry

Embedded Computer Systems: EE8205 Digital Camera Example 10

 Losses due to Delayed Market Entry

• Area = 1/2 * base * height

Peak revenue On-time = 1/2 * 2W * W
Peak revenue from
Delayed = 1/2 * (W-D+W)*(W-D)
Revenues ($)

On-time
delayed entry
• Percentage revenue loss =
Market rise Market fall (D(3W-D)/2W2)*100%
Delayed • Try some examples
– Lifetime 2W=52 wks, delay D=4 wks
D W 2W
– (4*(3*26 –4)/2*26^2) = 22%
On-time Delayed Time – Lifetime 2W=52 wks, delay D=10 wks
entry entry – (10*(3*26 –10)/2*26^2) = 50%
– Delays are costly!

Embedded Computer Systems: EE8205 Digital Camera Example 11

 NRE and Unit Cost Metrics
Costs:
 Unit cost: the monetary cost of manufacturing each copy of the system,
excluding NRE cost
 NRE cost (Non-Recurring Engineering cost): The one-time monetary cost of
designing the system
 total cost = NRE cost + unit cost * # of units
 per-product cost = total cost / # of units
= (NRE cost / # of units) + unit cost

Example:
– NRE=$2000, unit=$100
– For 10 units
– total cost = $2000 + 10*$100 = $3000
– per-product cost = $2000/10 + $100 = $300

Amortizing NRE cost over the units results in an

additional $200 per unit

Embedded Computer Systems: EE8205 Digital Camera Example 12

 NRE and Unit Cost Metrics
• Compare technologies by costs -- best depends on quantity
 Technology A: NRE=$2,000, unit=$100
 Technology B: NRE=$30,000, unit=$30
 Technology C: NRE=$100,000, unit=$2

$200,000 $200
A A
B B
$160,000 $160
C C
tota l c ost (x1000)

p er p rod uc t c ost
$120,000 $120

$80,000 $80

$40,000 $40

$0 $0
0 800 1600 2400 0 800 1600 2400
Num b e r o f units (v o lu m e ) Num b e r o f units (v o lu m e )

But, must also consider time-to-market

Embedded Computer Systems: EE8205 Digital Camera Example 13
 The Performance: A Design Metric
• Widely-used measure of system, widely-abused
 Clock frequency, instructions per second – not good measures
 Digital camera example – a user cares about how fast it processes images, not
clock speed or instructions per second
• Latency (response time)
 Time between task start and end
 e.g., Camera’s A and B process images in 0.25 seconds
• Throughput
 Tasks per second, e.g. Camera A processes 4 images per second
 Throughput can be more than latency seems to imply due to concurrency, e.g.
Camera B may process 8 images per second (by capturing a new image while
previous image is being stored).
• Speedup of B over S = B’s performance / A’s performance
 Throughput speedup = 8/4 = 2

Embedded Computer Systems: EE8205 Digital Camera Example 14

 Digital Camera Designer’s Perspective

Two key Tasks

• Processing images and storing in memory
 When shutter pressed:
o Image captured
o Converted to digital form by charge-coupled device (CCD)
o Compressed and archived in internal memory
• Uploading images to PC
 Digital camera attached to PC
 Special software commands camera to transmit archived
images serially

Embedded Computer Systems: EE8205 Digital Camera Example 15

 Charge-Coupled Device (CCD)
• Special sensor that captures an image
• Light-sensitive silicon solid-state device
When exposed to light, composed of many cells
each cell becomes Electromechanical shutter
electrically charged. This Lens area
is activated to expose the
charge can then be cells to light for a brief
converted to a n-bit value Covered columns
Electro- moment.
where 0 represents no mechanical
shutter
exposure while 2n-1 The electronic circuitry,
represents very intense when commanded,
Pixel rows

exposure of that cell to Electronic discharges the cells,

light. circuitry
activates electromechanical
shutter, and then reads the
Some of the columns are n-bit charge value of each
covered with a black strip cell. These values can be
of paint. The light- clocked out of the CCD by
Pixel columns
intensity of these pixels is ext logic through a parallel
used for zero-bias bus interface.
adjustments for all cells.

Embedded Computer Systems: EE8205 Digital Camera Example 16

 Zero-bias Error
• Manufacturing errors cause cells to measure slightly above or
below actual light intensity
• Error typically same across columns, but different across rows
• Some of left most columns blocked by black paint to detect
zero-bias error
 Reading of other than 0 in blocked cells is zero-bias error
 Each row is corrected by subtracting the average error found in blocked
cells for that row
Covered
cells Zero-bias
adjustment
136 170 155 140 144 115 112 248 12 14 -13 123 157 142 127 131 102 99 235
145 146 168 123 120 117 119 147 12 10 -11 134 135 157 112 109 106 108 136
144 153 168 117 121 127 118 135 9 9 -9 135 144 159 108 112 118 109 126
176 183 161 111 186 130 132 133 0 0 0 176 183 161 111 186 130 132 133
144 156 161 133 192 153 138 139 7 7 -7 137 149 154 126 185 146 131 132
122 131 128 147 206 151 131 127 2 0 -1 121 130 127 146 205 150 130 126
121 155 164 185 254 165 138 129 4 4 -4 117 151 160 181 250 161 134 125
173 175 176 183 188 184 117 129 5 5 -5 168 170 171 178 183 179 112 124

Before zero-bias adjustment After zero-bias adjustment

Embedded Computer Systems: EE8205 Digital Camera Example 17
 Compression

• Store more images

• Transmit image to PC in less time
• JPEG (Joint Photographic Experts Group)
 Popular standard format for representing compressed digital images
 Provides for a number of different modes of operation
 Mode used in this chapter provides high compression ratios using
DCT (discrete cosine transform)
 Image data divided into blocks of 8 x 8 pixels
 3 steps performed on each block
DCT, Quantization and Huffman encoding

Embedded Computer Systems: EE8205 Digital Camera Example 18

 DCT step
• Transforms original 8 x 8 block into a cosine-frequency
domain
 Upper-left corner values represent more of the essence of the image
 Lower-right corner values represent finer details
o Can reduce precision of these values and retain reasonable image quality
• FDCT (Forward DCT) formula
 C(h) = if (h == 0) then 1/sqrt(2) else 1.0
Auxiliary function used in main function F(u,v)
 F(u,v) = ¼ C(u) C(v) Σx=0..7 Σy=0..7 Dxy cos(π(2x + 1)u/16) cos(π(2y + 1)v/16)
Gives encoded pixel at row u, column v
Dxy is original pixel value at row x, column y

• IDCT (Inverse DCT)

 Reverses process to obtain original block (not needed for this design)

Embedded Computer Systems: EE8205 Digital Camera Example 19

 Quantization Step
• Achieve high compression ratio by reducing image
quality
 Reduce bit precision of encoded data
o Fewer bits needed for encoding
o One way is to divide all values by a factor of 2
Simple right shifts can do this
 Dequantization would reverse process for decompression
1150 39 -43 -10 26 -83 11 41 144 5 -5 -1 3 -10 1 5
-81 -3 115 -73 -6 -2 22 -5
Divide each cell’s -10 0 14 -9 -1 0 3 -1
14 -11 1 -42 26 -3 17 -38 value by 8 2 -1 0 -5 3 0 2 -5
2 -61 -13 -12 36 -23 -18 5 0 -8 -2 -2 5 -3 -2 1
44 13 37 -4 10 -21 7 -8 6 2 5 -1 1 -3 1 -1
36 -11 -9 -4 20 -28 -21 14 5 -1 -1 -1 3 -4 -3 2
-19 -7 21 -6 3 3 12 -21 -2 -1 3 -1 0 0 2 -3
-5 -13 -11 -17 -4 -1 7 -4 -1 -2 -1 -2 -1 0 1 -1

After being decoded using DCT After quantization

Embedded Computer Systems: EE8205 Digital Camera Example 20

 Huffman Encoding
• Serialize 8 x 8 block of pixels
 Values are converted into single list using zigzag pattern

• Perform Huffman encoding

 More frequently occurring pixels assigned short binary code
 Longer binary codes left for less frequently occurring pixels
• Each pixel in serial list converted to Huffman encoded values
 Much shorter list, thus compression
Embedded Computer Systems: EE8205 Digital Camera Example 21
 Huffman Encoding Example
Pixel frequencies on left
• Pixel value –1 occurs 15 times
• Pixel value 14 occurs 1 time Huffman
Pixel Huffman tree
Build Huffman tree from bottom up frequencie codes
s 6
• Create one leaf node for each pixel -1 15x 4
-1 00

value and assign frequency as node’s 0 8x

3
0 100
-2 110
value -2
1
6x
5x
29 5
1 010

• Create an internal node by joining any 2 5x

1
2
3
1110
3 5x 1 1010
two nodes whose sum is a minimal 5 5x
1
1
4 8 7 5 0110
value. This sum is internal nodes value -3 4x -1
5
1 1
-3 11110
9 -5
-5 3x 8 6 10110
• Repeat until complete binary tree -10 2x
5
0
0
-2
1
-10 01110
1
144
Traverse tree from root to leaf. To 144 1x
5
4
5
5 5
6
-9
111111
111110
-9 1x
obtain binary code for leaf’s -8 1x 5 3
2
2
2
-8 101111
2 4 -4
pixel -4 1x 2
-5
3
-3 6
101110
011111
6 1x -10
• Append 0 for left traversal, 1 for right 14 1x 1 1 1 1 1 1 14 011110

traversal 1 6 - -8 - 14
4 4 9 4
Huffman encoding is reversible
• No code is a prefix of another code

Embedded Computer Systems: EE8205 Digital Camera Example 22

 Archiving
• Record starting address and image size
 One can use a linked list structure
• One possible way to archive images. For example, if max
number of images archived is N
 Set aside memory for N addresses and N image-size variables
 Keep a counter for location of next available address
 Initialize addresses and image-size variables to 0
 Set global memory address to N x 4
o Assuming addresses, image-size variables occupy N x 4 bytes
 First image archived starting at address N x 4
 Global memory address updated to N x 4 + (compressed image size)
• Memory requirement based on N, image size, and average
compression ratio
Embedded Computer Systems: EE8205 Digital Camera Example 23
 Uploading to a Computer System
When connected to a Computer System and upload
command received
• Read images from the memory
• Transmit serially using UART
(e.g. via a USB port)
• While transmitting
Reset pointers, image-size variables and global
memory pointer accordingly

Embedded Computer Systems: EE8205 Digital Camera Example 24

 Requirements Specification
System’s requirements – what system should do
• Nonfunctional Requirements
 Constraints on design metrics (e.g. “should use 0.001 watt or less”)
• Functional Requirements
 System’s behavior (e.g. “output X should be input Y times 2”)
• Initial specification may be very general and come from
marketing department.
e.g. Short document detailing market need for a low-end digital camera:
 Captures and stores at least 50 low-res images and uploads to PC
 Costs around $100 with single medium-size IC costing less that $25
 Has long as possible battery life
 Has expected sales volume of 200,000 if market entry < 6 months
 100,000 if between 6 and 12 months
 insignificant sales beyond 12 months

Embedded Computer Systems: EE8205 Digital Camera Example 25

 Nonfunctional Requirements
Design metrics of importance based on initial
specification
• Performance: time required to process image
• Size: number of logic gates (2-input NAND gate) in IC
• Power: measure of avg. power consumed while processing
• Energy: battery lifetime (power x time)
Constrained metrics
• Values must be below (sometimes above) certain threshold
Optimization metrics
• Improved as much as possible to improve product
Metric can be both constrained and optimization
Embedded Computer Systems: EE8205 Digital Camera Example 26
 Nonfunctional Requirements
Performance
 Must process image fast enough to be useful
 1 sec reasonable constraint
Slower would be annoying and Faster not necessary for low-end of market
 Therefore, constrained metric
Size
 Must use IC that fits in reasonably sized camera
 Constrained and optimization metric: 200K gates, but lower is cheaper
Power
 Must operate below certain temperature (no-cooling fan) a constrained metric
Energy
 Reducing power or time reduces energy
 Optimized metric: want battery to last as long as possible

Embedded Computer Systems: EE8205 Digital Camera Example 27

 Informal Functional Specification
• Flowchart breaks
functionality down into CCD Zero-bias
simpler functions input adjust

• Each function’s details DCT

could then be described

in English Quantize
yes

 Done earlier in chapter no Done

Archive in
• Low quality image has memory ?

resolution of 64 x 64
yes More no Transmit
• Mapping functions to a 8×8 serially
serial output
e.g.
blocks
particular processor type ? 011010...

not done at this stage

Embedded Computer Systems: EE8205 Digital Camera Example 28

 Refined Functional Specification
• Refine informal specification into Executable Model of Digital Camera
one that can actually be executed
• Can use C/C++ code to describe
CCD.C
each function 10101101
01101010
 Called system-level model, 10010101
prototype, or simply model 101... CCDPP. CODEC.
 Also is first implementation C C
image file
• Can provide insight into
operations of system CNTRL.
C 10101010
 Profiling can find 10101010
computationally intensive 10101010
functions UART.C 1010...
• Can obtain sample output used to output file
verify correctness of final
implementation

Embedded Computer Systems: EE8205 Digital Camera Example 29

 CCD Module
Simulates a Real CCD
 CcdInitialize is passed name of image file
 CcdCapture reads “image” from file
 CcdPopPixel outputs pixels one at a time
#include <stdio.h>
#define SZ_ROW 64 void CcdInitialize(const char *imageFileName) {
#define SZ_COL (64 + 2) imageFileHandle = fopen(imageFileName, "r");
static FILE *imageFileHandle; rowIndex = -1;
static char colIndex = -1;
buffer[SZ_ROW][SZ_COL]; }
static unsigned rowIndex,
colIndex;
void CcdCapture(void) {
char CcdPopPixel(void) { int pixel;
char pixel; rewind(imageFileHandle);
pixel = for(rowIndex=0; rowIndex<SZ_ROW; rowIndex++) {
buffer[rowIndex][colIndex]; for(colIndex=0; colIndex<SZ_COL; colIndex++) {
if( ++colIndex == SZ_COL ) { if( fscanf(imageFileHandle, "%i", &pixel)
colIndex = 0; == 1 ) {
if( ++rowIndex == SZ_ROW ) { buffer[rowIndex][colIndex] = (char)pixel;
colIndex = -1; }
rowIndex = -1; }
} }
} rowIndex = 0;
return pixel; colIndex = 0;
} }

Embedded Computer Systems: EE8205 Digital Camera Example 30

 CCDPP (CCD PreProcessing) Module
Performs zero-bias Adjustment
• CcdppCapture uses CcdCapture and #define SZ_ROW 64
#define SZ_COL 64
CcdPopPixel to obtain the image static char
buffer[SZ_ROW][SZ_COL];
• Performs zero-bias adjustment after each static unsigned rowIndex,
colIndex;
row read in void CcdppInitialize() {
rowIndex = -1;
colIndex = -1;
}
void CcdppCapture(void) {
char bias; char CcdppPopPixel(void) {
CcdCapture(); char pixel;
for(rowIndex=0; rowIndex<SZ_ROW; rowIndex++) { pixel =
for(colIndex=0; colIndex<SZ_COL; colIndex++) { buffer[rowIndex][colIndex];
buffer[rowIndex][colIndex] = CcdPopPixel(); if( ++colIndex == SZ_COL ) {
} colIndex = 0;
bias = (CcdPopPixel() + CcdPopPixel()) / 2; if( ++rowIndex == SZ_ROW )
for(colIndex=0; colIndex<SZ_COL; colIndex++) { {
buffer[rowIndex][colIndex] -= bias; colIndex = -1;
} rowIndex = -1;
} }
rowIndex = 0; }
colIndex = 0; return pixel;
} }

Embedded Computer Systems: EE8205 Digital Camera Example 31

 UART Module
Actually a half UART
 Only transmits, does not receive
• UartInitialize is passed name of file to output to
• UartSend transmits (writes to output file) bytes at a
time
#include <stdio.h>
static FILE *outputFileHandle;
void UartInitialize(const char *outputFileName) {
outputFileHandle = fopen(outputFileName, "w");
}

void UartSend(char d) {
fprintf(outputFileHandle, "%i\n", (int)d);
}

Embedded Computer Systems: EE8205 Digital Camera Example 32

 CODEC Module
static short ibuffer[8][8],
obuffer[8][8], idx;

void CodecInitialize(void) { idx = 0;

• Models FDCT encoding }

• ibuffer holds original 8 x 8 block void CodecPushPixel(short p) {

if( idx == 64 ) idx = 0;
• obuffer holds encoded 8 x 8 block ibuffer[idx / 8][idx % 8] = p;
idx++;
}
• CodecPushPixel called 64 times to fill
void CodecDoFdct(void) {
ibuffer with original block int x, y;
for(x=0; x<8; x++) {
• CodecDoFdct called once to for(y=0; y<8; y++)
obuffer[x][y] = FDCT(x, y,
transform 8 x 8 block ibuffer);
}
 Explained in next slide idx = 0;
}
• CodecPopPixel called 64 times to
short CodecPopPixel(void) {
retrieve encoded block from obuffer short p;
if( idx == 64 ) idx = 0;
p = obuffer[idx / 8][idx % 8];
idx++;
return p;
}

Embedded Computer Systems: EE8205 Digital Camera Example 33

 CODEC
static const short COS_TABLE[8][8] = {
{ 32768, 32138, 30273, 27245, 23170, 18204, 12539, 6392 },
{ 32768, 27245, 12539, -6392, -23170, -32138, -30273, -18204 },
{ 32768, 18204, -12539, -32138, -23170, 6392, 30273, 27245 },
{ 32768, 6392, -30273, -18204, 23170, 27245, -12539, -32138 },
{ 32768, -6392, -30273, 18204, 23170, -27245, -12539, 32138 },
{ 32768, -18204, -12539, 32138, -23170, -6392, 30273, -27245 },
{ 32768, -27245, 12539, 6392, -23170, 32138, -30273, 18204 },
{ 32768, -32138, 30273, -27245, 23170, -18204, 12539, -6392 }
};

static short ONE_OVER_SQRT_TWO = 23170; static int FDCT(int u, int v, short img[8][8]) {
static double COS(int xy, int uv) { double s[8], r = 0; int x;
for(x=0; x<8; x++) {
return COS_TABLE[xy][uv] / 32768.0; s[x] = img[x][0] * COS(0, v) + img[x][1] * COS(1, v)
} + img[x][2] * COS(2, v) + img[x][3] * COS(3, v)
static double C(int h) { + img[x][4] * COS(4, v) + img[x][5] * COS(5, v)
return h ? 1.0 : ONE_OVER_SQRT_TWO / + img[x][6] * COS(6, v) + img[x][7] * COS(7, v);
32768.0; }
} for(x=0; x<8; x++) r += s[x] * COS(x, u);
return (short)(r * .25 * C(u) * C(v));
}

Implementing FDCT Formula

C(h) = if (h == 0) then 1/sqrt(2) else 1.0
F(u,v) = ¼ C(u) C(v) Σx=0..7 Σy=0..7 Dxy cos(π(2x + 1)u/16) cos(π(2y + 1)v/16)
Only 64 possible inputs to COS, so table can be used to save performance time
 Floating-point values multiplied by 32,678 and rounded to nearest integer
 32,678 chosen in order to store each value in 2 bytes of memory
 Fixed-point representation explained more later
FDCT unrolls inner loop of summation, implements outer summation as two consecutive for loops

Embedded Computer Systems: EE8205 Digital Camera Example 34

 CNTRL (controller) Module
Heart of the system
CntrlInitialize for consistency with other modules only
CntrlCaptureImage uses CCDPP module to input image and place in buffer
CntrlCompressImage breaks the 64 x 64 buffer into 8 x 8 blocks and performs FDCT on each
block using the CODEC module. Also performs quantization on each block
CntrlSendImage transmits encoded image serially using UART module
void CntrlSendImage(void) {
for(i=0; i<SZ_ROW; i++)
for(j=0; j<SZ_COL; j++) {
temp = buffer[i][j];
void CntrlCaptureImage(void) { UartSend(((char*)&temp)[0]); // send upper byte
CcdppCapture(); UartSend(((char*)&temp)[1]); // send lower byte
for(i=0; i<SZ_ROW; i++) }
}
for(j=0; j<SZ_COL; j++) }
buffer[i][j] = void CntrlCompressImage(void) {
CcdppPopPixel(); for(i=0; i<NUM_ROW_BLOCKS; i++)
} for(j=0; j<NUM_COL_BLOCKS; j++) {
for(k=0; k<8; k++)
for(l=0; l<8; l++)
#define SZ_ROW 64 CodecPushPixel((char)buffer[i * 8 + k][j * 8 + l]);
#define SZ_COL 64 CodecDoFdct();/* part 1 - FDCT */
for(k=0; k<8; k++)
#define NUM_ROW_BLOCKS (SZ_ROW / 8)
for(l=0; l<8; l++) {
#define NUM_COL_BLOCKS (SZ_COL / 8) buffer[i * 8 + k][j * 8 + l] = CodecPopPixel();
static short buffer[SZ_ROW][SZ_COL]; /* part 2 - quantization */
static short i, j, k, l, temp; buffer[i*8+k][j*8+l] >>= 6;
void CntrlInitialize(void) {} }
}
}

Embedded Computer Systems: EE8205 Digital Camera Example 35

 Overall System
• Main initializes all modules, then uses CNTRL module to
capture, compress, and transmit one image
• This system-level model can be used for extensive
experimentation
 Bugs much easier to correct here rather than in later models
int main(int argc, char *argv[]) {
char *uartOutputFileName = argc > 1 ? argv[1] : "uart_out.txt";
char *imageFileName = argc > 2 ? argv[2] : "image.txt";
/* initialize the modules */
UartInitialize(uartOutputFileName);
CcdInitialize(imageFileName);
CcdppInitialize();
CodecInitialize();
CntrlInitialize();
/* simulate functionality */
CntrlCaptureImage();
CntrlCompressImage();
CntrlSendImage();
}

Embedded Computer Systems: EE8205 Digital Camera Example 36

 The Design
Determine system’s architecture
• Any combination of single-purpose (custom/standard) or general-purpose
processors, Memories and buses
Map functionality to that architecture
• Multiple functions on 1 processor or 1 function on one/more processors
Implementation
• A particular architecture and mapping
• Solution space is set of all implementations
• Low-end general-purpose processor connected to flash memory
 All functionality mapped to software running on processor
 Usually satisfies power, size, and time-to-market constraints
 If timing constraint not satisfied then later implementations could:
Use single-purpose processors for time-critical functions and rewrite
functional specification

Embedded Computer Systems: EE8205 Digital Camera Example 37

First Implementation: One Microcontroller
• Low-end processor could be Intel 8051 microcontroller
• Total IC cost including NRE about $5
• Well below 200 mW power
• Time-to-market about 3 months
• However, one image per second not possible
 12 MHz, 12 cycles per instruction
o Executes one million instructions per second
 CcdppCapture has nested loops resulting in 4096 (64 x 64) iterations
o ~100 assembly instructions each iteration
o 409,000 (4096 x 100) instructions per image
o Half of budget for reading image alone
 Would be over budget after adding compute-intensive DCT and
Huffman encoding

Embedded Computer Systems: EE8205 Digital Camera Example 38

 2nd Implementation
Microcontroller and CCDPP SoC
EEPROM 8051 RAM

SOC UART CCDPP

• CCDPP function implemented on custom single-purpose processor

 Improves performance – less microcontroller cycles
 Increases NRE cost and time-to-market
 Easy to implement
o Simple datapath
o Few states in controller
• Simple UART easy to implement as single-purpose processor also
• EEPROM for program memory and RAM for data memory added as well

Embedded Computer Systems: EE8205 Digital Camera Example 39

 Microcontroller
Soft Core: Synthesizable version of 8051
 Written in VHDL Block diagram of 8051 processor core
 Captured at register transfer level (RTL)
• Fetches instruction from ROM Instruction 4K ROM
Decoder
• Decodes using Instruction Decoder Controller
• ALU executes arithmetic operations 128
ALU
RAM
 Source and destination registers reside
in RAM
• Special data movement instructions
To External Memory
used to load and store externally Bus

• Special program generates VHDL

description of ROM from output of C
compiler/linker

Embedded Computer Systems: EE8205 Digital Camera Example 40

 The UART
UART in idle mode until invoked
UART invoked when 8051 executes store
instruction with UART’s enable register as FSMD description of UART
target address
 Memory-mapped communication between invoked Start:
8051 and all single-purpose processors Idle: Transmit
I=0 LOW
 Lower 8-bits of memory address for RAM I<8
 Upper 8-bits of memory address for
memory-mapped I/O devices Stop: Data:
Transmit
• Start state transmits 0 indicating start of byte Transmi
data(I),
t HIGH
transmission then transitions to Data state I=8 then I++
• Data state sends 8 bits serially then
transitions to Stop state
• Stop state transmits 1 indicating
transmission done then transitions back to
idle mode
Embedded Computer Systems: EE8205 Digital Camera Example 41
 CCDPP
• Hardware implementation of zero-bias
operations FSMD description of CCDPP
• Interacts with external CCD chip
C < 66
 CCD chip resides external to our SOC as invoked GetRow:
Idle:
combining CCD with ordinary logic not feasible R=0
B[R][C]=Pxl
C=C+1
• Internal buffer, B, mem-mapped to 8051 C=0

• Variables R, C are row, column indices R = 64 C=

66
• GetRow reads in one row from CCD to B R < 64 ComputeBias:
 66 bytes: 64 pixels + 2 blacked-out pixels NextRow: C < 64
Bias=(B[R][11]
+ B[R][10]) / 2
• ComputeBias state computes bias for that R++
C=0
C=0
row and stores in variable Bias FixBias:
B[R][C]=B[R][C]-
• FixBias state iterates over same row C = 64 Bias
subtracting Bias from each element
• NextRow transitions to GetRow for repeat
of process on next row or to Idle state when
all 64 rows completed

Embedded Computer Systems: EE8205 Digital Camera Example 42

 Connecting SOC Components
Memory-mapped
 All single-purpose processors and RAM are connected to 8051’s memory bus
Read
 Processor places address on 16-bit address bus
 Asserts read control signal for 1 cycle
 Reads data from 8-bit data bus 1 cycle later
 Device (RAM or SPP) detects asserted read control signal
 Checks address
 Places and holds requested data on data bus for 1 cycle
Write
 Processor places address and data on address and data bus
 Asserts write control signal for 1 clock cycle
 Device (RAM or SPP) detects asserted write control signal
 Checks address bus
 Reads and stores data from data bus

Embedded Computer Systems: EE8205 Digital Camera Example 43

 Software
System-level model provides majority of code
 Module hierarchy, procedure names, and main program unchanged
Code for UART and CCDPP modules must be redesigned
 Simply replace with memory assignments
o xdata used to load/store variables over external memory bus
o _at_ specifies memory address to store these variables
o Byte sent to U_TX_REG by processor will invoke UART
o U_STAT_REG used by UART to indicate its ready for next byte
– UART may be much slower than processor
 Similar modification for CCDPP code
All other modules untouched

Original code from system-level model Rewritten UART module

#include <stdio.h> static unsigned char xdata U_TX_REG _at_ 65535;
static FILE *outputFileHandle; static unsigned char xdata U_STAT_REG _at_ 65534;
void UartInitialize(const char *outputFileName) { void UARTInitialize(void) {}
outputFileHandle = fopen(outputFileName, "w"); void UARTSend(unsigned char d) {
} while( U_STAT_REG == 1 ) {
void UartSend(char d) { /* busy wait */
fprintf(outputFileHandle, "%i\n", (int)d); }
} U_TX_REG = d;
}

Embedded Computer Systems: EE8205 Digital Camera Example 44

 Analysis
Entire SOC tested on VHDL simulator
• Interprets VHDL descriptions and
functionally simulates execution of Obtaining design metrics of interest
system
 Recall program code translated to
VHDL description of ROM VHDL VHDL VHDL
Power
• Tests for correct functionality equation
• Measures clock cycles to process VHDL Synthesis
one image (performance) simulator tool
Gate level
Gate-level description obtained by simulator
synthesis
gates gates gates
• Synthesis tool like compiler for Power
SPPs
• Simulate gate-level models to obtain Execution time Sum gates
Chip area
data for power analysis
 Number of times gates switch from
1 to 0 or 0 to 1
• Count number of gates for chip area

Embedded Computer Systems: EE8205 Digital Camera Example 45

 2nd Implementation:
Microcontroller and CCDPP

Analysis of the Implementation

 Total execution time for processing one image:
9.1 seconds
 Power consumption:
0.033 watt
 Energy consumption:
0.30 joule (9.1 s x 0.033 watt)
 Total chip area:
98,000 gates

Embedded Computer Systems: EE8205 Digital Camera Example 46

 3rd Implementation: Microcontroller
CCDPP/Fixed-Point DCT
• 9.1 seconds still doesn’t meet performance constraint
of 1 second
• DCT operation prime candidate for improvement
 Execution of 2nd implementation shows microprocessor
spends most cycles here
 Could design custom hardware like we did for CCDPP
More complex so more design effort
 Instead, will speed up DCT functionality by modifying
behavior

Embedded Computer Systems: EE8205 Digital Camera Example 47

 DCT Floating-point Cost
• Floating-point cost
 DCT uses ~260 floating-point operations per pixel transformation
 4096 (64 x 64) pixels per image
 1 million floating-point operations per image
 No floating-point support with Intel 8051 controller
o Compiler must emulate
– Generates procedures for each floating-point operation
mult, add
– Each procedure uses tens of integer operations
 Thus, > 10 million integer operations per image
 Procedures increase code size
• Fixed-point arithmetic can improve on this

Embedded Computer Systems: EE8205 Digital Camera Example 48

 Fixed-point Arithmetic
• Integer used to represent a real number
 Constant number of integer’s bits represents fractional portion of real number
More bits, more accurate the representation
 Remaining bits represent portion of real number before decimal point
• Translating a real constant to a fixed-point representation
 Multiply real value by 2 ^ (# of bits used for fractional part)
 Round to nearest integer
 e.g., represent 3.14 as 8-bit integer with 4 bits for fraction
 2^4 = 16
 3.14 x 16 = 50.24 ≈ 50 = 00110010
 16 (2^4) possible values for fraction, each represents 0.0625 (1/16)
 Last 4 bits (0010) = 2
 2 x 0.0625 = 0.125
 3(0011) + 0.125 = 3.125 ≈ 3.14 (more bits for fraction would increase accuracy)

Embedded Computer Systems: EE8205 Digital Camera Example 49

 Fixed-point Arithmetic Operations
Addition
 Simply add integer representations
 e.g., 3.14 + 2.71 = 5.85
 3.14 → 50 = 00110010

 2.71 → 43 = 00101011

 50 + 43 = 93 = 01011101

 5(0101) + 13(1101) x 0.0625 = 5.8125 ≈ 5.85

Multiply
 Multiply integer representations
 Shift result right by # of bits in fractional part
 E.g., 3.14 * 2.71 = 8.5094
 50 * 43 = 2150 = 100001100110

 >> 4 = 10000110

 8(1000) + 6(0110) x 0.0625 = 8.375 ≈ 8.5094

• Range of real values used limited by bit widths of possible resulting values

Embedded Computer Systems: EE8205 Digital Camera Example 50

 Fixed-point Implementation of CODEC
static const char code COS_TABLE[8][8] = {

• COS_TABLE gives 8-bit fixed- {

{
64,
64,
62,
53,
59,
24,
53,
-12,
45,
-45,
35,
-62,
24,
-59,
12 },
-35 },
point representation of cosine {
{
64,
64,
35,
12,
-24,
-59,
-62,
-35,
-45,
45,
12,
53,
59,
-24,
53 },
-62 },
values { 64, -12, -59, 35, 45, -53, -24, 62 },
{ 64, -35, -24, 62, -45, -12, 59, -53 },

• 6 bits used for fractional portion {

{
64,
64,
-53,
-62,
24,
59,
12,
-53,
-45,
45,
62,
-35,
-59,
24,
35 },
-12 }

• Result of multiplications shifted };

right by 6 static const char ONE_OVER_SQRT_TWO = 5;

static short xdata inBuffer[8][8];
static short outBuffer[8][8], idx;
static unsigned char C(int h)
void CodecInitialize(void) { idx = 0; }
{ return h ? 64 : ONE_OVER_SQRT_TWO;} void CodecPushPixel(short p) {
static int F(int u, int v, short img[8][8]) { if( idx == 64 ) idx = 0;
long s[8], r = 0; inBuffer[idx / 8][idx % 8] =
unsigned char x, j; p << 6; idx++;
for(x=0; x<8; x++) { }
s[x] = 0;
for(j=0; j<8; j++) void CodecDoFdct(void) {
s[x] += (img[x][j] * COS_TABLE[j][v] ) >> 6; unsigned short x, y;
} for(x=0; x<8; x++)
for(x=0; x<8; x++) r += (s[x] * COS_TABLE[x][u])>> 6; for(y=0; y<8; y++)
return (short)((((r * (((16*C(u)) >> 6) *C(v)) >> 6)) outBuffer[x][y]= F(x,y, inBuffer);
>> 6) >> 6); idx = 0;
}
}

Embedded Computer Systems: EE8205 Digital Camera Example 51

 Microcontroller, CCDPP and Fixed-
Point DCT (3rd Imp.)
Analysis of the implementation
 Use same analysis techniques as 2nd implementation
 Total execution time for processing one image:
1.5 seconds
 Power consumption:
0.033 watt (same as 2)
 Energy consumption:
0.050 joule (1.5 s x 0.033 watt)
Battery life 6x longer!!
 Total chip area:
90,000 gates
8,000 less gates (less memory needed for code)

Embedded Computer Systems: EE8205 Digital Camera Example 52

 Last Implementation: Microcontroller
and CCDPP/DCT

EEPROM 8051 RAM

CODEC UART CCDPP

SOC

• Performance close but not good enough

• Must resort to implementing CODEC in hardware
 Single-purpose processor to perform DCT on 8 x 8 block

Embedded Computer Systems: EE8205 Digital Camera Example 53

 CODEC Design
Four memory mapped registers
• C_DATAI_REG/C_DATAO_REG used
to push/pop 8 x 8 block into and out of
CODEC Rewritten CODEC software
• C_CMND_REG to command CODEC
Writing 1 to this register invokes static unsigned char xdata C_STAT_REG _at_ 65527;
static unsigned char xdata C_CMND_REG _at_ 65528;
CODEC static unsigned char xdata C_DATAI_REG _at_ 65529;
static unsigned char xdata C_DATAO_REG _at_ 65530;
• C_STAT_REG indicates CODEC done void CodecInitialize(void) {}
and ready for next block void CodecPushPixel(short p)
{ C_DATAO_REG = (char)p; }
Polled in software short CodecPopPixel(void) {
return ((C_DATAI_REG << 8) | C_DATAI_REG);
Direct translation of C code to VHDL }
for actual hardware void CodecDoFdct(void) {
C_CMND_REG = 1;
implementation }
while( C_STAT_REG == 1 ) { /* busy wait */ }

Fixed-point version used

CODEC module in software changed
similar to UART/CCDPP in 2nd
implementation
Embedded Computer Systems: EE8205 Digital Camera Example 54
 Microcontroller & CCDPP/DCT SoC
4th Implementation
• Analysis of the Implementation
 Total execution time for processing one image:
0.099 seconds (well under 1 sec)
 Power consumption:
0.040 watt
Increase over 2 and 3 because SOC has another processor
 Energy consumption:
0.00040 joule (0.099s x 0.040 watt)
Battery life 12x longer than previous implementation!!
 Total chip area:
128,000 gates
Significant increase over previous implementations

Embedded Computer Systems: EE8205 Digital Camera Example 55

 Summary of implementations
Implementation 2 Implementation 3 Implementation 4
Performance (second) 9.1 1.5 0.099
Power (watt) 0.033 0.033 0.040
Size (gate) 98,000 90,000 128,000
Energy (joule) 0.30 0.050 0.0040

3rd Implementation
• Close in performance
• Cheaper
• Less time to build
Last (4th) Implementation
• Great performance and energy consumption
• More expensive and may miss time-to-market window
 If DCT designed ourselves then increased NRE cost and time-to-market
 If existing DCT purchased then increased IC cost
• Which is better?

Embedded Computer Systems: EE8205 Digital Camera Example 56

 Summary
Digital Camera Case Study
 Specifications in English and executable language
 Design metrics: performance, power and area
Several Implementations
 Microcontroller: too slow
 Microcontroller and coprocessor: better, but still too slow
 Fixed-point arithmetic: almost fast enough
 Additional coprocessor for compression: fast enough, but
expensive and hard to design
 Tradeoffs between hw/sw – main lesson of this Case Study

Embedded Computer Systems: EE8205 Digital Camera Example 57