Partial Reconfiguration on FPGAs

Dirk Koch (koch@ifi.uio.no)

1
 Introduction: Terms and Definitions
Definition of the term „Reconfigurable Computing“ (RC)

A good definition for a reconfigurable hardware system was

introduced with the Rammig Machine (by Franz Rammig 1977):
… a system, which, with no manual or mechanical inter-
ference, permits the building, changing, processing and
destruction of real (not simulated) digital hardware

Reconfigurable computing (RC) is defined as

the study of computation using reconfigurable devices
This includes architectures, algorithms and applications

The term RC is often used to express that computation is

carried out using dedicated hardware structures (often utilizing
a high level of parallelism) which are mapped on reconfigurable
hardware (this is opposed to the sequential von Neumann
computer paradigm!!!). 2
 Introduction: Example
reconfigurable computing
for i = 0 to 7 do { A[0][3..0] RC benefits
tmp = A[i] & x"F"; + among von Neu-
tmp = tmp + 42; 42 * Q[0] mann machines:
Q[i]= tmp * 24; 24

loop unrolling
} A[1][3..0] • fast parallel
+
slow and processing
42 * Q[1]
power hungry
- pipelining
24
von Neumann computer - loop transform.

...
LDI reg_i,0 • no instr. fetch
A[7][3..0]
L1:ANDI r_tmp,$i,xF + (no extra
... 42 * Q[7] memory access)
BLI reg_i,L1 24
instruction pipelining • no instr. decode
stream
• possibility of
data ... A[7] ...
A[0]
dedicated instr.
stream 42 ...
Q[0] (e.g., MAC)
42 Q[7]
24 • lower power
24 3
 Introduction: Example (Benefits)
Reconfigurable computing permits to tradeoff between performance
(speed and/or latency) and area (number of used primitives) of the
reconfigurable architecture. This requires to solve the following steps:

Allocation: defining the resources / functional blocks

which are allowed for implementation
Binding: defining which operation is executed on a particular
allocated resource
Scheduling: defining the time when an operation is executed

Allocation, binding, and scheduling are fundamental problems that

have to be solved at different level of abstraction (e.g., system level,
architecture level, or all refinements.
This holds for both the hardware and the software part!

Further: RC removes architectural limitations (e.g., like shared

memory communication in GPUs)
4
 Introduction: Terms and Definitions
These RC benefits exist also for dedicated hardware (ASIC1,
ASIP2), but reconfigurable computing allows more:
Adaptability: react on environment changes or different
workload scenarios by adapting the behavior and structure
of a system (e.g., scaling a system with configuring more
instances of an accelerator module to a recenf. device)
Customization (post fabrication): allows for different features
for individual systems
Updatability: update to new standards, bug fixes,
after sales business with new features „hardware apps“

Possible by (re)configuration: Configuration (and respectively

reconfiguration) is the process of changing the structure of a
reconfigurable device at start-up-time (respectively run-time).
Mostly this means: sending new configurations to the device
ASIC1: application specific integrated circuit; ASIP2: application specific processor 5
 Introduction: Terms and Definitions
Reconfigurable architectures
Coarse grained:
ALU-like primitives with word
sized routing channels
Examples: NEC-DRC, PACT XPP,
Silicon Hive, Ambric, Picochip, TILERA, Nvidia GPGPU
Advantage: extreme performance for domain specific tasks
Fine grained: bit level primitives (e.g., look-up tables (LUTs))
and single wire routing
Examples: plenty of academic architectures, Atmel FPGAs
Advantage: can virtually implement anything
But often poor performance and/or chip utilization
Hybrid: fine-grained fabric with additional coarse-grained
primitives (e.g., hardware multipliers or CPUs)
Examples: Xilinx Virtex families (some with embedded PPC)
Aims at combining the advantages of both 6
 Introduction: the FPGA-ASIC Gap
Hybrid FPGAs are dominating reconfigurable market, but there is a
Gap between reconfigurable FPGAs and dedicated ASICs
@ 90nm process* FPGA versus ASIC
chip area ~ 18 x larger
dynamic power ~ 14 x more
clock speed ~ 3-5 x slower
*Kuon & Rose: Measuring the Gap Between FPGAs and ASICs, in Tr. On CAD, 2007.

Note that the gap towards a programmable von Neumann

machine could be even orders of magnitude higher!
also: lack of productive design tools (and skilled engineers)

Solution: partial run-time reconfiguration (PR):

reusing the resources of a reconfigurable architecture by
multiple modules over time. Only parts of a system might be
updated while continuing operation of the remaining system. 7
FPGA-based Systems everywhere, but not PR
FPGA-based systems are omnipresent in our daily life.

Each A380 contains more than

700 Actel FPGAs, e.g., for:
Engine control & monitoring
flight computers
braking systems
safety warning systems

8
 What we should know about FPGAs
Slow (~300 MHz), but highly parallel execution >1000 Operations
Moderate I/O throughput, but >1MB @ >1TB/sec (on-chip)
Difficult VHDL programming, but C++ is coming up

data flow oriented vs. control flow dominated

for i=1 to loop if (old_position) then
numbercrunching; case position is
42: if free then

9
PR Advantages: Area Saving
Networking:
Adapt to changing
protocols over

source:www.caida.org
time
Encapsulated
design of the
processing
modules
configuration
FPGA network processor
repository
dispatcher config. HTTP
VoIP
SSH
FTP
10
PR Advantages: Area Saving
Economics of ASIC- and FPGA designs

Source: Electronic News 16.03.2006

FPGA buyers: - reduce unit cost
- after sail business
FPGA vendors: more attractive for high volume designs
11
PR Advantages: Acceleration
Reduce latency by spending more area on submodules

A B C A
C
B
A
C
time
S0 A S0
S1 B latency S1 A B C A B C A
S2 C S2
t t
May alternatively allow to reduce clock frequency (and power)
Lower latency might reduce buffer sizes
Example: TLS/SSL, sorting (database acceleration)
May also increase throughput
12
PR Advantages: Faster Configuration
Full FPGA bistream can currently be > 20 MB
Flash memory performance 10-20 MB/s
(special high-speed Flash memories reach up to 100 MB/s)
Full initial configuration ~ 1-2 seconds in practice
an order of magnitude to slow for PCIe (setup within 100 ms)
Solution: Bootstrapping using PR

Initial config. from boot flash System config. via PCIe

conf. port conf. port

'empty'
empty FPGA 'empty'

PCIe core boot PCIe core boot

flash flash

13
 PR Advantages: IP Reuse
10 000 000
1 000 000
+58% / year
logic transistors/year 100 000
design gap 10 000
1000
100
+21% / year 10
productivity in tr. per man-month
1
1980 1985 1990 1995 2000 2005 2010
[International Technology Roadmap for Semiconductors]

High level of IP reuse Adapt the component-based system

PR design flow for a general design methodology
Idea: take as much as possible from an existing environment and
add only the application stecific parts. 14
 PR Advantages: SEU* Compensation
LUT-4 era LUT-6 era LUT-4 era LUT-6 era
# LUTs 1.2 M bitstream size ???

Stratix-IV
Virtex-6

Virtex-6
600 K 30 MB

Stratix-IV

Stratix-V
500 K 25 MB

Virtex-7
Stratix-V
Virtex-7
Stratix-III

Stratix-III
400 K 20 MB

Virtex-5
Virtex-5

Virtex-II Pro
Virtex-4
Virtex-II Pro

300 K 15 MB
Stratix-II

Stratix-II
Virtex-4
Virtex-II

Virtex-II
200 K 10 MB
Stratix

Stratix
100 K 5 MB

2000 2002 2004 2006 2008 2010 2000 2002 2004 2006 2008 2010
130 nm 90 nm 56 nm 40 nm 28 nm 130 nm 90 nm 56 nm 40 nm 28 nm

Smaller configuration SRAM cells

Exponetial rise in the total amount
Increased risc of *single event upsets (SEU)
Solution: Configuration Scrubbing
Continous reconfiguration during operation (repair)
Readback for SEU detection (before committing a result)
15
RC on FPGAs (Classification)
Classification of (run-time) reconfigurable FPGA-based systems
FPGA-based systems

one-time configurable reconfigurable

Actel SXA family (antifuse)
ASIC substitution*
global partial
older Altera FPGAs
in field update*
passive1 active2
* Typical use case Xilinx Spartan 3 Xilinx Virtex families
mode changing*
This lecture focuses on passive1 partial reconfiguration (interrupt
whole FPGA during reconfiguration) and active partial recon-
figuration2 (untouched parts continue execution) on FPGAs.
16
 Context-Switching on FPGAs
Partial reconfiguration is also referred as context switching.
What is the Context of an FPGA?
“Context” denotes a “state” which is stored in memory
Located in: 1) FPGA fabric (technology level)
2) Modules (logic level)
1) Present FPGA configuration 2) State of a module
• Register snapshot
• RAM blocks
• External state
Source: Christophe Bobda

Access via configuration port or

Access via configuration port
extra logic (e.g., scan-chain)
17
 Context-Switching on FPGAs
Classification Technology level (FPGA)
static dynamic
• Module runs forever • Configuration swapping
• Single configuration/ • Run-to-completion model
module context (no module context is
Logic level (module)

static • ASIC-like considered at start)

(e.g., memory controller) (e.g., motion-JPEG)

• Multiple module contexts • module preemption and

• on a single configuration resuming
• Configuration swapping
dynamic • Transparent (like software)
(e.g., multi channel crypto)
• Examined at UIO in the
COSRECOS project*

All variants may co-exist in a reconfigurable SoC

*Website: http://www.matnat.uio.no/forskning/prosjekter/crc 18
 Baseline Model of Partial Reconfiguration
The time-multiplex model: configuration data (bitstream)
internal configuration logic

FPGA phone
phone
video
<=> video
MP3
MP3

reconfigurable region surrounding system

Activate one module exclusively within a reconfigurable region

Swapping between modules by writing a partial bitstream to a
configuration port (defines the configuration time!)
Bitstream might be written by the FPGA itself selfreconfiguration
Used by the tools from Xilinx and Altera
19
 PR Time-Granularity (sub-cycle)
Tabula’s 3D Architecture
8 configuration planes
Reconfiguration @ 1.6 GHz
Within netlist reconfiguration
(uses forwarding registers
called „time via“)

8 folds @ 1.6 Ghz

200 MHz user clock

400 MHz user clock

20
 PR Time-Granularity (sub-cycle)
traditional FPGA implementation b3 a3 b2 a2 b1 a1 b0 a0
b31 a31 b1 a1 b0 a0
Example: VA VA VA VA
32- bit VA ... VA VA
s3 s2 s1 s0
adder s31 s1 s0
b7 a7 b6 a6 b5 a5 b4 a4
one memory access per time plane
(virtually 8 memory ports; holds time
VA VA VA VA via
only when not space folding)
s7 s6 s5 s4
Difficult to rate this approach:
extra multiplexer for plane

...

...
switching have to be b31 a31 b30 a30b29 a29 b28 a28
mapped on a 2D chip time
longer routing paths VA VA VA VA via
Difficult tools (manual ma- s31 s30 s29 s28
nipulation or simulation)
http://www.tabula.com
...
21
 PR Time-Granularity (single-cycle)
Multi-context FPGAs
originally proposed by
Scalera & Trimberger
single cycle confi-
guration swapping
idea: duplicating all
configuration bits
for each “plane”
and multiplexing between planes
Problem: extra multiplexer required for each configuration bit
All planes have to be mapped on a 2D chip (3D 2D mapping)
longer routing between the primitives <=> lower performance
Bad idea for FPGAs: most of the FPGA die area is spent on confi-
guration SRAM cells) usefull only for coarse-grained architectures
Better: multiplexing between different areas on the FPGA
22
 PR Time-Granularity (multi-cycle)
Configuration by writing a new configuration bitstream to the device

normal case for all FPGAs from Xilinx and Altera (starting with
the Stratix-5 family)

rapid partial module swapping

(e.g., swapping within a frame in a video processing system)

mode changing / field update

(typically used in combination with full FPGA reconfiguration,
e.g., in measurement equipment when changing settings
or for prototyping (ASIC emulation))

23
 PR in Time and Space
So far, we have only considered to have one module exclusively
placed with a reconfigurable region (temporal partial reconfigurati-
on) extension to multi-module placement of partially reconfigu-
rable modules (spatial partial reconfiguration)
Possibilities for tiling the reconfigurable area into resource slots:
island
a) island style style slot style
b) slot style c) grid grid
style style

m3

m4
m1 m2 m1 m2 m1 m2

m3
static part of the system unused reconfigurable area different modules
As smaller the slots, as lower the internal fragmentation (the waste
of logic resulting from fitting any sized module into a tile-grid (i.e.,
clustering the FPGA area into regular groups of resources)) 24
 PR in Time and Space: Efficiency
PR paradox: Runtime reconfiguration is brilliant, but not used!
internal fragmentation M

m1 m2 m3 m1 m2 m5 cconst
m4

m3
m4

m6
cconst
communication cost c M M M overhead
Internal fragmentation is dominating the overhead
Can be optimized with small slots 2D placement
(but might result in additional cost for the communication)
2D enhances BRAM/DSP utilization
2D is obligatory for newer FPGA Architectures (Virtex-5/6)
Requires adequate on-FPGA communication architectures
Buses
Point-to-point connections 25
 Optimal Resource Slot Size
Internal fragmentation results from fitting modules into a grid of
fixed resource slots.
Analog: storing files in a filesystem with fixed clusters
Average overhead of a module set of modules:
l : resources in a slot
c : communication
mi : resources of module i

– =

Optimal slot size depends on the modules and communication cost

26
 Optimal Resource Slot Size
Impact of the resource slot size and the communication cost on
the average module overhead
Scenario: 9701 modules with 300, 301, …, 10000 LUTs
with a communication cost of 0, 5, …, 25 LUTs per slot
average logic overhead

3000
2500
2000 0 5 10 15 20 25
1500
1000
500
0
50

50

50

50

50

50

50

50

50
0

0
25

45

65

85
10

12

14

16

18

20

22

24
resource slot size in terms of LUTs
Result: optimal slot size ~200–300 LUTs or ~25–40 CLBs
27
 Optimal Resource Slot Size
l : resources in a slot
c : communication
mi : resources of module i

Discussion:
If mi >> l the overhead converges to l/(l-c), meaning that for
large modules (with many resource slots) the internal
fragmentation becomes neglible.
The optimal slot size can be computed by differtentiating the
avarage module overhead with respect to the slot size l.
As the ceiling function is discontinuous, its bounds are
considered:
l l l

Lower bound: perfect fit

Upper bound: one slot is l
almost unused l
28
 Optimal Resource Slot Size
Discussion:
Upper bound: one slot is
almost unused l
l

l
Worst case: l
l

Avarage
case:
(achievable
only with 2D
grid style
placement)

29
Optimal Resource Slot Size
One of the best published solutions:
Hagemeyer et al., Design of
Homogeneous Communication
Infrastructures for Partially
Reconfigurable FPGAs
ERSA, USA 2007.
Master and slave support (32 bit)
16 sockets (XC2V4000)
Communication cost: 8554 LUTs
(~three 32-bit CPU-cores)
No I/O support
Resource slot size: 2560 LUTs

Catastrophic communication cost

and too large resource slots 30
 PR Space-Granularity (bitstream)
The behavior or structure of a system A3, A2, A1, A0
LUT-value LUT-value
AND gate OR gate
can be changed by small manipulations
0 OOOO 0 0
of the configuration bitstream. 1 OOO1 0 1
Manipulation of the routing 2 OO1O 0 1
(switch matrix multiplexer) 3 OO11 0 1
4 O1OO 0 1
Changing logic functions
5 O1O1 0 1
example: AND OR
6 O11O 0 1

0 7 O111 0 1
8 1OOO 0 1
1
9 1OO1 0 1
A 1O1O 0 1
LUT values
B 1O11 0 1
F C 11OO 0 1
D 11O1 0 1
A0
A1 E 111O 0 1
A2 Slice FF F 1111 1 1
A3
FF 31
 PR Space-Granularity (small modules)
Sometimes, even small modules can materially speed-up a system.
Example: reconfigurable customized instruction set extensions
(e.g., with instructions for CRC, DES round, bit swapping)
result result
a) b)
register file register file OP_B
OP_A
OP_A OP_B
instruction instruction conf. conf.
instr. instr.

Relatively small configurable instructions can speed up

execution by at least an order of magnitude.
(NIOS, GARP, DISC)
Typically non concurrent operation (blocking the ALU)
Difficulty: instructions have a high pin count per logic
Interfaces have to be ultra efficient!
Different logic requirements flexible instruction placement 32
 PR Space-Granularity (large modules)
Typically, systems consist of multiple concurrently working modules.
internal fragmentation M

m1 m2 m1 m2 m5 cconst

m3
m4

m3
m4

m6
cconst
communication cost c M M M overhead
Difficulty: modules have different resource requirements
Logic logic memory
Memory
reconfigurable
Multipliers L L L L M L L L L L L L L M L L FPGA region

Placement restrictions
placement
(string matching problem) options module
L LML

Interfaces should allow two-dimensional module placement

Further: placement impacts the communication!
33
 PR Space-Granularity (module coupling)
tightly coupled loosely coupled
register file cache system memory
complexity (size)
NIOS
[1] DISC

reg file memory

Reconfigurable HW
in parallel to the ALU

cache
ALU
bus

[1] Wirthlin and Hutchings:

DISC: Dynamic Instruction RHW
Set Computer (FCCM 1995)

34
 PR Space-Granularity (module coupling)
tightly coupled loosely coupled
register file cache system memory
complexity (size)
NIOS
NIOS II
[1] DISC

reg file memory

Reconfigurable HW
in parallel to the ALU

cache
ALU
bus
Module may contain
own register file
RHW

35
 PR Space-Granularity (module coupling)
tightly coupled loosely coupled
register file cache system memory
complexity (size)

[2] GARP
M.Blaze
NIOS
NIOS II
[1] DISC

reg file memory

Coprocessor-like
coupling of the

cache
ALU
bus
reconfigurable HW
[2] Hauser and Wawrzynek (FCCM 97):
GARP: A MIPS Processor with
RHW
a Reconfigurable Coprocessor

37
 PR Space-Granularity (module coupling)
tightly coupled loosely coupled
register file cache system memory
complexity (size)

[2] GARP
M.Blaze

PPC V4
NIOS
NIOS II
[1] DISC

reg file memory

Coprocessor-like
coupling of the

cache
ALU

reconfigurable HW bus

Decoupled by Fifo
channels (FSL-Fifo)
RHW
Parallel execution

38
 PR Space-Granularity (module coupling)
tightly coupled loosely coupled
register file cache system memory
complexity (size)

[2] GARP
M.Blaze

PPC V4
NIOS
NIOS II
[1] DISC

reg file memory

Connect reconfigurable
HW to the memory bus

cache
ALU
bus
Common FPGA-based
approaches require an interface interface

interface (in the easiest memory RHW RHW I/O

case a “bus-macro”)

40
 On-FPGA Communication
Goal: an efficient on-FPGA communication architecture that
supports the grid-style module placement.
Classification of different on-chip communication architectures:
On-Chip Communication

Point-to-point Interconnect Bus Network-on-Chip

Hierarchical
Custom Uniform Split bus Homogeneous Heterogene
shared Bus
source:

Custom Segmented Bus

for FPGAs: - buses (reading / writing of registerfiles and DMA)

- point-to-point links
(I/O-pin connection and data streaming)

41
 On-FPGA Communication (History)
Progress in Partial reconfiguration (physical implementation)
using the Xilinx tools over the last decade:
Fundamental problem: binding of the partial module entity sig-
nals to fixed routing resources of the FPGA fabric „module plug“
PR region static system

'
0' '
0' '
1' '
1' '
1' '
0' '
0' '
0' '
1' '
1' '
1' '
0'

NAND OR

„Xilinx Bus Macros“ for constraining the routing between the

static system and one or more PR regions (introduced 2002)
Costs two TBUFs per signal wire (in terms of latency and area)
Placement restrictions & device support
42
On-FPGA Communication (History)
Progress in Partial reconfiguration using the Xilinx tools
over the last decade:

"slice-based bus macro"

OR NAND
OR

„Slice-based Bus Macros“ (proposed by Hübner et al. in 2004)

More flexible (higher density of wires, more placement options)
Works with all Xilinx FPGAs (Virtex-II Pro: last FPGA with TBUFs)
Costs two LUTs per signal wire (in terms of latency and area)
43
On-FPGA Communication (History)
Progress in Partial reconfiguration using the Xilinx tools
over the last decade:

"proxy logic" "proxy logic"

OR NAND
OR

„Proxy logic“ (released for some devices by Xilinx in 2009)

Automatic placement of anchor primitives
Costs one LUT per signal wire (in terms of latency and area)
Only provided for some devices
Same approach is used in the upcoming Altera PR flow 44
 On-FPGA Communication
"PR link"

OR

NAND

„PR links“
Binding entity signals to the
wires crossing the border to
a reconfigurable module.
No logic overhead, cleaner design flow, supports S6 (V5, V6) 45
 On-FPGA Communication: Buses
Bus macros are best suited to integrate modules into islands!
The following slides present structured communication architec-
tures for slot-based (1D) or grid-style (2D) module placement

The Simple Formula for Building Bus-based

Reconfigurable Systems

46
ReCoBus Communication
All bus protocols can by implemented by the use of
four signal classes:
shared write shared read
dedicated write dedicated read

Master Slave 1 Slave 2

__
R\W
address
write_data
shared master read_data
write signals
interrupt_1 dedicated master
interrupt_2 read signals

select_1 dedicated master

shared master address write signals
decode select_2
read signals

Example: connecting an interrupt signal from a slave to an

interrupt controller is basically the same problem as connec-
ting a bus request from a master module to an arbiter 47
 ReCoBus: Shared Read
Shared read signals for connecting one selected module with the
static system:
a) module 0 module M-1 b) slot 0 slot 1 slot R-1
data_out . . . data_out data_out data_out fits into data_out
sel 0 sel1 one LUT selR-1
sel0 selM-1 d
& & &

master
master

& & u
m
≥1 ≥1 ≥1 m
≥1 y
... '
0'

Homogeneous (=identical) logic and routing footprint inside each

resource slot
Free module placement
Deep combinatory path (slow)
Massive resource overhead (has to replicated for each bit signal)
Only suitable for a coarse-grained placement grid
internal fragmentation 48
 ReCoBus: Interleaving
Problem: the structure of a distributed read multiplexer chain is
unlikely for very fine-grained resource slot layouts:
reconfigurable area
Slot 1 Slot 2 Slot 3

'
0'
D7..0
'
0'
D15..8
'
0'
D23..16
'
0'
D31..24

Logic overhead: 4/24 = 17%

49
 ReCoBus: Interleaving
Problem: the structure of a distributed read multiplexer chain is
unlikely for very fine-grained resource slot layouts:
reconfigurable area
SSlot
1 S 12 SSlot
3 S24 S Slot
5 S36 S Slot
7 S48 SlotS510 S 11
S9 SlotS612

'
0'
D7..0
'
0'
D15..8
'
0'
D23..16
'
0'
D31..24

Logic overhead: 4/6 = 66%; very high latency!

52
 ReCoBus: Interleaving
Solution: multiple interleaved read multiplexer chains

reconfigurable area
S1 S2 S3 S4 S5 S6 S7 S8 S 9 S 10 S 11 S 12

D31..24
D7..0 '
0'
D7..0
D
D15..8
D23..16
15..8

Low logic overhead, low latency and fine granularity!

53
 ReCoBus: Signal Alignment (1D)
Example system:

Module 0 Module 1
S1 S2 S3 S4 S5 S6 S7 S8
CPU en dout en dout en dout en dout en dout en dout
& & & & & &
≥1 ≥1 1 1 ≥1 ≥1 ≥1 ≥1

static system runtime reconfigurable system

Alignment-multiplexer allow free module placement

Interface grows together with the module complexity (size)
For example: a small UART might be connected using an 8-bit
data bus and a more complex Ethernet adapter with 32-bit
The first LUT function of each chain (here rightmost) must be
changed to an AND gate or an external source is needed 54
ReCoBus: Signal Alignment (1D)
Assuming an 8-bit interface pro slot, it takes at least four
consecutive slots to provide the full interface size
m1
0 1 2 3 0 1 2 3

0 start point & mux select value 0123 0123 0123 0123
0 used connection
0 unused connection D31...D24 D23...D16 D15...D8 D7...D0 55
 ReCoBus: Signal Alignment (2D)
The signal interleaving scheme can be extended to implement
buses allowing to integrate modules in a 2D grid style.
y m1
slot indexing: sloty,x 0 1 2 3 0 1 2 3
0

m2
1 1 2 3 0 1 2 3 0

m3
2 2 3 0 1 2 3 0 1

m4
3 3 0 1 2 3 0 1 2

≥1 ≥1 ≥1 ≥1 0 1 2 3 4 5 6 7
x

slot3,6
0 start point & mux select value 0123 0123 0123 0123
0 used connection
0 unused connection D31...D24 D23...D16 D15...D8 D7...D0
56
 ReCoBus: Dedicated Write Signals
LUTs can be used to decode an address within the bus value

(the table contains then a one-hot value, e.g. for addr. 0xA) 0 0
1 0
LUT values 2 0
LUT in 3 0
Sin
0 0 SRL16 mode 4 0
1 1 5 0
6 0

...
7 0
F F 8 0

A0 9 0
A0
A1 A1 A 1
A2 Slice FF A2
A3 B 0
A3
Sout C 0
FF FF
D 0
For setting an address, LUT values can be exchanged: E 0
Using the configuration port, F 0

Accessing the table with the user logic:

SRL16 shift register primitive or distributed memory 57
 ReCoBus: Dedicated Write Signals
Architecture: uniformed distributed address comparator inside
the bus (implemented by SRL16 shift register primitives)
fits into one
00 0 00 0 0 0 0 0 1 0 0 0 0 0 11 1 11 1 1 1 1 1 1 1 1 1 1 1 look-up table
EN
config_data

Q15
Din

Q0
config_clock module_reset
bus_enable 4
module_select

bus_read
& module_read

bus side reconfigurable select generator module side

Two-stage reconfiguration:
1. FPGA: initialize the shift register with 0xFFFF
2. Logic: configure address comparator and activate module
58
 ReCoBus: Dedicated Write Signals
Arrangement of the address comparators

module_select
module_reset

module side
module_read
look-up table
fits into one
module 1 module 2
slot 0 slot 1 slot 2 slot 3 slot 4 slot 5

&

reconfigurable select generator

read_en

read_en
CPU
select

select
reset

reset
Q15

module module
select select
bus_enable logic config_data logic bus_read
config_clock d
u
bus m
logic m
y
Q0

EN

4
Din
Allows module relocation

config_clock

bus_read
config_data

bus_enable
Multiple instances of a module

bus side
(individual module addresses)
Automatic reset generation
No interference by the reconfiguration process (Hot-Plug)
Extra register file look-up for alignment multiplexer control 60
ReCoBus: Dedicated Write Signals
Assuming an 8-bit interface pro slot, it takes at least four
consecutive slots to provide the full interface size
select 9876543210

15
14
13
12
11
10
master A7...A0
module F F reserved
select E E module_selectE
logic D D
A15...A12

A11...A8

A7...A0 C C
B B
A A FF
... 9 9 FE
8 8 module
&

...
7 7 register
≥1 6 6
bus_enable 5 5 file
01
≥1 ... 4 4 00
3 3
≥1 2 2
module 1 1 1
≥1 0 0 module_select0

Address mapping: the whole ReCoBus subsystem appears like one

module in the address space of the system
Up to 15 modules can be addressed (one encoding (0xF--) is used
for the case that no module is selected)
Wildcard addressing for multi cast operation (wired OR on read) 61
 ReCoBus: Dedicated Read Signals
Dedicated master read signals (interrupt)

module 1 module 2
slot 0 slot 1 slot 2 slot 3 slot 4 slot 5
IRQ dummy IRQ dummy
CPU sink sink

1 d
0 u
m
m
y

Idea: set connection within a module to an internal homogenously

routed interrupt wire by bitstream manipulation
The number of internal interrupt lines scales with the number of
modules (allows many tiny slots)
Crosspoints are directly implemented in the FPGA routing fabric
(no extra logic required)
In practice: internal wire sharing for interrupt and bus arbitration
(also: signal interleaving and masking in the static system) 62
ReCoBus Properties
Direct connection of a module to the bus
Compatible to all established standards (AMBA, PLB, …)
Module relocation & flexible module placement
Variable module sizes
Multiple instances of the same module
Very low logic overhead
Allows high speed / high throughput
Hot-swap module exchange: The reconfiguration process
is completely transparent for all bus transactions.

63
I/O-Bars

OK - We have a suitable Bus.

What about dedicated links or I/O?

64
I/O-Bars for Point-to-Point Links
Horizontal routing track within the reconfigurable area
Connections are set by modifying switch matrices
One bar per interface requirement (e.g., video, audio)
bypass
static system
Slot 0 Slot 1 Slot 2 Slot 3 Slot 4 Slot 5
video video
out in

audio audio
out in

ReCoBus

static system
65
I/O-Bars for Point-to-Point Links
• Read-modify-write connection
• Ideal for data streaming

static system
Slot 0 Slot 1 Slot 2 Slot 3 Slot 4 Slot 5
video video
out in

audio audio
out in

ReCoBus

static system
66
I/O-Bars for Point-to-Point Links
• I/O bar implementation

Incoming signals Route through signals

Outgoing signals

67
I/O-Bars for Point-to-Point Links
I/O-Bar implementation for 2D
Vertical routing is accomplished in the static part
Can be used with interleaving for decreasing latency
(requires signal alignment in each module)

68
Demo System
248 logic slots
(192 LUTs/slot)
+16 RAM slots
8-bit slave bus
(up to 48 bit via
6 sequent slots)
Video streaming
Free placement
Connection cost:
14 LUTs/slot
100 MHz
(XC2V6000-6)

69
Demo System
Regular structured ReCoBus macro (a macro contains logic
and routing and is instantiated like any other VHDL module)

Implementation on a
XC2V-6000
One CLB provides up
to 8 data signals
(for read and write)
Lower CLB packing can
improve routing (congestion around the connecting resources
70
 Design Flow
ReCoBus & connection bar protocol specification
Design Entry, Static/Dynamic
[ReCoBus-Builder]

Partitioning, and Verification

design entry, static /dynamic

partitioning, and verification
bus & bar static module1
module
module
RTL model system 11

functional simulation [Modelsim] ReCoBus & connection bar protocol specification

n
[ReCoBus-Builder]
OK?

physical implementation
budgeting floorplanning and communication budgeting
[Xilinx XST] synthesis [ReCoBus-Builder] [Xilinx XST] bus & bar static module1
module
module
static static ReCoBus module1
RTL model system 11
module1 module 1
netlist constraints I/O bars constraints templates
netlist

place & route static [PAR] buildpartial

build partial module
module
place&route module 1 [PAR]
11
functional simulation [Modelsim]
build static bitstream [bitgen] build module1 bitstream [bitgen]
bitstream assembly

static system module

fullmodule
module 11 n
bitfile bitfile 1
bitfile
bitfile OK?
partial bitstream extraction
bitstream linking [bitscan]
[bitscan]

initial system module

module
partial module 11
bitfile bitfile 1
bitfile
bitfile
repository for the run-time system

[ ] novel tool [ ] third party or vendor tool

72
 Design Flow
ReCoBus & connection bar protocol specification
Physical Implementation
[ReCoBus-Builder]

design entry, static /dynamic

partitioning, and verification
bus & bar static module1
module
module
RTL model system 11

functional simulation [Modelsim] OK?

n
OK?

floorplanning and
budgeting budgeting

physical implementation
budgeting floorplanning and communication budgeting communication synthesis
[Xilinx XST] synthesis [ReCoBus-Builder] [Xilinx XST] [Xilinx XST] [Xilinx XST]
[ReCoBus-Builder]
static static ReCoBus module1 module1
module 1
netlist constraints I/O bars constraints templates
netlist
static static ReCoBus module1 module1
module 1
place & route static [PAR] buildpartial
build
place&routepartial
modulemodule
module
1 [PAR]
11
netlist constraints I/O bars constraints templates
netlist
build static bitstream [bitgen] build module1 bitstream [bitgen]
bitstream assembly

static system module

fullmodule place & route static [PAR] buildpartial
build
place&routepartial
modulemodule
module
[PAR]
module 11
bitfile 1 1 11
bitfile bitfile
bitfile

partial bitstream extraction

bitstream linking [bitscan]
[bitscan]

initial system module

module
partial module 11
bitfile bitfile 1
bitfile
bitfile
repository for the run-time system

[ ] novel tool [ ] third party or vendor tool

73
 Design Flow
ReCoBus & connection bar protocol specification
Bitstream Assembly
[ReCoBus-Builder]

design entry, static /dynamic

partitioning, and verification
bus & bar static module1
module
module
RTL model system 11

functional simulation [Modelsim] build static bitstream [bitgen] build module1 bitstream [bitgen]
n
OK?
static system module
fullmodule
module 111
bitfile bitfile
bitfile
bitfile

physical implementation
budgeting floorplanning and communication budgeting
[Xilinx XST] synthesis [ReCoBus-Builder] [Xilinx XST]

static static ReCoBus module1 module1

module partial bitstream extraction
netlist constraints I/O bars templates
1
bitstream linking [bitscan]
constraints netlist
[bitscan]
place & route static [PAR] buildpartial
build
place&routepartial module
module
module 1 [PAR]
11

initial system module

module
partial module
build static bitstream [bitgen] 11 1
build module1 bitstream [bitgen]
bitfile bitfile
bitfile
bitfile
bitstream assembly

static system module

fullmodule
module 11
bitfile bitfile 1
bitfile
bitfile

partial bitstream extraction

bitstream linking [bitscan]
[bitscan]

initial system module

module
partial module 11
bitfile bitfile 1
bitfile
bitfile
repository for the run-time system

[ ] novel tool [ ] third party or vendor tool

74
Design Flow
Tested Design
floorplanning and
budgeting budgeting
communication synthesis
[Xilinx XST] [Xilinx XST]
[ReCoBus-Builder]

static static ReCoBus module1 module1

module 1
netlist constraints I/O bars constraints templates
netlist

place & route static [PAR] buildpartial

build
place&routepartial module
module
module 1 [PAR]
11

build static bitstream [bitgen] build module1 bitstream [bitgen]

static system module

fullmodule
module 111
bitfile bitfile
bitfile
bitfile

partial bitstream extraction

bitstream linking [bitscan]
[bitscan]

initial system module

module
partial module 11 1
bitfile bitfile
bitfile
bitfile
repository for the run-time system
bitlink module.bit X Y \
static.bit initial.bit 75
Design Flow
Modules might be implemented using different
shapes/resources (design alternatives)
Goal: higher utilization
Interesting for com-
ponent based system
design (no place and
route)
Simplified system
integration based on
standardized interfaces
Enhanced IP-reuse
2 multiplier,
logic only 2 multiplier, 6 logic slots
30 slots 6 logic slots (includes gap)

76
Design Flow: Blocking

77
Design Flow: Xilinx PlanAhead
New advanced GUI for the complete FPGA design flow

Project management

Floorplanning

Critical path analysis

(timing)

Implementation viewer
Source: Xilinx

Integration of the vendor specific partial flow

78
Design Flow: Xilinx PlanAhead
1. Step: Synthesis of all partial and static modules in
individual netlists
(Static netlist has black boxes for the modules)
2. Step: Creation of a new
PlanAhead project
3. Step: Creation of
Reconfigurable Partitions
A reconfigurable partition
(RP) consists of several
reconfigurable modules
(RM)
Assign a partial netlist to
each RM
A RM can also be a black
box (empty module)
79
 Design Flow: Xilinx PlanAhead
4. Step: Floor planning of the reconfigurable partitions
Create Area Groups
PlanAhead automatically creates the communication ports
for the reconfigurable partition

Port proxy logic:

LUT1 (anchor re-
quired for physical
implementation)

PlanAhead
automatically
creates the user
constraints file
(UCF) with the
bounding box Source: Xilinx
definitions of the RPs 80
 Design Flow: Xilinx PlanAhead
5. Step: Run design rule check (DRC) to verify the design
6. Step: Create the first reconfigurable configuration
Consisting of the static module and for each RP a RM
Implement this configuration
Promote this configuration
7. Step: Create further
configurations
for each module in a RP:
Import the static design
Implement the partial
module

8. Step: Create the static

and partial configuration bitfiles

81
 Design Flow: Xilinx PlanAhead
Differences between the ReCoBus-Builder approach and PlanAhead:
Slot-style or grid-style vs. island style reconfiguration (island style
has no external fragmentation problem simple placement)
ReCoBus allows module relocation
"proxy logic"
and multi module instantiation

Proxy logic bounds a module to a

particular fixed region (RP)
Example: 3 islands and 4 kinds of
NAND
OR
modules requires 3x4=12 physical
implementations (place&route)
All partial modules have to be re-implemented in case of changes
in the static system (does not scale for complex systems)
82
 Design Flow: FPGA Issues
In Xilinx FPGAs, the smallest atomic piece of configuration data is a
configuration frame that contains data for all (older devices) or a set
of vertical aligned CLBs (newer devices)
Arbitrary configuration update is possible using
readback-modify-write

Instead of readback, a configuration image might be stored in

memory to avoid the relatively slow readback process
Warning: Using LUTs as memory elements (e.g., SRL16 mode)
might result in side effects, when updating modules above or
below these primitives because LUT values get overwritten. 83
 Run-time Management
Main problem: online temporal module placement
Problem: map a DFG
G = (V,E)
onto a reconfigurable v1 v2
area such that the
schedule is feasible
v3 v4
and the total execution
time is minimized.
v5

In other words:
computing module placement positions
and schedules

Question: predictable (offline) vs. unpredictable (oline) problem

84
 PR example: Sorting for Database Acceleration
Sorting contributes to 30% of the CPU time in huge databases
input: unsorted data stream

mem-contr.
FPGA initial step: fully sorted sequences

DDR3
[intermediate steps]: merging
2GB/s PCIe
final step: merge and emit result
8x
MEM prefetcher
unsorted stream

A
max burst size
B
> > > > max latency
C A B C D
> >
D FPGA >
FPGA
sorted output
initial step context switching final step

50% area saving or 4 times larger problems as compared to a static design

Next step: hierachical reconfiguration: swap comparator
cells for different data types (integer, text, …) 85
 PR example: Custom instructions
Fine-grained communication architecture for flexible instruction
placement
result OP_A
register file OP_B OP_B
OP_A B A
instruction
instruction conf. conf.
instr. instr. register
file

Identical routing for OPs and results in each slot

Both operands are available in each slot
(end point & middle access)
Commutative instructions (e.g., A > B)
Implementation alternative
Bitstream manipulation
86
 PR example: Custom instructions
instruction slices slots bitstream latency (max/av)
64-bit XOR gate 19 (40%) 1 2.64 KB 7.04 / 5.95 ns
CCITT CRC 33 (34%) 2 5.28 KB 5.32 / 3.98 ns
sat. add/sub 70 (73%) 2 5.28 KB 9.89 / 7.81 ns
barrel shifter 90 (94%) 2 5.28 KB 11.07 / 7.88 ns
'
1' -bit counter 214 (89%) 5 13.2 KB 11.37 / 8.25 ns
mask & permute 16 (33%) 1 2.64 KB 5.94 / 4.05 ns

Direct connection (no „proxy logic“)

Swapping of instructions:
Dedicated load commands
Triggered by a trap handler

87