Optimize SQL in PL/SQL

Steven Feuerstein
PL/SQL Evangelist
www.quest.com steven.feuerstein@quest.com
Copyright © 2006 Quest Software
How to benefit most from this session
• Watch, listen, ask questions.
• Download the training materials and supporting
scripts:
– http://oracleplsqlprogramming.com/resources.html
– "Demo zip": all the scripts I run in my class available at
http://oracleplsqlprogramming.com/downloads/demo.zip
filename_from_demo_zip.sql

• Use these materials as an accelerator as you

venture into new territory and need to apply new
techniques.
• Play games! Keep your brain fresh and active by
mixing hard work with challenging games.
– I strongly recommend Set (www.setgame.com)

2
Optimize SQL in PL/SQL programs
• Take advantage of PL/SQL-specific
enhancements for SQL.
– BULK COLLECT and FORALL
– Table functions
• Top Tip: Stop writing so much SQL
– A key objective of this presentation is to have you stop taking
SQL statements for granted inside your PL/SQL code.
– Instead, you should think hard about when, where and how SQL
statements should be written in your code.

3
What is wrong with this code?
• Time to “process” 5,000 employees in a department....
CREATE OR REPLACE PROCEDURE process_employee (
department_id IN NUMBER)
IS
l_id INTEGER;
l_dollars NUMBER; For a particular
l_name VARCHAR2 (100); department ID, get all
/* Full name: LAST COMMA FIRST (ReqDoc 123.A.47) */ the employees,
CURSOR emps_in_dept_cur IS construct the “full
SELECT employee_id, salary
, last_name || ',' || first_name lname name” and update the
FROM employees salary.
WHERE department_id = department_id;
BEGIN
OPEN emps_in_dept_cur;

LOOP
FETCH emps_in_dept_cur INTO l_id, l_dollars, l_name;
I found at least 15
analyze_compensation (l_id, l_dollars); items that needed fixing
UPDATE employees SET salary = l_salary
(not all of them SQL-
WHERE employee_id = employee_id; related!).
EXIT WHEN emps_in_dept_cur%NOTFOUND;
END LOOP;
END; 4
wrong_code.sql
Turbo-charged SQL with
Bulk Processing Statements
• Improve the performance of multi-row SQL
operations by an order of magnitude or
more with bulk/array processing in PL/SQL!
CREATE OR REPLACE PROCEDURE upd_for_dept (
dept_in IN employee.department_id%TYPE
,newsal_in IN employee.salary%TYPE)
IS
CURSOR emp_cur IS
SELECT employee_id,salary,hire_date
FROM employee WHERE department_id = dept_in;
BEGIN
FOR rec IN emp_cur LOOP

adjust_compensation (rec, newsal_in);

Row by row
UPDATE employee SET salary = rec.salary
WHERE employee_id = rec.employee_id;
processing of data
END LOOP;
END upd_for_dept; 5
Underneath the covers: SQL and PL/SQL
Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block
Procedural
statement
FOR rec IN emp_cur LOOP executor
UPDATE employee SQL
SET salary = ... statement
WHERE employee_id = executor
rec.employee_id;
END LOOP;

Performance penalty
for many “context
switches”
6
A different process with FORALL
Oracle server

PL/SQL Runtime Engine SQL Engine

PL/SQL block
Procedural
statement
FORALL indx IN executor
list_of_emps.FIRST.. SQL
list_of_emps.LAST statement
UPDATE employee executor
SET salary = ...
WHERE employee_id =
list_of_emps(indx);

Much less overhead

for context switching
7
Bulk Processing in PL/SQL
• You should consider moving all multi-row
SQL processing over to these new
approaches.
• BULK COLLECT
– Use with implicit and explicit queries.
– Move data from tables into collections.
• FORALL
– Use with inserts, updates and deletes.
– Move data from collections to tables.

8
Use BULK COLLECT INTO for Queries
DECLARE
Declare a TYPE employees_aat IS TABLE OF employees%ROWTYPE
collection of INDEX BY BINARY_INTEGER;
records to hold
l_employees employees_aat;
the queried data. BEGIN
SELECT *
bulkcoll.sql
BULK COLLECT INTO l_employees
bulktiming.sql
Use BULK FROM employees;
COLLECT to
FOR indx IN 1 .. l_employees.COUNT
retrieve all rows. LOOP
process_employee (l_employees(indx));
END LOOP;
END;
Iterate through
the collection
contents with a
loop. WARNING! BULK COLLECT will not raise
NO_DATA_FOUND if no rows are found.
Always check contents of collection to confirm that
something was retrieved.
9
Limit the number of rows returned by
BULK COLLECT
CREATE OR REPLACE PROCEDURE bulk_with_limit
(deptno_in IN dept.deptno%TYPE)
IS
CURSOR emps_in_dept_cur IS
SELECT *
FROM emp
WHERE deptno = deptno_in;

TYPE emp_tt IS TABLE OF emps_in_dept_cur%ROWTYPE;

emps emp_tt;
BEGIN
OPEN emps_in_dept_cur; Use the LIMIT clause with the
LOOP INTO to manage the amount of
FETCH emps_in_dept_cur memory used with the BULK
BULK COLLECT INTO emps
COLLECT operation.
LIMIT 1000;

EXIT WHEN emps.COUNT = 0; Definitely the preferred

approach in production
process_emps (emps); applications with large or
END LOOP; varying datasets.
END bulk_with_limit;
bulklimit.sql
10
Tips and Fine Points for BULK COLLECT
• Can be used with implicit and explicit cursors
• Collection is always filled sequentially, starting at
row 1.
– So you are always safe using a FOR loop to iterate through, as shown
on previous page.
• Production-quality code should generally use the
LIMIT clause to avoid excessive memory usage.
• Note: Oracle will automatically optimize cursor FOR
loops to BULK COLLECT performance levels.
– But it will not be sufficient if the cursor FOR loop contains DML
statements!

emplu.pkg
10g_optimize_cfl.sql 11
The FORALL Bulk Bind Statement
• Instead of executing repetitive, individual DML
statements, you can write your code like this:
PROCEDURE upd_for_dept (...) IS
BEGIN
FORALL indx IN list_of_emps.FIRST .. list_of_emps.LAST
UPDATE employee
SET salary = newsal_in
WHERE employee_id = list_of_emps (indx);
END;

• Things to be aware of:

– You MUST know how to use collections to use this feature!
– Only a single DML statement is allowed per FORALL.
– New cursor attributes: SQL%BULK_ROWCOUNT returns number of rows
affected by each row in array. SQL%BULK_EXCEPTIONS...
– Prior to Oracle10g, the binding array must be sequentially filled.
– Use SAVE EXCEPTIONS to continue past errors.

bulktiming.sql
bulk_rowcount.sql 12
Better Exception Handling Oracle9i
for Bulk Operations
• Allows you to continue past errors and
obtain error information for each individual
operation (for dynamic and static SQL).
CREATE OR REPLACE PROCEDURE load_books (books_in IN
book_obj_list_t)
IS
bulk_errors EXCEPTION;
PRAGMA EXCEPTION_INIT ( bulk_errors, -24381 );
BEGIN
Allows processing of
FORALL indx IN books_in.FIRST..books_in.LAST
SAVE EXCEPTIONS all rows, even after an
INSERT INTO book values (books_in(indx)); error occurs.
EXCEPTION
WHEN BULK_ERRORS THEN
FOR indx in 1..SQL%BULK_EXCEPTIONS.COUNT New cursor
LOOP attribute, a
log_error (SQL%BULK_EXCEPTIONS(indx)); pseudo-collection
END LOOP;
END;
13
bulkexc.sql
Tips and Fine Points for FORALL
• Use whenever you are executing multiple
single-row DML statements.
– Oracle suggests you will see benefit with 5 or more rows.
• Can be used with any kind of collection.
• Collection subscripts cannot be expressions.
• You cannot reference fields of collection-
based records inside FORALL.
– But you can use FORALL to insert and update entire records.
• Prior to Oracle10g Release 2, the bind
collections must be densely filled.
– The newer VALUES OF and INDICES OF clauses give you
added flexibility (coming right up!).
14
Collections impact on "Rollback segment too
small" and "Snapshot too old" errors

• Rollback segment too small...

– Cause: so many uncommitted changes, the rollback segment
can't handle it all.
– FORALL will cause the error to occur even sooner. Use a
variation on incremental commits with FORALL.
• Snapshot too old...
– Cause: a cursor is held open too long and Oracle can no longer
maintain the snapshot information.
– Solution: open-close cursor, or use BULK COLLECT to retrieve
information more rapidly.

forall_incr_commit.sql
15
Cursor FOR Loop ... or BULK COLLECT?

• Why would you ever use a cursor FOR loop

(or other LOOP) now that you can perform a
BULK COLLECT?
– If you want to do complex processing on each row as it is queried
– and possibly halt further fetching.
– You are not executing DML within your cursor FOR loop and you
are on Oracle Oracle Database 10g – Oracle will automatically
optimize the code for you.
• Otherwise, moving to BULK COLLECT is a
smart move!

cfl_vs_bulkcollect.sql
cfl_to_bulk.sql
16
More flexibility for FORALL Oracle10g

• In Oracle10g, the FORALL driving array no

longer needs to be processed
sequentially.
• Use the INDICES OF clause to use only the
row numbers defined in another array.
• Use the VALUES OF clause to use only the
values defined in another array.

17
Using INDICES OF
DECLARE
TYPE employee_aat IS TABLE OF employee.employee_id
%TYPE
INDEX BY PLS_INTEGER;
• It only l_employees employee_aat;
TYPE boolean_aat IS TABLE OF BOOLEAN
processes INDEX BY PLS_INTEGER;
l_employee_indices boolean_aat;
the rows BEGIN
l_employees (1) := 7839;
with row l_employees (100) := 7654; 10g_indices_of.sql
10g_indices_of2.sql
numbers l_employees (500) := 7950;
--
matching l_employee_indices (1) := TRUE;
l_employee_indices (500) := TRUE;
the defined l_employee_indices (799) := TRUE
rows of the --
FORALL l_index IN INDICES OF l_employee_indices
driving BETWEEN 1 AND 500
UPDATE employee
array. SET salary = 10000
WHERE employee_id = l_employees (l_index);
END;
18
Using VALUES OF
DECLARE
TYPE employee_aat IS TABLE OF employee.employee_id
%TYPE
INDEX BY PLS_INTEGER;
• It only l_employees employee_aat;

processes TYPE values_aat IS TABLE OF PLS_INTEGER

the rows INDEX BY PLS_INTEGER;
l_employee_values values_aat;
with row BEGIN
numbers l_employees (-77) := 7820;
l_employees (13067) := 7799;
matching l_employees (99999999) := 7369;
the content --
l_employee_values (100) := -77;
of a row in l_employee_values (200) := 99999999;
--
the driving FORALL l_index IN VALUES OF l_employee_values
array. UPDATE employee
SET salary = 10000
WHERE employee_id = l_employees (l_index);
END;

10g_values_of.sql 19
The Wonder Of Table Functions
• A table function is a function that you can call
in the FROM clause of a query, and have it be
treated as if it were a relational table.
• Table functions allow you to perform
arbitrarily complex transformations of data
and then make that data available through a
query.
– Not everything can be done in SQL.
• Combined with REF CURSORs, you can now
more easily transfer data from within PL/SQL
to host environments.
– Java, for example, works very smoothly with cursor variables

20
Building a table function
• A table function must return a nested table
or varray based on a schema-defined type.
– Types defined in a PL/SQL package can only be used with
pipelined table functions.
• The function header and the way it is
called must be SQL-compatible: all
parameters use SQL types; no named
notation.
– In some cases (streaming and pipelined functions), the IN
parameter must be a cursor variable -- a query result set.

21
Simple table function example
• Return a list of names as a nested table, and
then call that function in the FROM clause.

CREATE OR REPLACE FUNCTION lotsa_names (

base_name_in IN VARCHAR2, count_in IN INTEGER
)
RETURN names_nt
SELECT column_value
IS
FROM TABLE (
retval names_nt := names_nt ();
lotsa_names ('Steven'
BEGIN
, 100)) names;
retval.EXTEND (count_in);
COLUMN_VALUE
FOR indx IN 1 .. count_in
LOOP
retval (indx) :=
------------
base_name_in || ' ' || indx;
Steven 1
END LOOP;

RETURN retval;
...
END lotsa_names;
tabfunc_scalar.sql 22

Steven 100
 Streaming data with table functions
• You can use table functions to "stream" data through
several stages within a single SQL statement.
– Example: transform one row in the stocktable to two rows in the tickertable.

CREATE TABLE stocktable (

ticker VARCHAR2(20),
trade_date DATE,
open_price NUMBER,
close_price NUMBER
)
/
CREATE TABLE tickertable (
ticker VARCHAR2(20),
pricedate DATE,
tabfunc_streaming.sql
pricetype VARCHAR2(1),
price NUMBER)
/ 23
Streaming data with table functions - 2
• In this example, transform each row of the
stocktable into two rows in the tickertable.
CREATE OR REPLACE PACKAGE refcur_pkg
IS
TYPE refcur_t IS REF CURSOR
RETURN stocktable%ROWTYPE;
END refcur_pkg;
/

CREATE OR REPLACE FUNCTION stockpivot (dataset refcur_pkg.refcur_t)

RETURN tickertypeset ...

BEGIN
INSERT INTO tickertable
SELECT *
FROM TABLE (stockpivot (CURSOR (SELECT *
FROM stocktable)));
END;
/

tabfunc_streaming.sql 24
Use pipelined functions to enhance
performance.
CREATE FUNCTION StockPivot (p refcur_pkg.refcur_t)
RETURN TickerTypeSet PIPELINED

• Pipelined functions allow you to return data

iteratively, asynchronous to termination of
the function.
– As data is produced within the function, it is passed back to the
calling process/query.
• Pipelined functions can only be called within
a SQL statement.
– They make no sense within non-multi-threaded PL/SQL blocks.

25
Applications for pipelined functions
• Execution functions in parallel.
– In Oracle9i Database Release 2 and above, use the
PARALLEL_ENABLE clause to allow your pipelined function to
participate fully in a parallelized query.
– Critical in data warehouse applications.
• Improve speed of delivery of data to web
pages.
– Use a pipelined function to "serve up" data to the webpage and
allow users to being viewing and browsing, even before the
function has finished retrieving all of the data.

26
Piping rows out from a pipelined
function
CREATE FUNCTION stockpivot (p refcur_pkg.refcur_t)
RETURN tickertypeset
Add PIPELINED
keyword to header PIPELINED
IS
out_rec tickertype :=
tickertype (NULL, NULL, NULL);
in_rec p%ROWTYPE;
BEGIN
LOOP
FETCH p INTO in_rec;
EXIT WHEN p%NOTFOUND;
out_rec.ticker := in_rec.ticker;
Pipe a row of data out_rec.pricetype := 'O';
back to calling block out_rec.price := in_rec.openprice;
or query
PIPE ROW (out_rec);
END LOOP;
CLOSE p;
RETURN...nothing at
all! RETURN;
END;

tabfunc_setup.sql 27
tabfunc_pipelined.sql
 Enabling Parallel Execution

• You can use pipelined functions with the Parallel

Query option to avoid serialization of table function
execution.
• Include the PARALLEL_ENABLE hint in the program
header.
– Choose a partition option that specifies how the function's execution
should be partitioned.
– "ANY" means that the results are independent of the order in which the
function receives the input rows (through the REF CURSOR).

{[ORDER | CLUSTER] BY column_list}

PARALLEL_ENABLE ({PARTITION p BY
[ANY | (HASH | RANGE) column_list]} )
28
Table functions - Summary
• Table functions offer significant new
flexibility for PL/SQL developers.
• Consider using them when you...
– Need to pass back complex result sets of data through the
SQL layer (a query);
– Want to call a user defined function inside a query and
execute it as part of a parallel query.

29
Top Tip: Stop writing so much SQL!
"Why does Steven make such a
big deal about writing SQL
inside PL/SQL? It's a no-
brainer in PL/SQL, the last
thing we have to worry about!"

• I moan and groan about SQL because it is the

"Achilles Heel" of PL/SQL.
– It's so easy to write SQL, it is too easy.
– The result is that most programmers take SQL totally for
granted and write it whenever and wherever they need. Bad
idea!
– Let's see why I say this....
30
Why We Write PL/SQL Code
• PL/SQL is an embedded language. Its purpose is to
provide high-speed, easy access to underlying
datasets.
• Those datasets are always changing – both the data
and the structure of the tables.
– So the code should not get in the way of that change, but should support
and enable that change.
Order Entry
Program

Bottom line: if everyone writes SQL

whenever and wherever they want
to, it is very difficult to maintain and Order Item
optimize the code. Table Table

31
 Single Point of (SQL) Robust Definition
• General principle: figure out what
is volatile and then hide that stuff
behind a layer of code to minimize
Application
impact on application logic. Code
• For SQL: if the same statement
appears in multiple places in Intermediate Layer
application code, very difficult to
maintain and optimize that code.
• So we should avoid repetition at all Order Item
Table Table
costs!
• But how....???

32
How to Avoid SQL Repetition

• You should, as a rule, not even

write SQL in your PL/SQL

SQL
programs
– You can't repeat it if you don't write it

• Instead, rely on pre-built, pre-

tested, written-once, used-often
PL/SQL programs.
– "Hide" both individual SQL statements
and entire transactions.
– And revoke privileges on tables!

33
About comprehensive table APIs
• Many (not all!) of the SQL statements we need to
write against underlying tables and views are very
common and predictable.
– Get me all rows for a foreign key.
– Get me one row for a primary key.
– Insert a row; insert a collection of rows.
• Why write these over and over? Instead, rely on a
standard, preferably generated, programmatic
interface that takes care of this "basic plumbing."

SOA for PL/SQL

Developers! The Quest CodeGen Utility
SQL is a service. www.qcgu.net

Error mgt is a service.

34
Clear benefits of encapsulated SQL
• Change/improve implementation without
affecting application layer of code.
– Switch between types of queries (implicit vs explicit)
– Take advantage of data caching, bulk processing, SQL
enhancements like MERGE.
• Consistent error handling
– INSERT: dup_val_on_index?
– SELECT: too_many_rows?
– Much less likely to be ignored when the developer writes
SQL directly in the application.

35
Example: Quest Code Tester backend
• For each table, we have
three generated packages:
– <table>_CP for DML
– <table>_QP for queries
– <table>_TP for types
• And for many an "extra
stuff" package with custom
SQL logic and related
code:
– <table>_XP

qu_outcome_xp.qu_outcomes
qu_outcome_xp.int_create_outcomes
36
Let’s correct that “wrong code.” • Page 1 of 2

CREATE OR REPLACE PROCEDURE adjust_compensation (

department_id_in IN employees.department_id%TYPE
)
IS
TYPE id_aat IS TABLE OF employees.employee_id%TYPE
INDEX BY PLS_INTEGER;

l_employee_ids id_aat;

TYPE salary_aat IS TABLE OF employees.salary%TYPE

INDEX BY PLS_INTEGER;

l_salaries salary_aat;

TYPE fullname_aat IS TABLE OF employee_rp.fullname_t

INDEX BY PLS_INTEGER;

l_fullnames fullname_aat;
BEGIN

wrong_code.sql
37
Let’s correct that “wrong code.” • Page 2 of 2

BEGIN
SELECT employee_id, salary, employee_rp.fullname (first_name, last_name)
BULK COLLECT INTO l_employee_ids, l_salaries, l_fullnames
FROM employees
WHERE department_id = department_id_in;

FOR indx IN 1 .. l_employee_ids.COUNT

LOOP
analyze_compensation (l_employees (indx).employee_id
, l_employees (indx).salary
);
END LOOP;

FORALL indx IN 1 .. l_employees.COUNT

UPDATE employees
SET ROW = l_employees (indx).salary
WHERE employee_id = l_employees (indx).employee_id;
END adjust_compensation;

wrong_code.sql
38
Optimizing SQL in PL/SQL: Think
"services" and leverage key features!

• Don't take SQL for granted.

– Just because it's easy, doesn't mean it's not significant.
• Hide SQL behind an API: serve up the SQL
via procedures and functions.
• Take advantage of key features to improve
performance and usability.
– BULK COLLECT, FORALL, table functions

39
Copyright © 2006 Quest Software