Best Practices For Large Oracle

Apps R12 Implementations

Ajith Narayanan
Technical Lead-Oracle ERP Configuration Management
Imagination at work. GE Healthcare, Bangalore, India
8th Dec-2014, Liverpool, U.K.
Disclaimer
The views/contents in this slides are those of the author and do not
necessarily reflect that of General Electric Company or Oracle
Corporation and/or their respective affiliates/subsidiaries.

The material in this document is for informational purposes only, and is

provided on an AS-IS basis.

11/14/2014
Who Am I?
Ajith Narayanan
Technical Lead – Oracle ERP Configuration Management
GE Healthcare, Bangalore, India

• 10 years of Oracle [APPS] DBA/Technical Architect experience.

• Oracle ACE Associate
• Blogger :- http://oracledbascriptsfromajith.blogspot.com
• Member:-OAUG, AIOUG
• Website Chair (2011-2013) – Oracle RACSIG http://www.oracleracsig.org
• AIOUG Real Application Clusters SIG Leader

11/14/2014
Agenda
Why do we need to implement best practices?
• Application Tier
• R12 Tech Stack
• Forms
• Response Time
• Web Applications
• Concurrent Manager
• Workflow Engine
• Taking Diagnostics Help
• CBO Statistics
• Application Patching
• Network & Latency Check

11/14/2014
Agenda
• Database Tier
• CPU Utilization Observations and analysis
• High DB_IO observation and correlation with CPU usage.
• Identify Top SQLs
• Effective usage of AWR, ADDM Reports
• ORACHK Check (From Oracle RAC assurance team)
• ADR Command Interpreter (ADRCI)
• Cluster Callout scripts example.
• Looking At Issues Proactively & From Greater Heights
• References
• Q&A

11/14/2014
Why do we need to implement best
practices?

11/14/2014
Why do we need to implement best
practices?
Oracle Applications R12 Best Practices
• Huge infrastructure • Benefits
– Oracle Applications R12 – Enables us being proactive
– 8 Node middle tiers – simple in terms of manageability
– 16 Node RAC database tier. – Helps in support cost reduction.
– Database Size 40+ TB – Time & effort saving
– Annual DB growth of 2 TB
– Daily Orders: 45 K –3 M order
lines
– 1500 online users
– 1M+ XML messages/day

11/14/2014
Application Tier
Upgrade Tech Stack
• R12 Tech Stack
• OC4J: Oracle AS 10g 10.1.3./ Weblogic Managed Servers (FMW)
• Forms: Oracle AS 10.1.2
• JAVA, JRE Plugin

Useful MOS Articles

Oracle E-Business Suite Technology Stack Release Notes for Release 12.1.3 (MOS Note - 1098650.1)

Oracle E-Business Suite Technology Stack Release Notes for Release 12.2*

Oracle E-Business Suite - Technology Stack : Navigation (landing) Page (MOS Note - 1480550.1)

Troubleshooting Assistant: EBS Technology Stack (Doc ID 1607365.2)

11/14/2014
Application Tier

10

11/14/2014
Application Tier
Forms
• Socket Mode for Internal users (MOS 384241.1)
• Servlet mode is default in R12 and recommended for external access.
– cat $FORMS_WEB_CONFIG_FILE|grep serverURL=
If the serverURL parameter has no value then Forms is implemented in socket mode else it is servlet

• Forms dead client detection (DCD) and Abnormal termination handler.

– Value specified in minutes : FORMS_TIMEOUT=10
env | grep -i FORMS_TIMEOUT
– Terminates fwebmx processes for dead clients.
– Enable Forms Abnormal Termination Handle
– Do not set FORMS_CATCHTERM
– SQLNET.EXPIRE_TIME – Probes client connection at regular interval.

Useful MOS Articles

Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID 745711.1)

11

11/14/2014
Application Tier
Response Time/CPU Issues
• When users complain about response time, configure Apache to log the time it
takes to service a request.
• Access logs monitoring should be good way to keep a tab on the response times of
user requests.
– Configure Apache to log the time it takes to service a request
– Logs: $LOG_HOME/ora/10.1.3/Apache/access_log*
– Edit: $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogFormat "%h %T
(Or)
– Enable Execution Content ID's (ECID) for the access_log in Release 12. ECID's allow correlation of
requests across log files so that one can map the flow of a given request across the various
components of iAS.
LogFormat "%h %l %u %t [ecid: %{Oracle-ECID}i] \"%r\" %>s %b [%T (secs)]" combined

Refer to http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for getting http return code definitions

Note - R12.2 replaces Oracle 10gAS OC4J containers with Weblogic managed servers.

12

11/14/2014
 Application Tier
Web Applications
• Train users to logout when transactions are complete.
– Logging out gracefully releases the memory and corresponding resources (i.e. connections, etc.)
Releases process resources sooner

• JVM/OC4J Sizing + Heap Size

– OACORE, OAFORMS, OAFM etc..
– Start with no more than 100 active users per JVM/OC4J instance.
– Start with 1 active JVM/OC4J instance per CPU core
Note:- Each 'core' in a multi-core system is considered to be a separate CPU. For example, a dual quad-
core chip would be the same as stating 2 x 4 x CPU or 8 CPUs.

Useful MOS Articles

JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID 362851.1)

MOS Note 357597.1 on how to enable SQL trace for OA Framework applications

13

11/14/2014
 Application Tier
EBS Concurrent Processing Analyzer
• The EBS Concurrent Processing Analyzer is a self-service health-check script
• Reviews the overall Concurrent Processing
• Analyzes the current configurations & settings for the environment.
• Providing feedback and recommendations on best practices.
• APPS Workload (Concurrent, FORM, ICX) observations.
• Mandatory Concurrent Programs fro purging.

Sample Output 2DEMO_cp_analyzer.html

Useful MOS Articles

Best Practices for Performance for Concurrent Managers in E-Business Suite [Doc ID- 1057802.1]

Concurrent Processing - CP Analyzer for E-Business Suite (Doc ID 1411723.1)

14

11/14/2014
Application Tier
EBS Workflow Analyzer
• The EBS Workflow Analyzer is a script that reviews the current Workflow Footprint.
• Analyzes the workflow configurations & Provides feedback and makes
recommendations on best practices and areas of concern.
• Provides immediate analysis and output of the EBS Workflow environment.
• Workflow Process Stuck & Work Flow Checks

Sample Output 2DEMO_wf_analyzer.html

Useful MOS Articles

How to run EBS Workflow Analyzer Tool as a Concurrent Request (Doc ID 1425053.1)

11i-12 Workflow Analyzer script for E-Business Suite Workflow Monitoring and Maintenance [Video] (Doc
ID 1369938.1)

How To Use Concurrent Program "Purge Order Management Workflow" (Doc ID 878032.1)

15

11/14/2014
 Application Tier
OAM Diagnostics Help
• Oracle Application Object Library Best Practices: Database Security Tests Health
Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Security Tests
Health Check Test
• Oracle Application Object Library Best Practices: E-Business Suite Diagnostic
Tests Health Check Test
• And many more……

16

11/14/2014
 Application Tier
CBO Statistics
• FND_STATS uses the standard the DBMS_STATS package, but enhances the
functionality as it provides several additional PL/SQL procedures for
gathering statistics at the table, schema, or database level.

• The FND_STATS package also supports histograms, table exclusions,

extended stats, incremental stats gathering for partitioned tables, and
concurrent statistics gathering.

Useful MOS Articles

Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID - 1586374.1)

bde_last_analyzed.sql - Verifies CBO Statistics (Doc ID - 163208.1)

17

11/14/2014
 Application Tier
Application Patching
• Thorough patch analysis with timing reports on sand box & non-prods
environments before proceeding with the Prod patching.

• In Release 12, you register your customized files in the Register Flagged Files tool
in Oracle Applications Manager (OAM), which replaces the
$APPL_TOP/admin/applcust.txt file.

Useful MOS Articles

See Note 9766881 "My server does not have internet connection Can I Still use Patch Wizard?“

Patching Best Practices And Reducing Downtime (Doc ID 225165.1)

18

11/14/2014
Application Tier
Network & Latency
• There are two diagnostic tools available in the Oracle E-Business Suite:

The Network test Form – This form can be added to user’s menu , So that they can
run the tests when they feel a performance degradation
Note:- Network latency should be always low (<3ms) and bandwidth should be high.

19

11/14/2014
Application Tier
Network & Latency
The Client Analyzer

Useful MOS Articles

Oracle E-Business Suite Network Utilities: Best Practices (Doc ID 556738.1)

20

11/14/2014
Database Tier
CPU Utilization Observations and
analysis
• CPU spikes for more than 30 minutes – How many of you think RAM
continuously is a candidate for RCA. access is 10,000 times faster than
• High DB_IO observation and Physical disk access?
correlation with CPU usage. – In real world, LIO is only 25-100
• I/O Thresholds: Logical RDS > times cheaper than PIO
300K/sec, Physical RDS> 3000/Sec are – Internal locks & latch serialization
candidate for RCA. mechanisms involved.
– Targeting only PIO counts(or high
cache hit ratios) important pitfall
to avoid. Even with no PIOs, a
query can still be outrageously
inefficient
– LIO are a critical component of
query cost

21

11/14/2014
 Database Tier
AWR, ASH, ADDM Reports
• Identify Top SQL, Latch Spin Counts – ORAchk daemon auto-start mode
etc., Memory Advisors after node reboot (init integration)
• Identify top wait events (System, User – Merge multiple ORAchk collection
IO, Cluster, Concurrency, reports
Configuration, Applications, Network) – Exclude checks based on profile
• Fragmented Interface tables – Upload of installed patches to
• Object to be pinned – (Frequently used database
database objects) – Collection Manager for ORAchk,
RACcheck and Exachk (Doc ID
• RAC Check (From Oracle RAC
1602329.1)
assurance team) RACcheck - RAC
Configuration Audit Tool (Doc ID – ORAchk signature file in /tmp on
1268927.1) all nodes to verify last ORAchk run
• ORAchk - Health Checks for the Oracle – New checks and bug fixes,
Stack (Doc ID 1268927.2) including o 30 Oracle Ebusiness
AP module data integrity checks
• Sample ORACHK report ORAchk-Report-Example.html
– 12 new Database checks
• ORAchk-Collection Manager – 8 new Solaris system checks

22

11/14/2014
 Database Tier
ADR Command Interpreter (ADRCI)

• ADRCI has multiple features such as

– Analyze diagnostic data within the Automatic Diagnostic Repository (ADR).
– Create a package with incident / problem information for Oracle Support.
– Diagnostic data includes incident and problem descriptions, trace files,
dumps, health monitor reports, alert log entries, and more.
– Run the Health Checks
– Helps in purging old diagnostics trace & dump files
– Simple to use a rich command set

Set correct ORACLE_HOME and just enter adrci

The utility starts and displays the following prompt:
adrci>

23

11/14/2014
Database Tier
• At the ADRCI prompt, enter the following command:
adrci> HELP

To get help for a specific ADRCI command:

adrci> HELP command

• For example, to get help on the SHOW TRACEFILE command, enter the following:
adrci> HELP SHOW TRACEFILE

adrci> set homepath diag/rdbms/orclbi/orclbi2

adrci> show homes
ADR Homes:
diag/rdbms/orclbi/orclbi2

• To view the alert log:

1. Start ADRCI in interactive mode, and ensure that the homepath points to the
correct directory within the ADR base directory hierarchy.

24

11/14/2014
Database Tier
2. At the ADRCI prompt, enter the following command:
SHOW ALERT

The following are variations on the SHOW ALERT command:

SHOW ALERT -TAIL

SHOW ALERT -TAIL 50

SPOOL /tmp/MYALERT.LOG
SHOW ALERT
SPOOL OFF
This outputs the alert log, without XML tags, to the file
/tmp/MYALERT.LOG.

SHOW ALERT -P "MESSAGE_TEXT LIKE '%ORA-600%'"

Useful MOS Articles

11G ADR DOCUMENTATION (Doc ID 445022.1)

How to Invoke ADR Command Interpreter (adrci) in FMW home? (Doc ID 1669923.1)

25

11/14/2014
 Database Tier
Cluster callout scripts. – Quickly get alerted on any cluster events.
#!/bin/ksh
# # Author: Ajith Narayanan
## http://oracledbascriptsfromajith.blogspot.com
## Version 1.0
## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.
##
umask 022
FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.log
EVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.log
EVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.log
MAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.log
WEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_ds
echo $* "reported="`date` >> $FAN_LOGFILE &
tail -1 $FAN_LOGFILE > $EVENTLINE
awk '{
for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {
for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }
}' $EVENTLINE > $EVENTLINE_MID
SER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`
DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`
INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`
HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`
STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`
if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; then
if [ $STAT = nodedown ]; then
cat $EVENTLINE_MID > $MAIL_CONT

26

11/14/2014
Database Tier
echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONT
elif [ $STAT = up ]; then
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES RESTORED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
else
cat $EVENTLINE_MID > $MAIL_CONT
echo "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONT
grep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT
#fi
cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.com
fi
rm $EVENTLINE $EVENTLINE_MID $MAIL_CONT

27

11/14/2014
Looking At Issues Proactively & From
Greater Heights

28

11/14/2014
References

• My Oracle Support
• Steven Chan’s blog (https://blogs.oracle.com/stevenChan)

29

11/14/2014
Q&A

30

11/14/2014
Thank You!

31

11/14/2014