Vol. 13, No.

www.processgroup.com/newsletter.htm

October 2006

IMPLEMENTING PRODUCT
AND PROCESS QUALITY ASSURANCE
With permission from a client of The Process Group
One of the Process Areas (PA's) in the CMMI* is Process and
Product Quality Assurance (PPQA). This process area supports
the delivery of high-quality products and services by providing
the project staff and managers with appropriate visibility into, and
feedback on, processes and associated work products throughout
the life of the project.
In this article we share an example process definition for PPQA
from one of our clients.
The company uses seven part-time Software Quality Assurance
(SQA) staff, each person working eight hours per month on
average performing audits. Each auditor is either a developer or
tester as his or her primary job. To ensure objectivity, each person
is assigned to a project other than the one they work on. All audit
activities are managed by the leader of the Engineering Process
Group (EPG), a part-time position of seven hours per month for a
population of 80 software developers.
The benefits of implementing a lightweight audit process are to:
Maintain the gains made in process improvement over the
previous five years
Detect process refinements needed to simplify process implementation
Share engineering and management practices across the development groups

Process and Product Quality Assurance (PPQA)

Audit Process
1.0 PURPOSE
The purpose of this Audit Process is to provide SQA staff
with a standard method for performing audits.
2.0 SCOPE
This process applies to software projects within the XYZ
organization.
3.0 RESPONSIBILITIES
The SQA staff is responsible for conducting audits using this
process and the audit checklist(s) (see
www.processgroup.com/PPQA_Audit-Checklists-v1.zip).
* Capability Maturity Model Integration v1.1 / v1.2

4.0 CONDUCTING AUDITS

4.1 Audit Preparation
Each SQA staff member (auditor) will keep appraised of
the project he or she has been assigned to by reviewing
the project's Software Project Management Plan (SPMP)
and schedule. The auditor will meet with the Project
Leader approximately one week prior to a scheduled
audit. At this time the following shall be discussed
and/or identified:
The actual date of the audit
The location of project plans and artifacts to be
audited
The names of project team members whom the
auditor can contact should questions arise
Prior to the actual date of the audit, the auditor shall:
Identify and prepare copies of the required audit
checklists
Review associated project plans and artifacts
4.2 Conducting Audits
The auditor shall perform audits of the project teamwork
activities by reviewing resulting work artifacts and
making comparisons to the associated process within the
team's documentation. As a minimum, the audit shall
follow the items contained in the audit checklist(s). The
results will be recorded on the checklist.
The Project Leader will be informed of the audit results
via the reporting activity and must resolve any noncompliant issues. Resolving noncompliant issues should be
handled in a timely fashion (less than two weeks is
expected) unless some unavoidable circumstances
prohibit resolution.
Noncompliant issues that are not resolved promptly and
are not included in a plan for resolution shall be
escalated by notifying the EPG Lead of the situation. If
the EPG Lead cannot bring about resolution of noncompliant issues, then he or she will escalate the issues to
the Technical Director.
CONTINUED ON PAGE 2

IMPLEMENTING PRODUCT AND PROCESS QUALITY ASSURANCE (Continued from page 1)

5.0 AUDIT REPORTING
The completed checklist acts as the audit report.

Below the Project and Project Leader, a separate row shall be

used to record metric data for each audit. If multiple audit
types are audited together (e.g. Project Planning, Project
Monitoring and Control, and Requirements), a unique row
shall be used to record the data for each audit type.

5.1 Naming Conventions

A standard naming convention for audit report files is
used. File names will be a concatenation (with underscores) of: Group name_Project name_Audit
name_Date.

In addition, a separate row shall be used for each audit to

record issues found in each process area (PA). In the sample
(Table 1) a project-planning audit was conducted on
11/03/xx. During that audit, four major and two minor issues
were found in Project Planning (PP), two major and five
minor issues in CM, and three major and one minor issue(s)
in Measurement & Analysis (MA). Both the Audit Name and
Process Area columns are populated using pre-defined pulldown menus. The Audit Name correlates to the checklist

The group, project, and audit names can be abbreviated

to avoid long file names. The audit name shall
correspond to one of the audit types in Table 3. The date
is in the form, mmddyy, and is the actual date of the
audit. An example would be Project1_Release215_
PMC_060204.doc

Project
Name
Project
Leader
Project 1
(Smith)

Project 3
(Smith)

Audit
Date
11/03/xx

11/25/xx

Planned/
Unplanned
Audits
Planned

Planned

Audit
Name
PP

Process
Area
PP

Work
Products
Reviewed
2

PMC

CM
MA
PMC

1
2
0

Number of
Major
NonCompliant
Issues
4

Number of
Minor
NonCompliant
Issues
2

2
3
0

5
1
0

Number of
Major
Number of
NonMajor
Compliant
NonIssues
Compliant
Elevated to
Issues
EPG Lead
Closed
0
0
0
0
0

0
0
0

Number of
Major NonCompliant Number of
Issues
Hi-Priority
Elevated to Risk Items
Senior
in Project
Management Risk List
4
0
0
0
0

Hours to
Prepare
and
Conduct
the Audit
3

Auditor
Liu

Liu

0
0
0

Table 1 Record of PPQA Audit Metrics

5.2 Location of Reports

The audit reports shall be stored into the group
Configuration Management (CM) system
(//qa-reports/project-X/<file>).

Auditors shall email a copy of each audit report to the

associated Project Leader and the EPG at epg@xyz.com
when the report is completed. The EPG shall be responsible for storing the file in the appropriate directory.

name while the Process Area corresponds to the CMMI PA

name. A blank entry is provided in the Audit Name dropdown menu for audits with multiple rows. The checklist has a
column to identify the PA for each item checked. Below is an
example of a checklist to illustrate.
6.2 Planned/Unplanned Audits
This metric allows the auditor to specify if the audit was
planned (scheduled in the SPMP) or unplanned. Unplanned
audits are conducted at the discretion of the auditor, based
on previous evaluation of process performance.

6.0 METRICS
Auditors shall be responsible for providing the EPG with
metric information on an individual
ITEM PROCESS
VERIFICATION
project basis. Auditors shall use a
NO.
AREA
ITEM
copy of a blank metrics spreadsheet
1.
RD
Are the requirements documented using the
obtained from the Process Asset
template in the PAL or using a customer-defined format?
Library (PAL)
2.
REQM
Is there a traceability matrix for the requirements?
(www.processgroup.com/
Note: a traceability matrix implies that a requirement is traced to a
PPQA-Metrics-v1.xls). An example
design element, test case and area of code.
is shown in Table 1. Once filled in,
the auditor shall email the spreadsheet to the EPG at epg@xyz.com.
6.1 Project
In the spreadsheet, "Project
Name" shall be replaced by the
actual name of the project, using
an abbreviation that makes sense
if necessary. Below the project
name, the project leader's name
shall be entered in parenthesis.

3.

REQM

Are requirements tracked and updated according to the

Software Project Management Plan (SPMP), or the Software
Configuration Management Plan (SCMP) if it is a separate document?
Need to verify that changes to requirements are controlled.

4.

RD

Are requirements analyzed using a technique other than a peer review

(e.g. use cases, Object Oriented Design methods, writing test cases
for each requirement, etc.)?

5.

REQM

YES

NO
Maj Min

N/A

Are requirements under Configuration Management?

Table 2 Excerpt from an Audit Checklist

CONTINUED ON PAGE 3

IMPLEMENTING PRODUCT AND PROCESS QUALITY ASSURANCE (Continued from page 2)

This metric serves to compare planned work to actual work,
which has a direct impact on hours and funding. This metric
must be tracked as each SQA audit is completed. A significant number of unplanned audits shall trigger analysis of
actual versus planned funding for SQA activities.
6.3 Audit Name
This column is used to record the type of audit being
conducted. A separate row shall be used to record metric data
for each audit type. Table 3 illustrates the phases of a project
and the abbreviation to be entered into this column of the
metric spreadsheet. This table is provided on the Info tab of
the spreadsheet. Use a blank in the Audit Name column when
an audit requires more than a single row as described in
section 6.1. Options QA and EPG are provided for use in
recording the results when QA and EPG activities are
audited.
Audit Type

Abbreviation

Blank
Project Planning
Project Monitoring and Control
Requirements Management
Risk Management
Design
Verification - Validation
Status Meeting
Peer Review
Quality Assurance
Engineering Process Group

PP
PMC
REQM
RSKM
DESIGN
VER-VAL
STSMTG
PRVW
QA
EPG

Table 3 Audit Types

6.4 Process Area

This column contains the CMMI Process Area of the issue
being reported.
6.5 Work Products Reviewed
During the life cycle of a project, the team will generate
project work products (e.g. Software Requirements
Specification, Design Review, etc.). The SQA auditor will
enter the number of work products that were assessed during
the audit into the Number of Work Products Reviewed
column. Note that this figure is the total number of work
products reviewed. For example, if 20 test logs were
reviewed in addition to the test plan the total number of work
products reviewed would be 21.

6.6 Number of Major Noncompliant Issues

This metric tracks the number of major noncompliances.
Section 4.2, Conducting Audits, describes escalating noncompliant issues. A major noncompliant issue is identified as a
significant deviation from the documented process, for
example, when the project lacks something that is necessary
and specifically named in the process documentation. The
following are examples of major noncompliant issues:
Lack of an SPMP or other plan
Projects not following what is documented in the SPMP
SPMP missing major section(s) (e.g. MA)
Documents not being kept up to date
Lack of a schedule and/or Work Breakdown Structure
6.7 Number of Minor Noncompliant Issues
Minor noncompliant issues shall be annotated as such in the
comments section of the associated checklist. Example
noncompliant issues include:
Typographical errors
Documents with titles or locations that are not exactly as
referenced in a plan
Documents with noncurrent release attributes
Late posting of meeting minutes, peer reviews, etc.
Minor noncompliant issues are problems that should be tracked
to closure. However, they are not collected for metrics analysis.
6.8 Number of Major Noncompliant Issues Closed
This metric tracks the number of noncompliances closed or
resolved. There should be no noncompliant issues left
unresolved. Section 4.2, Conducting Audits, describes
escalating noncompliant issues.
6.9 Number of Major Noncompliant Issues Elevated to EPG Lead
The EPG Lead shall review the audit reports and will be
apprised of any noncompliant issues. Section 4.2, Conducting
Audits, describes escalating noncompliant issues.
6.10 Number of Major Noncompliant Issues Elevated to
Senior Management
This metric tracks the number of noncompliances elevated
to senior management. Section 4.2, Conducting Audits,
describes escalating noncompliant issues.
6.11 Number of High Priority Risk Items
This metric tracks the number of risk items that have a
priority higher than 50 (out of 100). The metric allows the
EPG to monitor the number of high priority risks being
tracked from the planning to closure phases of a project.
6.12 Hours
This metric tracks the time it takes an auditor to prepare,
conduct, and document an audit.
6.13 Auditor
The auditor shall enter his or her last name here. A first
initial shall be employed when different auditors have the
same last name.

Practical Solutions for your

Software Development Challenges
o Understand customer needs. Clarify product requirements early.
This two-day workshop, IN SEARCH OF EXCELLENT REQUIREMENTS, teaches software
engineers, managers, requirements analysts and user representatives how to gather,
document, analyze and manage customer requirements for software applications.
o Manage projects effectively. Meet project deadlines and reduce risks.
This three-day workshop, SOFTWARE PROJECT PLANNING AND MANAGEMENT, teaches
project managers and their teams how to meet deadlines through better estimation,
reduce surprises using risk management, schedule work for better optimization,
understand and negotiate project trade-offs, and track progress.
o Meet project deadlines. Scope and estimate the project work.
This one-day workshop, SOFTWARE ESTIMATION, (a subset of Software Project
Planning and Management) helps teams develop more accurate estimates.
o Avoid schedule delays caused by needless product rework.
Find defects rapidly.
This two-day workshop, INSPECTION (PEER REVIEWS), teaches teams to efficiently
find defects in code and documentation. (Includes moderator skills.)
o Hands-on SEI CMMI. Perform a CMMI gap-analysis.
The following workshops are available:
o SEI CMMI: Overview (half day), LEVEL 2 (one day), LEVEL 3 (two days)
o SEI INTRODUCTION TO CMMI (three days)
o Identify critical changes to improve organizational results. Benchmark against
the CMMI.
A SOFTWARE PROCESS APPRAISAL examines your organizations engineering and
management practices and generates a focused list of the critical areas for
improvement. Our SEI-authorized Lead Appraisers conduct customized CMMIbased appraisals.
o Goal/problem-based improvement.
This two-day workshop, MAKING PROCESS IMPROVEMENT WORK, provides a
systematic approach for organizations to improve their development capability.
It includes: getting management support, focusing the organization on the critical
issues, planning the improvement and effecting change.
o Manage your subcontractors.
This one and one-half-day workshop, SOFTWARE SUBCONTRACT MANAGEMENT,
teaches software engineers, managers and subcontract managers how to define a
software product to be outsourced, write a subcontract management plan, select
appropriate vendors and manage the project to completion.
o Tailored assistance. Dedicated phone-based assistance.
This service consists of customized education and coaching on your specific
problems (e.g., meeting deadlines, quality and cultural change.)
Detailed information on our services is available at www.processgroup.com.
Contact us at 972-418-9541 or help@processgroup.com to discuss your needs.

Come see our book!

Also available in Chinese and Japanese.
See www.processgroup.com/tpgbook.htm

Here is the books Table

of Contents:
Foreword by
Karl Wiegers
Preface
Acknowledgements
Chapter 1.
Developing a Plan
Scope the Improvement
Develop an Action Plan
Determine Risks and Plan to Mitigate
Chapter Summary
Chapter 2. Implementing the Plan
Sell Solutions Based on Need
Work with the Willing and Needy First
Keep Focused on the Goals and Problems
Align the Behaviors of Managers and
Practitioners
Chapter Summary
Chapter 3. Checking Progress
Are We Making Progress on the Goals?
Are We Making Progress on our
Improvement Plan?
Are We Making Progress on the
Improvement Framework?
What Lessons Have We Learned So Far?
Chapter Summary
Conclusion
Appendices
References

The Process Group

Mailing address: The Process Group
P.O. Box 700012
Dallas, TX 75370
Telephone number: 972-418-9541
Fax number: 972-618-6283
E-mail: help@processgroup.com
Web: www.processgroup.com
POST back issues are on line