Software Release Process
Software Release Process
1.0 Introduction............................................................................................................. 3
2.0 Multi-Tiered Multi-Flow Process ........................................................................... 4
3.0 Standard Release Process Overview....................................................................... 6
4.0 Preview Release Process Overview ........................................................................ 7
5.0 Patch Release Process Overview ............................................................................ 8
6.0 Process Phases Explained ....................................................................................... 9
6.1 Software Development........................................................................................ 9
6.2 Software Development Testing........................................................................... 9
6.3 Alpha Characterization Testing ........................................................................ 10
6.4 Alpha Regression Testing ................................................................................. 10
6.5 External Preview Testing.................................................................................. 11
6.6 Internal and External Beta Testing ................................................................... 11
6.7 Release .............................................................................................................. 12
6.8 Final Release..................................................................................................... 12
With opposing requirements in terms of their demands upon the software development
and testing infrastructure, it is virtually impossible to satisfy both camps simultaneously.
In addition to the two contrasting customer needs noted above, there may occasionally be
the need to provide urgent patches in a rapid time frame.
It is advantageous to both Net Integration and to the willing customers to provide them
with the ability to preview the software and influence the development of the software
during an early stage where it is much easier to accommodate changes.
Rather than planning on achieving the impossible task of trying to tailor a single software
release process that is capable of meeting all the disparate needs noted above, it is more
prudent to separate the software release process in order to meet the many different
demands.
Net Integration thus releases software using the three process flows diagrammed in
Figure 2-1 - Net Integration's Release Processes. The exact process flow being utilized
is dependant upon the disposition of the particular software release.
Standard Release
Internal Beta
Testing
External Beta
Testing
Software Alpha
Software
Development Regression
Development
Testing Testing
External
Preview
Testing
Patch Release
Software Alpha
Software Release
Development Regression Final
Development Candidate
Testing Testing
Internal Beta
Testing
External Beta
Testing
Net Integration utilizes the Process shown in Figure 3-1 - Standard Release Process
Flow for the release of all software that is destined for customer release (this excludes
patch releases and preview releases discussed later).
During the initial software development, the flow allows for early testing to be tightly
coupled within the Software Development team, permitting rapid resolution of bugs.
At the Alpha level, Net Integration’s software Quality Assurance (QA) team becomes
involved. The team provides additional resources and a different point of view and
approach.
During the Beta stage, the deployment scenarios are further broadened by the
incorporation of external beta customers.
In the final stage, the software is promoted to Release level. At this point, all of Net
Integration’s internal systems are migrated to this software level. Customers also have
the ability to load the software on their systems, allowing early adopters to access the
latest features.
If a Release level has been determined to be stable and bug-free after a good amount of
live use, the software is promoted to Final Release.
Alpha
Extensive
Testing
Software Alpha
Software
Development Regression
Development
Testing Testing
External
Preview
Testing
Net Integration is built upon a solid foundation of being able to deliver products that meet
our customers’ needs. Thus, it is beneficial to allow customers to have the ability to
influence the software development as it occurs.
Preview releases are prepared during very early software development and the release is
earmarked specifically for “Preview Purposes Only” (PPO). The intent of the PPO
release is to solicit feedback on the new feature(s).
To facilitate a rapid feedback cycle, the standard software release process is modified to
accommodate the altered testing requirements. Specifically, the preview release
requirements are to ensure that software reaching the Preview stage is free of critical bugs
that prevent the system from being recovered, including reverting to an older Final or
Release level of software. The new features will be required to work with only low to
medium priority bugs present in order to be promoted to Preview level, however, it is to
be understood by all parties that the PPO software may contain bugs that are serious in
nature.
Both the External Beta/Preview customers and the internal QA team will work at the
Preview level to provide feedback and bug reports to the development team.
Software Alpha
Software Release
Development Regression Final
Development Candidate
Testing Testing
Despite everyone’s best efforts, bugs may appear at the Release and Final Release level.
Depending upon the urgency and severity of the bug, it may necessitate an ultra-rapid
patch release to the customer(s).
To provide the rapid response time required, a third process flow is required (see Figure
5-1 - Patch Release Process).
Once Net Integration is made aware of the critical bug (i.e. security flaw), the software
development team immediately works on the bug as a top priority.
During this process, the software development team tests the software to ensure that the
bug is fixed; the QA team verifies the bug fix and checks that the bug does not manifest
itself in a different manner; additionally, the Alpha level software is put through a limited
regression test to ensure that the system is capable of reverting to a previous software
version, and that no critical bugs were introduced. Upon completion of the steps noted
above, the software is released to Release level where the typical Release level rules
apply.
Each of the different processes covered share several distinctive stages. Each step serves
a unique purpose in helping Net Integration deliver a quality end-product in an
expeditious manner.
Software development has been completed with modularity in mind. Each development
team is responsible for a subset of the features within each release. Testing is performed
by the development teams on their components. Upon the completion of the features
scheduled for the specific software release version, the separate software components are
merged together. This merged software is then released for testing.
The development team itself is the first team to test the completed and merged software.
The goal of this testing is to ensure the base level functionality is present and that there
are no obvious problems. The purpose of this stage of testing is to ensure the software
image is not overtly problematic.
If there are any bugs found during this time, the bugs are reported immediately and the
software is returned to the software development phase. Software that passes this testing
(as deemed by the Software Development Manager) is promoted to the Alpha level.
Alpha Characterization Testing is mostly comprised of feature targeted test suites. The
test suites are intended to go through the complete functionality of the new features. The
objective is to understand and characterize the behavior of the features and/or
functionality. All medium to critical level bugs will necessitate a return to the software
development stage. All low priority bugs will be judged on a case-by-case basis with
respect to their impediment to the continuance of the software towards becoming a
Release.
Promotion of the software shall be made upon the judgment of the Software QA
Manager.
If the software is marked as “Preview Purposes Only”(PPO), the alpha testing will be
constrained to a limited set of regression tests and feature walkthroughs. The objective is
to ensure that all high to critical level bugs are found and reported, and all low to medium
priority bugs are documented. Critical bugs will necessitate a return to the software
development phase. High-level bugs will be judged on a case-by-case basis. In any
circumstance, the software shall not be released to Preview level if it is not possible to
safely revert to a previous Final software release version.
Alternatively, if the software is intended to ultimately reach Release/Final Release
(Standard Release Process), tests are performed to ensure that the behavior of features
from previous software release versions have not been unintentionally modified.
Additionally, testing is conducted to ensure that the systems can safely backtrack to
previous software versions. Once the internal QA team has deemed that the software has
passed regression testing, is feature complete, and does not have any known medium to
critical level bugs, the software is promoted to Beta level.
In the case that the software is a Patch Release, the Alpha Regression Testing follows an
accelerated path that ensures that the behavior of all features remain intact with the
possible exception of the feature affected by the patch. Additionally, testing is conducted
to ensure that the systems can safely backtrack to previous software versions. Once the
internal QA team has deemed that the software has passed the accelerated regression
testing, the software is promoted to Release level.
Judgment on the suitability of the promotion of the software to Beta or Preview level
shall be made by the QA Manager (consultation with the Support and Development
Managers is expected). For Patch releases, promotion of the software to Release shall be
made at the discretion of the Support Manager.
Early revisions of software will be made available for Preview level testing. Software
that is released for External Preview Testing will not have received the normal amount of
testing, so bugs are to be expected. The known bugs will be documented in the Preview
release notes. However, Net Integration will only release software to the Preview level
that we deem to be free of critical bugs.
Only Beta Testers will have access to Preview level software. Feedback is expected
through the use of surveys to help guide the development of the software on the specific
feature(s). Beta Testers are to understand and accept that PPO software has not received
the same level of internal scrutiny as standard Beta Level software.
The goal of the Beta Level testing is to expose the software to a larger range of usage and
deployment scenarios.
The internal QA team transitions to testing with the intent of exposing any potential
longer-term stability and integration problems; it is not feasible to cover off all
deployment scenarios.
Thus, at the Beta level, external beta customer testing is incorporated. Beta customers
are required to explicitly sign a waiver and non-disclosure agreement. Feedback from
beta customers is solicited through the implementation of a Beta Program.
If an external Beta Tester finds any problems, the internal QA team will work towards the
replication of the problem(s). If the problem has been verified, the decision will be made
by QA and Support as to the pending status of the release. One of the following
outcomes is expected:
A) Critical Bug – Software is instantly returned to Software Development phase.
Beta Customers are notified of the findings and are requested to continue at their
own discretion.
B) Medium to High Priority Bug – Software is returned to Software Development
phase. Beta Customers are updated with information pertaining to the bug. Beta
customers are requested to continue, but are asked to avoid specific scenarios.
C) Low priority bug - Internal decision is made as to correct this bug or await next
software release. Beta Customers are updated with information pertaining to the
bug. Beta customers are requested to continue, but are asked to avoid specific
scenarios.
D) No Bugs – no action required.
If no bugs are found that require the software to return to the software development
phase, the software is then promoted to Release at the discretion of the Support Manager.
Release level software is provided to serve two purposes. The first purpose is to allow
for early technology adopters to get the latest available software as quickly as possible.
The second purpose is to allow for a method to distribute critical patches as swiftly as
possible.
Release level software is available to all customers, but requires them to explicitly accept
the terms of use before it is installed on their system.
Should customers encounter any erratic behavior with Release level software, they are to
report the behavior to Customer Support. Net Integration Technologies’ QA team will
work with the support team to reproduce and understand the behavior and file it as a bug
as necessary. Depending on the nature of the bug the following actions may take place:
A) Critical Bug – the release is returned to the Software Development phase; the
software is removed from Release level status.
B) Medium to High Priority Bug – the software is returned to the Software
Development phase; release notes are updated to indicate the bug and Release
level users are requested to avoid specific scenarios.
C) Low priority bug – an internal decision is made as to correct this bug or await
next software release; release notes are updated to indicate the bug and Release
level users are requested to avoid specific scenarios.
D) No Bugs – no action required.
Software at the Release level is monitored as to the prevalence of its usage. Using time
and number of deployments as factors, a determination is made as to whether a specific
software version has received enough stable usage to be promoted to Final Release. All
promotion of software to Final level is conditional upon the assessment of the Support
Manager.
Software that has reached the Final Release level is deemed to be as tested and stable as
possible. This is the software version universally recommended to all Net Integration
customers. For customers that demand stability above all else, it is highly recommended
that they only use software that has been promoted to Final Release level.