Software Build - Overview
Software Build - Overview
DEFINING A BUILD
I like to say that there are really only two types of builds: ones that work
and ones that don’t. Seriously, though, when you’re shipping a product, you
should consider these two different types of builds:
1
2 Chapter 1 Defining a Build
The developer’s build process should be optimized for speed, but the
project build process should be optimized for debugging and releases. I am
talking about optimizing the process, not compiler or linker optimization
switches. Although speed and debugging are important to everyone who is
writing code, you must design a project build process to track build breaks
and the offender(s) as quickly as possible because numerous people are
waiting for a build to be released. For a developer, what seems to be most
important is clicking some type of Build and Run button to make sure the
code compiles without errors and then checking it in. For the build team,
building without errors and having the ability to track down the person
who broke the build is the most important thing.
NOTE In some simple scenarios, these two build cases can use the same
process. If this is the case, the team—what I refer to as the Central Build Team—
should dictate the build process. This team—not the developers—should design
the project build process. All too often, the developers design the project build
process, which causes problems. Because developers usually build just the code
modules that they work on and not the whole project on a regular basis, they
look for shortcuts that are not necessarily in the best interest of building
the entire project. For example, they might use file references instead of project
references.
If a developer specifically references a file in Visual Studio and the sources
of that file change, they are not automatically picked up because a specific ver-
sion of the file was referenced instead of the project that builds the referenced
file. In the interest of saving time, developers use file references. They are not
interested in picking up the latest sources of the specified file, but it is not recom-
mended to use file references in a project build.
The Central Build Team should never be at the mercy of mandatory build
environment settings for building a specific component. If such a setting is neces-
sary to build a component, it should be proposed to the Central Build Team for
inclusion. Then the CBT can determine the impact of the addition or change to
the entire project and approve or disapprove the proposal.
Building from the Inside Out 3
1. DEFINING A BUILD
“Every day.” Of course, as you might suspect, I am not talking about releas-
ing a daily build, but a new build process. The fact that so many companies
do not release new build processes on a regular basis does not surprise me.
This is because traditionally creating a build process is an afterthought
when all of the specifications of a project have been written. Many project
and program managers think that the actual building of a project is pretty
trivial. Their attitude is that they can simply have the developer throw
his code over the wall and hire someone to press a Build button, and
everything will be fine. At Microsoft, we understand that whether you’re
building the smallest application or something huge and complicated like
Windows, you should plan and think through the process thoroughly in
advance.
Again, I recommend that you consider the build process a piece of
software that you regularly revise and deploy throughout your product
team. You should also add to your project schedule some “cushion time”
to allow for unforeseen build breaks or delays, I would at least pad the
milestone dates one week for build issues.
The concept of “building from the inside out” tends to confuse custom-
ers who are not familiar with a centralized build process. The idea is that
the Central Build Team determines what the build process is for a product
and then publishes the policies to an internal build site. All development
teams in the project must comply with the Central Build Team process;
otherwise, their code check-in is not accepted and built. Unfortunately,
this concept is usually the complete opposite of how a build system for a
project actually evolves over time. The Central Build Team for a project
usually goes out of its way to accommodate the way developers build their
code. “Building from the inside out” means that the Central Build Team
figures out the best way to get daily builds released, and everyone uses that
process independently or in parallel with the way his specific development
team builds. This total change in development philosophy or religion can
be a culture shock to some groups. I talk more about changing a company’s
culture or philosophy in Chapter 18, “Future Build Tools from Microsoft.”
For now, let’s stay on the topic of builds.
4 Chapter 1 Defining a Build
What we did in the past in the Windows group—and what they still do
today—is to deploy new releases of the build process at major milestones
in the project life cycle. Sometimes the new releases involve tool changes
such as compilers, linkers, and libraries. At other times, there are major
changes such as a new source code control tool or a bug tracker.
Because a build lab tends to have some downtime while the build team
waits for compiles, links, and tests to finish, it should take advantage of
these slow times to work on improvements to the build process. After the
lab tests the improvements and confirms they are ready for primetime, it
rolls out the changes. One way to deploy a new build process after a ship-
ping cycle is to send a memo to the whole team pointing to an internal Web
site that has directions on the new process that the Central Build Team will
be using in future product builds.
I need to define some common build terms that are used throughout this
book. It is also important for groups or teams to define these terms on a
project-wide basis so that everyone is clear on what he is getting when a
build is released.
More Important Build Definitions 5
1. DEFINING A BUILD
build verification tests (BVTs) are run. This also includes security
tests to make sure the correct code was built and nothing was fused
into the build.
■ Clean build—Deleting all obj files, resource files, precompiled
headers, generated import libraries, or other byproducts of the
build process. I like to call this cleaning up the “build turds.” This
is the first part of a clean build definition. Most of the time, build
tools such as NMake.exe or DevEnv.exe handle this procedure
automatically, but sometimes you have to specify the file exten-
sions that need to be cleaned up. The second part of a clean build
definition is rebuilding every component and every piece of code in
a project. Basically the perfect clean build would be building on a
build machine with the operating system and all build tools freshly
installed.
100
90
80
1. DEFINING A BUILD
70
60
% Regression Daily Build
50
Tests Pass IDW Baseline
40
30
20
10
0
1 2 3 4 5 6 7 8 9 10 11
Build Number
The build team would get the data for the quality chart from the test
teams and publish it as soon as it was available. This is how we controlled the
flow of the product. In a “looser” use of the word build, the quality became
part of the definition of a build number. For example, someone might say,
“Build 2000 was an excellent build” or “Build 2000 was a crappy build,”
depending on the test results and personal experience using the build.
Program/
Development
Product Test team
team
Managers
1. DEFINING A BUILD
Feature/Bug
Database
(store
everything) Happens everyday
or as needed.
Happens
everyday
Needs implementing Not approved Re-assign religiously!
Assign Bug WAR Meeting
Approved priority/
Set Pri/Sev
schedule
Implemented
Check-in OK
ponement is acceptable.
■ Severity—This communicates how damaging a bug is if or when it
is encountered.
■ Sev 1—This involves an application crash, product instability, a
1. DEFINING A BUILD
I once was asked by a test manager to summarize everything I learned about
builds in one sentence. I told him that “there are no free lunches, especially
in the build lab, but there might be free beer.” He told me that he was disap-
pointed that I did not have anything deeper than that. He then said his motto
was “Testers close bugs.” I knew what he meant, so I said with tongue-in-cheek,
“Wow, that’s deep.” I’m not sure if he took that as a compliment or just thought I
was not very funny. Regardless, he did have a good point.
Let’s break down the details of “a bug’s life…”
When a developer fixes a bug on his machine, he marks the bug’s substatus
as Fix Available and keeps it assigned to himself. After he checks in the change
to the team branch or tree, he resolves the bug (changing the status from Active
to Resolved) and reassigns the bug to the original bug opener or a tester who
owns that area of the product.
The original bug opener or tester then waits until an official build comes out
that contains the bug fix. He then walks through the repro steps to ensure that
the bug has truly been fixed. If it has, he closes the bug by changing the status
from Resolved to Closed. If the issue still exists, the bug opener or tester reacti-
vates the bug by resetting the status to Active and reassigning it to the develop-
er. This continues until the bug is fixed or gets postponed for the next milestone
or release.
Kill
Bugs!
1. DEFINING A BUILD
teams to deliberate about what work items or bugs are fixed or open. This is
called triage.
Scott Guthrie is the product unit manager in Visual Studio. He explains tri-
age in his blog:
During tell mode, teams within our division are still given discretion to fix
any bugs they want—they just need to be prepared to present and explain
why they chose the ones they did to the central division ship room. This
ends up ensuring a common bar across the division, slows the rate of fixes,
and slowly brings up build quality. You might naturally wonder how not
fixing bugs could possibly bring up build quality, since this obviously seems
counterintuitive. Basically, the answer lies in the regression percentage I
talked about earlier for check-ins. Even with a low regression number,
you end up introducing new bugs in the product. (And when you have a
division of over 1,000 developers, even a low percentage regression rate
can mean lots of bugs introduced per week.) By slowing the rate of check-
ins, you slow the number of regressions. And if you focus the attention on
bad bugs and add [an] additional review process to make sure these fixes
don’t introduce regressions, the quality will go up significantly.
During ask mode, teams within our division then need to ask permission
of our central ship room committee before making a check-in—which
adds additional brakes to slow the check-in rate. In addition, all bugs
in ask mode must go through a full nightly automation run and buddy
testing (which takes at least 12 hours) to further guard against introducing
problems. Ask mode will also be the time when we’ll drive our stress-
passing numbers up to super-high levels, and we’ll use the low rate of
check-ins to find and fix pesky, hard-to-find stress failures.
You can read the entire entry at http://weblogs.asp.net/scottgu.
I talk more about processes to control all check-ins into the source tree in
Chapter 10, “Building Managed Code.”
14 Chapter 1 Defining a Build
Important Definitions
The following sections discuss terms that are specific to Visual Studio but
that are used all over the Web and at various companies I have visited.
Solution Files
If you are new to Visual Studio .NET, you probably are not familiar with
the term solution. A solution essentially represents everything you are
currently working on. Visual Studio .NET uses solutions as containers
for individual projects, which generate your system components (.NET
assemblies). Solution files maintain project dependency information and
are used primarily to control the build process.
Project
In the context of this book, projects are one of three types:
1. DEFINING A BUILD
from the Microsoft Consulting Services (MCS) group and is based on
the terms and the way that Microsoft organizes its software development
groups. The funny thing is that many people on the Microsoft product
teams have never heard of MSF. They use the processes or know the
terms, but they do not realize that Microsoft has been teaching this to
customers for years.
That is a good example of how a documented process came from an
informal undocumented process. Now the documented process (MSF) is
the leader, and many new terms in the product teams come out of MSF.
MSF will be included in the upcoming Visual Studio Team System. It’s a
great high-level view of how Microsoft runs its product teams. Because a
ton of information about MSF is available on the Microsoft Developers
Network (MSDN http://msdn.microsoft.com), I will show just one
chart that sums up the whole process (see Figure 1.4).
The Goals of Each MSF Role
Program
Satisfied Management Building to
customers specification
Product
Development
Management
Communication
User
Test
Experience
0321332059
01Fig04.eps
5/02/05
Scan Group/pab
16 Chapter 1 Defining a Build
Summary
Recommendations