0% found this document useful (0 votes)
126 views41 pages

Developer Velocity

Development

Uploaded by

Williams Truman
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
0% found this document useful (0 votes)
126 views41 pages

Developer Velocity

Development

Uploaded by

Williams Truman
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
Download as pdf or txt
You are on page 1/ 41

Developer Velocity

Lessons from Digital Leaders on


Accelerating Business Performance
through Software Excellence
Developer Velocity: Lessons from Digital Leaders 2

Contents

Executive summary 3

About the research 7

Deep dive on learnings in banking 10

Deep dive on learnings in retail 21

Learnings that apply across companies 30

Implications for technology leaders 36

How to increase Developer Velocity in your organization 37

Building a more resilient future together 38

About the Authors 40

Resources 41
Developer Velocity Phase II Report 3

Executive summary

Companies across industries are shifts have accelerated digital transformation


pursuing digital strategies and timelines, leading to a greater urgency for
transforming their business through companies to invest in software development
to adapt and thrive.
software development. Doing this
effectively requires companies to To understand how companies build high-
achieve superior Developer Velocity performing development organizations
Index (DVI) scores. while facing the challenges of COVID-19 and
remote development, we recently completed
This enables companies to tap into the full a second phase of research on Developer
range of skills and creativity of developer Velocity. This research focused on digital
teams by empowering them, creating the leaders in retail and banking—two industries
right environment to innovate, removing that have seen high impact and acceleration
points of friction, and turning ideas into of software investments through 2020.1,2 For
software that supports customer needs and example, in retail, e-commerce spending was
business objectives. up 30 percent between March and April,3 and
many customers expect to continue using
In early 2020, we conducted in-depth
online channels post-pandemic. Additionally,
research with more than 400 companies
both industries have unique challenges when
to both quantify the impact of Developer
building software. For example, banks need
Velocity on business performance and
to build systems that can handle complex
identify which drivers matter more than
regulatory and compliance requirements
others. Following our initial research,
and retailers need to build systems that can
significant durable shifts have occurred
handle millions of transactions daily across
driven by the COVID-19 pandemic. These
multiple channels.

¹Retail Reimagined report (McKinsey, 2020)


²Transforming the US consumer bank for the next normal (McKinsey, 2020)
³The reinvention of retail report (McKinsey, 2020)
Developer Velocity: Lessons from Digital Leaders 4

Specifically in this phase, we studied five In retail, we found Albertsons and a global
leading companies in depth:⁴ ABN AMRO, Quick Service Restaurant (QSR) faced a
Capital One, Albertsons, a leading global different set of challenges and have taken
Quick Service Restaurant (QSR) chain, and slightly different approaches to building
a large global US-based bank. In addition, software capabilities. However, some
we interviewed a few additional leading common themes emerged including:
retailers with global footprints. Each of these organizing around autonomous teams that
companies have made it a core focus to build can own a product end-to-end (checkout,
world-class software and build a culture that POS, etc.), and an aggressive push to cloud
attracts technical talent. In this whitepaper, for business applications—often prioritizing
we detail the steps they have taken and customer-facing workloads (e.g., the
the Developer Velocity drivers they have e-commerce stack) and the core backend
deployed. While these insights contain several systems (e.g., merchandising). For both
learnings for other retail and banking leaders, retailers, the impact has been high, enabling
many learnings also apply across industries. them to increase digital revenue growth.
For example, e-commerce was a key growth
We observed common patterns that predict driver for achieving its third straight quarter
success in the three banks we profiled. of more than 200 percent sales growth, up
Included among them is a strong push on 225 percent in Q3 of FY 2020.
software and cloud that functions as a joint
partnership across business and technology.
Each company also took a phased and
multi-year approach to building software
capabilities. This starts with the foundations
of adopting Agile, rearchitecting the code
base to be loosely coupled, up-leveling talent,
and migrating to public or hybrid cloud.
While doing this, each of the banks took a
thoughtful approach to security and risk—
automating and embedding controls within
the pipeline versus driving manual processes.

⁴We included 3 banks in this research, but in this document we will only discuss two banking
case studies in-depth.
Developer Velocity: Lessons from Digital Leaders 5

As mentioned above, there are


themes that can be universally
applied across industries.

In technology:

Successful cloud adoption requires Data architecture and data platform


a joint partnership across business modernization are the next evolution
and technology with a focus on of application platform modernization
accelerating business outcomes. In efforts for even the most successful
successful companies, business and companies. Data architecture is
technology often align on specific inseparable from companies’ efforts
timelines and outcomes that enable to capture application modernization
and empower development teams benefits such as enhanced
to identify the optimum cloud personalization, improved fraud
adoption path. detection, and real-time alerts.

For tools, companies often look Security is a major area of ongoing


to standardize their software investments, including shift-left,
development toolchain in key areas, automation, and embedding controls in
typically: CI/CD pipeline, planning, the pipeline itself. Companies see cloud
and code management tools, as an enabler to improve security.
leaving flexibility in other areas
to empower individual teams and
developers.
Developer Velocity: Lessons from Digital Leaders 6

In organizational enablement: In working practices:

Unsurprisingly, world-class talent Autonomous scope and loosely coupled


differentiates the top performers architecture are must-haves for Agile
from the rest. Companies across companies to succeed.
industries compete with leading tech
companies for the best talent through Open source software is a differentiator
culture and emphasizing their unique between the top performers versus
value proposition (i.e., the ability to the rest, particularly in attracting and
work on products that “impact the retaining top talent and increasing speed
lives of millions of people every day”). of innovation.

Product Management (PM) and Inner source adoption is early across the
adoption of a product and platform- board, but a top priority for companies
centric operating model is the next that are looking to achieve top-quartile
frontier for many organizations. Developer Velocity.
Common challenges include;
balancing PM skills across business This research has significant implications
domain knowledge, experience for technology leaders. In the early stages
design and technical skills, lack of a of transformation, leaders should look to
well-defined operating model across invest in foundations such as adoption
“business” and “technology,” and of Agile practices, integration of business
lack of a strong Technical Product and technology, cloud adoption, and re-
Management (TPM) function. organization into small autonomous teams. It
is critical for leaders to guide the organization
Finally, cloud adoption and modern through this early phase, constantly
tools differentiate companies that adopting the latest cloud technologies,
have parity between in-person and rearchitecting into microservices, driving
remote developer productivity. organizational agility, adopting open source,
and continuously integrating and deploying
(CI/CD) processes. The most successful
organizations are taking this continuous
improvement journey further by adopting
DevSecOps and building true product
management and design capabilities.
Developer Velocity Phase II Report 7

About the research

We define Developer Velocity as the second, third, and fourth quartiles


the ability to drive transformative is marginal. In addition, top-quartile
business performance through players appear to be more innovative,
scoring 55 percent higher on innovation
software development. This is
than bottom-quartile companies. Similar
achieved by empowering developers,
patterns hold within specific industries and
creating the right environment for sectors. For example, in financial services
them to innovate, and removing and retail, top-quartile companies saw
points of friction. positive revenue growth while average
revenue declined in the other quartiles.
In phase one of our research (early 2020), Additionally, of the more than 40 drivers
we surveyed 440 large enterprises and tested, four universally show up as the
quantified the impact of Developer Velocity most important for business performance:
on business performance, identifying which product management capabilities, culture,
drivers have more impact than others. As developer tools, and talent management.
a result, we created the Developer Velocity
Index (DVI), which pinpoints the most critical However, depending on where you are in
factors in achieving Developer Velocity, the journey or your industry, what matters
as well as some factors that are not as can vary. For example, for companies in the
important as many executives and observers bottom quartile, cloud adoption and Agile
might believe. One of the key findings from team practices matter significantly. For
the research was that across industries, companies in the top quartile, open source
companies with top quartile Developer adoption is the biggest driver of business
Velocity are growing revenue four to five performance. In retail and banking, cloud
times faster than their counterparts. Top- adoption is a major driver of performance.
quartile companies also have 60 percent In banking specifically, security and
higher total shareholder returns and 20 compliance practices are a major driver of
percent higher operating margins. However, Developer Velocity.
financial performance differences between
Developer Velocity: Lessons from Digital Leaders 8

In late 2020, we launched phase two of the 2. Developer Velocity drivers deployed
developer research focused on a select set Which capabilities have been most
of digital leaders to better understand how impactful in driving performance of
they have accelerated Developer Velocity. software development teams to achieve
This work investigates what has and has not business outcomes. In these areas, how did
worked well, and the impact on business they drive adoption and scale changes. We
performance. These companies have made asked what key learnings other software
significant investments in software over leaders could take away from their
the past several years. Within each of these experience, and what two to three areas
companies we interviewed senior technology they are looking to focus on next.
leaders, product managers, software
3. Impact of investments
architects, and developers across a range
The quantified impact of investments the
of product areas. This focus allows us to
companies made in Developer Velocity
present insights for organizations that have
drivers on business performance.
committed to technology transformation
with specific details best practices.
While each company is unique, the research
We focused our investigation in three areas: is meant both to distill the lessons from
these companies’ experiences and to identify
1. Overall software journey patterns that lead to success. Additionally, in
How the company has invested in this second phase of research we also added
software development capabilities and a set of new drivers focused on remote work
top business priorities. to address changes with COVID-19 and
application modernization.
Exhibit 1

Developer Velocity involves 57 drivers across 14 dimensions

Architecture Infrastructure and platform Tools


• Software architecture • Public cloud adoption (IaaS, PaaS) • Planning tools
Technology

• Data architecture • Infrastructure as code • Collaboration tools


• Application modernization • Development tools
Testing
• DevOps tools
• Test automation
• Low- or no-code tools
• Test-driven development
• AI assistance in development
Working practices

Engineering practices Security and compliance Agile team practices


• Technology debt • Security practices • Work-in-progress
management practices • Compliance practices management
• Coding guidelines • Agile ceremonies
• Code reviews Open source and inner source • Definition of done
• Open source usage and contribution
• CI/CD practices
• Inner source adoption

Team characteristics Talent management Organizational agility


• Autonomous scope • Incentives • Dependency management
• Limited context switching • Capability building • Funding mechanisms
Organizational enablement

• Cross-functional teams • Recruiting • Portfolio management


• Colocation of teams • Team health management
Remote work
• Employee value proposition
Culture • Remote work model
• Engineering career paths
• Psychological safety • Mental health
• Collaboration and Product management • Manager coaching
knowledge sharing • Product management capabilities • Asynchronous communication
• Continuous • Product telemetry • Video based communication
improvement culture
• Product vision • Remote collaboration tools
• Servant leadership
• Linkage between strategy and • Single source of truth
• Culture of customer team metrics • Core working hours
obsession
• Rapid prototyping • Security
• Remote working toolchain
Developer Velocity Phase II Report 10

Deep dive on learnings


in banking

Financial services technology is technology capabilities (e.g., significant


currently in the midst of a profound shifts in demand for digital channels,
transformation. Customer preferences remote working, new cybersecurity threats),
requiring executives to anticipate and
are shifting to include digital and
prepare for the future.⁵ For example, 15 to 20
omnichannel experiences.
percent of US customers said they expect to
increase their use of digital channels once the
Additionally, the importance of data to
pandemic has passed.⁶
drive initiatives such as new products and
underwriting capabilities, personalization,
However, driving a software transformation
fraud detection, and automation of core
in banking also has unique challenges. For
banking processes is higher than ever. In
example, banks operate in an environment of
a competitive environment of rising cost
complex regulations with stringent security
pressures, where rapid action and response
and compliance requirements. These are
is imperative, financial institutions must
also rapidly evolving to incorporate more
modernize their technology function to
complex risks as adoption of cloud, AI/ML
support expanded digitization of both the
techniques, and new technologies like crypto
front and back ends of their businesses.⁵
expand. This requires banks to automate
Software has accelerated some critical
governance and controls to ensure business
use cases for banks on both the front end
and technology teams have ready access
(e.g., digital banking, next-product-to-buy
to appropriate datasets, with the necessary
recommendations) and the back end (e.g.,
controls for security and permission where
credit risk analytics, fraud detection).
needed. Additionally, many banks recognize
that their feature development and release
Furthermore, the durable shifts driven by
cycles are longer and more complex due to
COVID-19 are putting immense pressure on

⁵Next-gen technology transformation in financial services report (McKinsey, 2020)


⁶Transforming the US consumer bank for the next normal (McKinsey, 2020)
Developer Velocity: Lessons from Digital Leaders 11

regulatory compliance issues, and thus are


adapting a fit-for-purpose Agile methodology.
Long-term planning methods are used
to create feature plans and to manage
dependencies. These plans are then broken
into checkpoints and MVPs for development
teams to build using Agile methodologies.

To understand how banks are adapting


and using software to accelerate business
performance, we studied three banks
in-depth: ABN AMRO, Capital One, and a
leading North American bank.

While we included three banks in this research,


for this document we will discuss only two case
studies in-depth. All three are large global
institutions that have taken very different
paths to becoming world-class software
organizations. However, we observed some
common patterns with banking; a strong,
all-in mindset with regards to software that is
jointly driven by business and technology, and
a phased, multi-year approach starting with
building the foundations before advancing to ABN AMRO
more mature capabilities. They have also taken
a very thoughtful and balanced approach
to driving speed and agility while meeting
the highest standard around security and
compliance, and viewing cloud adoption as an
Capital One
opportunity rather than a barrier.
Developer Velocity: Lessons from Digital Leaders 12

ABN AMRO

Strong focus on software development


to address growing demand for digital
banking.

Increased productivity by standardizing


development tools and the CI/CD
pipeline, and reinventing its developer
team operating model.

Today, ABN AMRO’s investments have led


to dramatic improvements in software
development: testing time has reduced
by 95 percent, 45 percent of workloads
are in cloud (with Microsoft Azure as
primary cloud vendor), and 100 percent
of teams use Agile methodology.
Developer Velocity: Lessons from Digital Leaders 13

ABN AMRO is a leading global bank Standardizing


that focuses on commercial, private,
development tools
and retail banking. As digital banking
grew in the early 2010s, ABN AMRO
and practices
2016–2017
recognized the need for greater
control over its digital products and ABN AMRO quickly realized that standardizing
faster time-to-market. toolsets across teams would help ensure that
developers could easily move among teams,
It embarked on an ambitious software
and that employees from one org could easily
development transformation through two
work with employees from another—whether
concurrent strategic initiatives: standardizing
developers or business leaders. The company
development tools and practices and
created a Center of Excellence (CoE) to develop
reinventing the team working model.
tooling standards, enforce adoption, and
provide training to all teams (e.g., Jenkins and
Azure DevOps for deployment). Importantly,
standardized tooling extended to business
functions and external vendors. For example,
everyone uses JIRA for product management,
scheduling feature requests, and release
schedules. Teams may request exceptions
when standardized tools do not meet their
requirements.

ABN AMRO then focused on consolidating


CI/CD pipelines into one pipeline per platform,
each of which enforces numerous quality checks.
Every release candidate must pass stringent
requirements, including security, automated
unit/functional/regression testing, and coding
style. Standardized controls ensured that
quality expectations were fully transparent to
developers, while also providing easier oversight
and management to technical leads.
Developer Velocity: Lessons from Digital Leaders 14

When we standardized and


enforced controls, the quality
improved dramatically, especially
in security … everyone knew the
standard they were held to."
Developer Velocity: Lessons from Digital Leaders 15

Reinventing team
operating models
2016–2017

ABN AMRO wanted the core unit of Further, a core ABN AMRO value is that
development to be a single Agile team, any third party it works with is considered
with each team accountable for a product a partner, not a contractor. Partners were
end-to-end. Technical leaders thus created included in broader decisions (such as
standardized teams of 8 to 10 people, each technical roadmaps) and internal trainings.
with developers who specialized in specific Some ABN AMRO employees even relocated
areas such as tooling, security, and testing. to offshore development facilities to form
Teams often included an embedded closer bonds with partner teams. Their efforts
business product owner who guided were so successful that many partner teams
planning and ensured sprint cycles aligned hung pictures of Amsterdam on their walls and
with business priorities. adopted ABN AMRO’s values as their own.

Impact

The investments to standardize tools, engineering


practices and reinvent the team operating model
have paid significant dividends. Productivity
33%
Productivity increased
increased by 33 percent after the changes, all

95%
teams use Agile methodology, testing time
has declined by 95 percent, and 45 percent of
workloads are cloud-based, with Microsoft Azure
as primary cloud vendor. Less testing time
Developer Velocity: Lessons from Digital Leaders 16

Capital One

The bank is an early adopter in digital


and achieved software success through
a well-planned, three-phase journey
beginning in 2011.

Some of the highlights include 80


percent of workloads are now cloud-
based and 75 percent of customer
interactions are digital.
Developer Velocity: Lessons from Digital Leaders 17

Capital One is a leading bank focused Phase 1:


on credit card, consumer banking, Laying the foundation
and commercial banking. Its CEO and 2011–2014
executive team have for more than
a decade recognized that an all-in The journey started with owning the
technology stack end-to-end to enable the
commitment to technology, and
flexibility and speed that full ownership
specifically cloud technology, was the
provides. This included in-sourcing tech
best way to ensure Capital One could talent, adoption of Agile and DevOps, and
meet the growing customer demand modernizing tech infrastructure.
for digital banking and stay ahead of
fintech disruptors. In-sourcing talent:
Capital One values in-house talent across
Through this period, it has successfully the company to drive employee ownership
moved the majority of workloads to and quality. Hiring technical talent became
public cloud hosting and the majority of a key focus, especially concentrating talent
software development in-house. It also on primary areas that drive business growth.
has cloud-native development, attracted Building partnerships with universities was
top development talent, and built a especially important, as was training and
strong technology culture. Capital One’s growing early-in-career talent with both
transformation was executed in the three banking domain expertise and technical
phases outlined below. knowledge. Capital One attracted talent with
a unique value proposition: solving real world
customer problems at a scale of millions of
users per day.

Agile development practices:


In early stages of the transformation, every
team adopted Agile with consistent team
set-up and sprint cycles. Shifting the culture to
thinking in terms of continuous delivery and
improvement enabled Capital One to break
ambitious projects into concrete, incremental
steps that delivered on time. Teams also
adopted shared collaboration tools to ensure
Agile teams could work together easily.
Developer Velocity: Lessons from Digital Leaders 18

Early adoption of agile was key.


We had to shift the developers’
mindset before we changed
anything else."
Developer Velocity: Lessons from Digital Leaders 19

Moving to a microservice oriented


architecture and APIs:
Capital One wanted each Agile team to have
end-to-end ownership of a specific service, and The aggressive, top-
thus architects decomposed the monolithic down commitment to
code base into a service-oriented architecture.
Giving each team ownership of an independent
the long-term vision
service enabled Capital One to prioritize was remarkable…the
upgrading individual services according to fearlessness of an all-in
business needs using a single accountable team.
Furthermore, embedded business specialists
commitment to cloud
helped translate business priorities into cannot be underestimated."
technical upgrades scheduled for specific Agile
sprints, ensuring complete alignment between Development teams choose their own
business and technical priorities. migration schedule and strategy:
Tech leaders gave teams control over
their migration schedule and strategy, but
Phase 2: provided an assessment framework to aid
Migrating to the cloud decision-making. The framework included
2014–2018 both business and technical needs, from
anticipated usage and cost to security needs
All-in, joint business and technical and software dependencies on other teams.
commitment to migration: Teams then made their own migrations plans,
The CEO announced an all-in commitment to with light-touch coordination from a central
a cloud-based model, and the senior business infrastructure team.
and technical leaders pushed progress to
quickly remove roadblocks through daily Early adopter teams share best practices:
stand-up meetings. Early adopter teams pioneered cloud
migration. The earliest teams experimented
with different migration strategies, cloud
architectures, and tools until they found a set
that worked for the bank’s needs. Those teams
then helped guide the next waves of adopters
by establishing best practices, toolsets, and
step-by-step guides.
Developer Velocity: Lessons from Digital Leaders 20

Phase 3: tests, security compliance checks, style-guide


Advanced phases conformity, and more. Only code that passes
all tests is approved for release. Capital One
of maturity therefore preserved teams’ autonomy to
2018–present
develop and innovate, while also gaining
efficiency and ensuring all software meets the
The current phase focuses on enforcing
same stringent quality standards for release.
consistently high-quality delivery across all
teams. Historically, teams had the autonomy
to develop their own CI/CD practices with
light guardrails to ensure they were meeting
company standards. When technical leaders
started seeing diminishing returns from team- Aligned autonomy … it is
level investments in DevOps tooling, they
giving our teams enough
adjusted the approach, moving to a shared
enterprise pipeline framework that would shared context and clarity
enable more rapid automation of controls and on where to go, and then
standardization of critical processes. All code
let them be autonomous to
must pass more than 20 compliance checks
enforced in the CI/CD pipeline, including accomplish it."
unit, functional, integration, and regression

Impact

Some of the highlights of impact include


majority of workloads being cloud-based, 75%
majority of software development in-house,
cloud-native development and talent, and
a strong tech culture. More than 75 percent
customer interactions are digital today.⁷ Digital customer interactions

⁷Doing business the digital way: How Capital One fundamentally disrupted the financial services industry
(Capgemini Consulting, 2017)
Developer Velocity Phase II Report 21

Deep dive on
learnings in retail

For retailers, software has increasingly brands or retailers through the pandemic and
become the basis for growth, e-commerce spending was up 30 percent from
including hyper-personalized the beginning of March through mid-April of
2020 as the appetite for digital and contactless
experiences and omnichannel
ways of shopping intensified.
adoption. These changes drive high-
priority digital use cases on both the However, driving a software transformation
front-end (which affects customer in retail is not without its unique challenges.
experiences) and the backend (which Large retailers handle millions of transactions
helps manage core retail systems such across diverse channels, and need to maintain
as merchandising and supply chain). consistency of experience and data across
these transactions and channels. Additionally,
The rewards for those who get this right can retailers that are driving growth across physical
be significant. For example, McKinsey research and digital channels can have a fragmented
suggests that effective personalization can technology landscape. For example, they
increase store revenues by 20 to 30 percent.⁸ may be using legacy on-prem technology
for point-of-sale and store management
In this context, our earlier research highlighted systems at their physical locations and cloud
the importance of Developer Velocity in applications for their e-commerce stores. It
retail. In the light of the COVID-19 crisis, may only take a few minutes to publish new
there is even more urgency for companies features to their mobile apps, yet updating
to invest in effective and impactful software physical store systems may take weeks.
development to adapt and thrive. COVID-19
drove significant durable changes for retail; Finally, many retailers own multiple brands
by some estimates,⁹ retailers have vaulted across multiple geographies, often as the
10 years ahead in consumer and business result of acquisitions. This can also produce a
digital penetration in less than three months. fragmented technology stack, with disparate
Forty-six percent of US consumers switched systems that do not communicate.

⁸A transformation in store (McKinsey, 2019)


⁹The reinvention of retail report (McKinsey,2020)
Developer Velocity: Lessons from Digital Leaders 22

Many retailers recognize this challenge and For each of these digital leaders, we observed
are investing in the consolidation of their a methodical and phased approach to
technology stack data platforms, and are driving a multi-year software program. They
building loosely coupled services that are have invested in foundational capabilities
available as a platform for developers. with Agile, DevOps, and close integration
of business and technology. They have also
To understand the ways that successful made an ambitious push to public cloud
retailers are overcoming these constraints, starting with the customer facing experiences
and leveraging software to differentiate and the e-commerce stack, with an eye to
themselves, we deeply studied two leading driving speed.

Finally, many retailers own multiple The companies we studied have seen massive
brands across multiple geographies, impact from their investments in software.
often as the result of acquisitions. Their digital businesses have grown one to
This can also produce a fragmented three times in recent years, driving business
technology stack, with disparate success even during the COVID-19 pandemic.
systems that do not communicate.
Many retailers recognize this
challenge and are investing in the
consolidation of their technology
stack data platforms, and are building
loosely coupled services that are
available as a platform for developers.

companies that operate in very different


parts of retail: Albertsons—a grocery
company—and a global Quick Service
Restaurant (QSR) chain. Both are large global
Albertsons
brands that have software as a company
priority, and are on a mission to become a
world-class place for developers to grow and
innovate. Aside from in-depth research into Global Quick Service
these two retailers, we also interviewed two Restaurant (QSR) chain
more leading retailers with global footprints QSR chain
for additional insights.
Developer Velocity: Lessons from Digital Leaders 23

Albertsons

Transformed software development to meet


rapidly growing demand for online orders.

Focused on having cross-functional Agile


teams, developing a loosely coupled
architecture, and migrating to public cloud.

Impact of the investments has been high:


Albertsons e-commerce is a key growth
driver, as it recently achieved its third straight
quarter of more than 200% sales growth, up
225% in Q3 of FY 2020.
Developer Velocity: Lessons from Digital Leaders 24

Albertsons Companies is a leading US Laying the foundation


grocer with 2,000+ physical stores and Building standardized
more than 20 store brands including Agile teams that own
Safeway, Albertsons, Vons, Jewel-Osco,
unique services
Shaw’s, Acme, Tom Thumb, Randalls,
2017–2018
and United Supermarkets.
The first challenge the Albertsons team
Albertsons transformed its software took on was making the organization more
development capabilities in multiple responsive to customer needs. Albertsons
phases. First, Albertsons adopted a structure rearranged teams into cross-functional,
for its development teams that enabled independent teams with 6–10 developers,
autonomy and agility. Teams would be PMs, architects, and testers. Each team now
cross-functional and own the product end- owns a service end-to-end. Teams also began
to-end. Simultaneously, Albertsons drove using a set of standard collaboration and code
an aggressive migration to public cloud and management tools, including GitHub.
developed a loosely coupled architecture.
A central team of enterprise architects then
led the effort to develop loosely coupled
services (e.g., in e-commerce: shopping cart,
checkout.) Each team then owned a single
service over which it had full accountability
and ownership.
Developer Velocity: Lessons from Digital Leaders 25

Moving beyond basics ensure momentum. Individual services were


Migrating the rearchitected independently with a test-and-
learn approach to ensure zero downtime.
e-commerce Services were migrated sequentially; each
stack to Azure individual service was upgraded, stabilized,
and rearchitecting and incorporated into the e-commerce
stack before moving to the next service. The
individual services central architect team sequenced services
2018–2019
based on a combination of business and
Albertsons chose a two-step approach to technical needs, with critical customer
migrate the e-commerce technology stack features or foundational tech dependencies
to Azure. First, they adopted a lift and shift (e.g., the data model) going first. This
approach to migrate the e-commerce stack to sequential, progressive modernization also
cloud. The leadership team ensured there was ensured the team was able to learn best
a clear execution plan along the way that was practices from early upgrades that were
communicated to both business and tech to useful for later services.

Impact

85%
Impact of investments at Albertsons has been
high. E-commerce is a key growth driver for
the company, as it recently achieved its third
straight quarter of more than 200% sales
growth—up 225% in Q3 of FY 2020. Developers Launch time decreased

50%
are also proud of their work. Launch time for
new customer features has reduced by more
than 85 percent, and testing time by more than
50 percent, while maintaining quality. About
70 percent of the customer-facing e-commerce Less testing time
stack is in the public cloud, compared to zero
percent just three years ago.
Developer Velocity: Lessons from Digital Leaders 26

Global Quick
Service Restaurant
(QSR) chain

Digital transformation with a goal to


meet customers’ increasing demand for
digital experiences, especially mobile
ordering, loyalty, and AI-driven apps.

Transformed its development org with


investments in cloud, standardization
of engineering practices, and focus on
talent and culture.

One achievement that stands out is that


digital sales have grown 100 percent in
the last two years.
Developer Velocity: Lessons from Digital Leaders 27

This retailer is a global, fast, casual As a starting point, the company decided a
restaurant chain with $10B+ revenues cloud-migration initiative supported with
in 2019. It has transformed its software executive commitment was the best way to
ensure a successful migration. Executives were
development organization over
given specific cloud-adoption initiatives and
the years via a strong focus in three
targets to incentivize migration. They also
areas: cloud migration, adoption of recognized that each department had its own
world class tools and engineering technical needs, maturity, and business case for
practices, and a focus on talent cloud migration. Each department was given
and culture. These key investments target timelines to migrate but empowered
helped it dramatically improve digital to choose the right strategy. For example, one
performance. In the last two years, team strategically migrated services according
digital orders have grown 100 percent. to the business case, optimizing for cost and
rapid scalability to meet growing demand.
In contrast, another team migrated more
opportunistically. Many strategies can lead to
Migration to a successful cloud migration, and the retailer
public cloud recognized that its orgs were best suited to
decide their own paths.
In 2017, the company drove an aggressive
push to public cloud. It adopted a three- Lastly, technical leaders created a cloud Center
pronged approach to making this transition of Excellence (CoE) to accelerate the migration.
successful; a joint partnership across business Their focus was on evangelizing the technical
and technology, ownership and autonomy to benefits of cloud migration, leading education
teams, and a cloud technical director leading a and training efforts on cloud technology,
cloud Center of Excellence (CoE). creating learning curricula for developing in the
cloud, and disseminating best practices. The
CoE ensured that every organization had access
to the expertise and coaching needed to make
the migration successful.
Developer Velocity: Lessons from Digital Leaders 28

Adoption of world
class tools and
engineering practices
across the entire
organization
On tools and engineering practices, the for its DevOps requires them to enforce strict
company focused in three areas: standardized quality and compliance controls to ensure
tools and processes for cross-functional high standards are met across all teams. The
collaboration while maintaining autonomy QSR requires all code to be released through
for teams, full-stack DevOps teams using standardized, automated CI/CD pipelines
standardized CI/CD pipelines, and security to enforce quality at each step. At code
embedded into day-to-day development. check-in, security and compliance checks run
automatically. Tests, including unit, functional,
As a foundational capability, the QSR drove integration, regression, and capacity tests,
high levels of consistency in tools and automatically run every night and during
processes, but empowered the teams to deployment. Finally, the production code
have the flexibility to select tools and ways is constantly monitored, and alerts sent
of working that are unique to the individual whenever issues are detected.
team. JIRA planning tools, Confluence
documentation, and GitHub repositories are Lastly, embedding security in the day-to-day
required for working across teams. Cross- development cycles is a priority for teams. All
training and standardized learning journeys developers are trained in secure-by-design
ensure that developers are well trained on all methodology and incorporate best practices
toolsets, enabling developers to move among into the code. They also enforce robust data
teams with minimal friction. classification systems, ensuring that data
receives differential treatment and privacy
Additionally, the company has made a depending on its sensitivity. A central security
significant push on adopting the DevOps team is also available to help developers refine
model. All teams are full-stack DevOps teams; their work, from providing risk assessments to
responsibility for the service end-to-end instills consulting on how to improve service security.
a sense of ownership and enables teams to
see the impact they have on customers daily.
However, granting every team responsibility
Developer Velocity: Lessons from Digital Leaders 29

Investments in talent
and culture
The retailer recognized early on that meeting that scale and crossing the physical-
developer talent was critical to its success. digital divide attracts developers looking for
When competing for talent, the focus is on unique challenges. The company also focuses
clearly articulating the mission and what on engaging with the developer community
makes it a unique place to work. For example, via conferences and events. Most importantly,
the ability to work on systems that transact the retailer is well known for deeply caring
millions of orders per day globally, across for its employees, and that reputation alone
both digital channels and through thousands attracts thousands of applicants every year.
of brick-and-mortar stores. The demands of

Impact

100%
The investments in cloud, engineering
practices, and talent have paid significant
dividends. Almost all code is deployed
through standardized CI/CD pipelines, and a
Digital sales growth
significant portion of infrastructure is on the
public cloud. Digital sales have grown 100
percent in two years.
Developer Velocity Phase II Report 30

Learnings that apply


across companies

Even though every company is On Technology


unique, technology leaders at each
1. For cloud adoption and modernization
of the companies we studied faced of business applications, two elements
similar challenges in their software emerge as enablers for success:
transformation. Their responses
Joint partnership across business and
and lessons are valuable for other
technology with focus on accelerating
companies regardless of industry.
business outcomes

Successful companies, often have


business and technology come
together to agree on specific timelines
and outcomes and then enable and
empower development teams to
identify the optimum migration path.
For example, one US banking CEO
set hard timelines for teams to move
resulting in a successful transition. At
another bank, effort floundered for a
long period as there was no top-down
mandate and migration was seen as an
“engineering effort.” In coming together,
business and technology can agree on
specific timelines and outcomes and
then enable and empower development
teams to identify the optimum
migration path.
Developer Velocity: Lessons from Digital Leaders 31

Cloud Center of Excellence (CoE) that We observe that standardization in a few key
builds a “scaffolding” and provides areas such as the CI/CD pipelines can drive
support for teams as they migrate higher levels of autonomy and psychological
safety for developers. Standardization helps
Successful companies create a increase the confidence of developers to push
recommended path for cloud adoption code to production while reducing friction
and application modernization that and manual quality, risk, and compliance
makes it easy for a vast majority of reviews. Standardization of planning and
teams to thrive. They also give teams code-management tools helps organizations
with advanced track records or unique coordinate and manage dependencies more
needs the flexibility to choose their own easily, allowing developers to distribute
tools, architecture, and services as long knowledge and share learnings. In other areas,
as they adhere to baseline standards organizations should look to be flexible and
on security, quality, availability, allow developers to pick tools that support
performance, etc. in the cloud. success. For example, flexibility in dev tools such
as an IDE is a critical factor that contributes to
2. For tools, companies often look to developer happiness in many organizations.
standardize the software development
toolchain in a few key areas—typically Some of our case studies found that early
CI/CD pipeline, planning, and code software transformations were fragmented by
management tools—and leave flexibility organizations that experimented to identify
in other areas to individual teams and the best tools. This fragmentation can make
developers. the adoption of standard tools difficult. After
ensuring uptake is supported by changes
Earlier research showed that tools are the top
in culture, decision makers can clearly and
driver of Developer Velocity; companies with
positively communicate the tools' benefits for
top-quartile scores on tooling have developer
the individual developer.
satisfaction scores that are 47 percent higher
than companies in the bottom quartile. 3. Data architecture and data platform
However, many companies struggle with modernization are the next evolution of
the balance between maintaining a level of application platform modernization efforts
standardization versus empowering teams with for even the most successful companies.
the flexibility to use tools of their choice.
Developer Velocity: Lessons from Digital Leaders 32

In most cases, application platform such as data governance tools and


modernization—through application processes, incorporating data scientists
rearchitecture and cloud adoption—is a and engineers into Agile teams, and
high priority at the outset of a software training teams on responsible usage of
transformation. Many organizations have AI/ML.
varied investments for different categories
4. Security is a major area of ongoing
of apps. For example, while moving apps to
investments (including shift-left,
the cloud, many companies incrementally
automation, and embedding controls in
rearchitect applications using microservice-
the pipelines itself). Companies approach
based architecture and Kubernetes as a
cloud as an opportunity when doing this.
standard way for customers to orchestrate
containers at scale.
In a world where customers infrequently
contact staff, but rather interact almost
However, data architecture and platforms
entirely through digital channels, “digital trust”
are often under-invested areas when
has fast become a significant differentiator
addressing the full scope of application
of customer experience. Especially with
modernization. Data is inseparable from a
cloud adoption, security is fundamentally
company’s effort to capture benefits such
shifting from being centralized and run by a
as enhanced personalization, improved
handful of security experts, to decentralized
fraud detection, and real-time alerts. The
and in the hands of thousands of individual
data lakes and customer data platforms
developers. Successful companies are
required to support such technologies
building security at the most fundamental
have increased the complexity of a
level in their tech stack. They are embedding
company’s data architecture and created
security in their pipelines, driving high levels
more heterogeneity and duplication. The
of automation, and embedding secure
most successful companies recognize the
development practices into the Agile teams
challenges in data, and are starting to
themselves. As they do this, companies
prioritize and invest in this area. Specifically,
approach cloud as an enabler, through ability
these companies have begun to modernize
to automate, embed controls and provide
their data ecosystems and architecture,
access to the latest tools and technologies.
including consolidating data pipelines,
low-to-no duplication of databases, and
standardized tools and data structures. They
are also building supporting capabilities
Developer Velocity: Lessons from Digital Leaders 33

On working practices

5. Autonomous scope and loosely coupled We found in our earlier research that open
architecture are must-haves for Agile to source adoption is highly correlated with
succeed. outperformance in business results and
innovation. Companies with high open source
Agile methodologies, where teams are not
scores realize three times more impact on
autonomous and cannot deliver business
innovation and 20 percent higher developer
value independently, are unlikely to deliver
satisfaction compared to peers, assuming
favorable results. Successful companies
foundational capabilities are strong.
tackle these topics concurrently. For example,
one executive, when asked about what
Using and contributing to open source
drove success early in their company’s
projects is a significant way to attract and
software journey, told us “Agile is equal to
retain development talent. Indeed, the ability
architecture.” On the other hand, the lack
to contribute to open source projects is an
of architectural changes alongside an Agile
important way for developers to build their
transformation can result in a high degree
network and create a stronger technical
of interdependences in backlog, significant
reputation. Usage and contribution to
time spent on “stabilization” or “integration”
open source code is an important way for
of codebase, and a general culture of lack of
companies to signal that they embrace
ownership, empowerment, and accountability.
innovation, no matter the source.

In addition to architectural changes,


7. Inner source adoption is early across the
companies also reorganize teams to have a
board, but a top priority for more mature
more autonomous scope. In this model, teams
companies.
have end-to-end ownership of a product, are
able to push to production independently, and Inner source is a topic that we found to be
can be accountable for delivering business top of the agenda for all companies as they
outcomes. look to move into more advanced stages of
software development excellence. This is an
6. Open source software is a differentiator area that is still nascent in its adoption, yet
between the top performers versus companies are seeing strong results even in
the rest—particularly for attracting and early stages of adoption. Like open source,
retaining top talent and increasing speed of inner source adoption creates a culture of
innovation. openness, transparency, and knowledge
sharing, increasing the speed of innovation
Developer Velocity: Lessons from Digital Leaders 34

and bringing follow-on benefits in culture Another approach we see companies take is
and the ability to retain the best talent. to focus on a smaller percentage of top talent
Some successful companies are increasingly that can be embedded in the most important
shifting to a platform-oriented model, roles. They use this critical pool of talent to both
encouraging teams to both contribute accelerate the most important projects and
services to the platform and leverage build capabilities in the rest of the organization.
services wherever possible. The focus should not just be on leaders, but also
strong individual contributors who can have
On organizational enablement disproportionate impact. The hiring, retention,
8. Unsurprisingly, world-class talent performance management, and succession
differentiates the top performers from planning in these critical roles are often led
the rest. directly by CxOs.¹⁰
Companies across industries compete with
leading tech companies for the best talent 9. Product Management (PM) is the next
through culture and emphasizing their frontier for many organizations. Common
unique value proposition (i.e., “the ability challenges include: balancing PM skills across
to work on products that impact the lives of business domain knowledge, experience
millions of people every day”). design and technical skills, lack of a well-
defined product- and platform-centric
In our previous research, talent showed up operating model across “business” and
as top four driver for business performance. “technology,” and lack of a strong TPM
Successful companies make technical talent function.
a C-suite priority and articulate a distinctive
value proposition to appeal to technical Based on our previous research, a well-rounded
talent. For instance, many retailers tout the product management function is critical. A
opportunity for developers to solve technical strong product management function is even
problems that affect millions of consumers, more critical with remote and distributed teams,
especially in the context of the COVID-19 crisis. where clarity and alignment on outcomes
The top performers in attracting talent also matter more than ever. Far too often, product
invest in campus programs and leverage their managers in banks and retailers have been
brand recognition to attract talent from top merely a re-branding of an existing business
local and national programs. role, resulting in gaps in capabilities of areas

¹⁰Linking talent to value (McKinsey, 2018)


Developer Velocity: Lessons from Digital Leaders 35

such as experience design and technical a comparable role and struggle to achieve
skills. Successful companies realize that there their goals from these efforts.
are no shortcuts, and they are investing
significantly in hiring and developing PMs 10. Finally, cloud adoption and modern tools
who have balanced skills across business differentiate companies that have parity
domain knowledge, experience design, and between in-person and remote developer
technical skills. productivity.

The context of the COVID-19 pandemic


Another common failure model we observed
accelerated the shift to remote around the
is the presence of an unclear product- and
globe. Public cloud adoption and tooling
platform-centric operating model. This often
sophistication enabled software development
starts with the lack of a clear definition of a
teams in retailers and banks to have
product or platform. An effective operating
developer productivity parity between in-
model should include a holistic definition of
person and remote. For example, adoption
product and platform that is not constrained
of collaboration tools such as Microsoft
to monetized experiences, but rather any
Teams skyrocketed for organizations to
software that delivers value to an internal
connect remotely. Existing CI/CD toolchains
or external stakeholder. Effective operating
enabled development teams to collaborate
models should also include clear product
effectively on code base, and public cloud
roadmaps (versus Short-term projects),
adoption enabled companies to quickly scale
Agile funding mechanisms, outcome-based
resources to support an influx of employees
goal setting (leveraging OKRs for example),
working remotely—and millions of customers
quarterly and monthly product reviews, and
performing transactions online.
well-defined roles and accountabilities across
the entire development lifecycle. Many employees are likely to continue to work
remotely after the COVID-19 crisis. An April
Finally, the challenge of applying a product
2020 Gallup poll found that three in five US
management discipline is even greater for
workers who have been working from home
complex or large engineering projects, such
during the COVID-19 pandemic would prefer
as modernizing the tech stack. At software
to continue to work remotely as much as
companies, technical product managers
possible, even after public health restrictions
(TPMs) tend to fulfill these roles, but many
are lifted.¹¹
companies outside the software industry lack

¹¹U.S. workers discovering affinity for remote work (Gallup, 2020)


Developer Velocity Phase II Report 36

Implications for
technology leaders

Becoming world-class at developing agility (that is, going beyond Agile at the team
software is a strategic imperative level to agility in enterprise-wide planning,
for companies across industries. As dependency management, risk management,
etc.); continuously integrating; and
we have seen from the case studies,
continuously refining deployment processes.
getting there is a multi-year journey.
If done well, the above capabilities will deliver
The most successful companies, ones significant results through the ability to reach
that are almost indistinguishable from customer and market inputs quickly and go
a native software company, have been from idea to release in days and weeks versus
on this journey for close to a decade. months and years. However, leaders cannot
And in the true spirit of continuous rest on their achievements. The most successful
improvement, they are continuing to organizations are taking their continuous
explore new horizons. improvement journey and investing in areas
such as DevSecOps, building true product
management and design capabilities (similar
In the early stages of transformation, leaders to software companies, where product
should look to invest in adopting Agile managers can own and deliver end to end
practices and restructuring into small, business outcomes), adoption of low-code
autonomous teams with persistent missions. tools, and transforming their data and analytics
Leaders should bring business and technology architecture.
together on a day-to-day level, attracting
top-tier talent in-house, and investing in cloud The individual journey, pace, and sequence
adoption and application modernization. of how the different levers are deployed will
be unique to every company. However, there
Leaders should help guide the organization is one element that we believe is a constant:
through ongoing challenges: the continued transformation requires a close partnership
adopting of the latest cloud technologies; across business and technology, with the focus
rearchitecting into microservices; adopting of the entire leadership team, over a long
open source; embracing organizational period of time.
Developer Velocity Phase II Report 37

How to increase Developer


Velocity in your organization

Developer Velocity helps you unleash The Developer Velocity Index (DVI) is a score
the full potential of your developer calculated by assessing a company in three
categories: Technology, Working Practices, and
teams, drive innovation, and boost
Organizational Enablement. You can benchmark
business performance. Today, more
the maturity DVI score for your organization
than ever, investing in software relative to the peers in your industry. The
excellence and building a culture Developer Velocity assessment provides key
that empowers developer teams learnings and actionable guidance to boost your
will continue to be critical for every Developer Velocity and drive business outcomes.
organization’s success.
Calculating your Developer Velocity Index is
The digital transformation journey often straightforward. For this purpose, Microsoft is
begins when a company realizes it increasingly helping many of our customers by providing
needs to operate like a software company. unique guidance and recommendations to
Developer Velocity is essential for a company increase Developer Velocity through the
to drive better business results through Developer Velocity Assessment tool.
empowered developer teams.

Take a self-guided
Developer Velocity Assessment by
visiting https://aka.ms/dva or
by clicking on “Take the assessment”
at azure.com/developervelocity.
Developer Velocity Phase II Report 38

Building a more
resilient future together

Our mission at Microsoft is to Regardless of coding ability, low-code


empower every person and tools continue to grow for professional
organization on the planet to achieve developers and business users. Best-in-class
tools empower developers within every
more. And our goal is to empower
organization to keep developer talent
developers with the platforms
happy and motivated and accelerate digital
and tools that will make every skilling.
organization incredibly successful.
To help developers build productively,
collaborate securely, and scale innovation—
Over the past few years, we’ve seen many no matter where they are—Microsoft offers
developers around the world building the world’s most comprehensive developer
amazing customer applications and internal toolkit with Visual Studio, GitHub, Power
line-of-business applications. Apps, and Azure. This unique toolset makes
developer teams incredibly productive
In recent months, every organization when building cloud-based solutions.
has been affected by COVID-19 with
unprecedented disruption. We’ve seen Developers are also playing a key role
massive shifts in the way many organizations in accelerated innovation, building and
connect with employees and customers. shipping global and secure cloud-based
Businesses and organizations of all sizes are applications faster.
adopting solutions to enable remote work
and are creating brand-new, cloud-based Our research shows that companies often
applications while staying secure. realize large improvements in business
performance when cloud adoption is
Superior tools can enable developers to high. Public-cloud adoption as a catalyst
be more resilient and help them code, of Developer Velocity is especially strong
collaborate, and ship from anywhere, securely. for non-software companies, having up
They prefer developer tools and cloud to four times the impact on their business
platforms that offer ready access to open performance than it does for software
source ecosystems. companies.
Developer Velocity: Lessons from Digital Leaders 39

Application migration and modernization like Microsoft 365, LinkedIn, Dynamics 365,
is top of mind for many organizations, and the Power Platform to connect your
and more recently, public cloud adoption employees and business processes to your
has enabled companies to quickly scale custom applications.
resources to support an influx of employees
working remotely—and millions of As we enter a new post-COVID era, we see
customers shifting to online. massive improvements on application time-
to-market thanks to shifts to remote developer
Increasingly, we are seeing customers who productivity and public cloud adoption.
value the idea of innovating at a faster pace
by using cloud-native architectures with Whether your goal is to modernize and build
loosely coupled microservices, managed new apps, continuously and securely deploy
databases, AI, DevOps support, and built-in your code to the cloud, contribute to open
monitoring to detect failures and problems source, or rapidly build apps with low-code
before they happen. platforms, Microsoft is pleased to play a
small part supporting developer teams and
Such apps are typically powered by elastic thousands of organizations around the world
infrastructure that adjusts to the varying to increase Developer Velocity.
load, supports zero downtime rollouts,
and provides low latency access to data
worldwide.

Microsoft is working closely with thousands


of organizations around the world to help
them use the Microsoft cloud to address
today’s biggest challenges. One of the things
that makes the Microsoft cloud unique is
how comprehensive it is. With Azure, we
provide an end-to-end cloud platform that
enables you to develop custom applications
and leverage cloud infrastructure. You can
also take advantage of our SaaS solutions
Developer Velocity: Lessons from Digital Leaders 40

About the Authors

Erika Ehrli is a Director of Product Marketing with Microsoft Azure focused


on cloud application development with Visual Studio, GitHub, and Azure.
She serves as a cross-functional business leader working across engineering,
product marketing, and sales leadership on defining go-to-market strategy
for application development in Azure and open source in Azure. She leads a
technical product marketing team focused on narratives, customer stories,
technical content, as well as flagship event content strategy to support high-
impact product launches. In 2019 she launched the cross-company Developer
Velocity Thought Leadership program, research, and go-to-market including
envisioning and launching the Microsoft Developer Velocity Assessment Tool.
Erika has worked at Microsoft for 17 years with different roles in consulting,
engineering, and developer marketing.

Larry Larsen is a Senior Product Marketing Manager with Microsoft Azure


focused on cloud application development and customer engagement. He
serves as a cross-team storyteller and produces event content and Developer
Stories that explore the challenges and successes of cloud customers. Larry
previously led the Channel 9 developer network hosting hundreds of technical
shows and Microsoft's live developer events. Prior to Microsoft Larry worked at
The Poynter Institute where he founded Poynter Labs, an experimental hands-
on technology center that taught thousands of working journalists from around
the world how to embrace and leverage new technologies.

Microsoft would like to thank Dr. Microsoft would like to thank McKinsey & Company,
Nicole Forsgren, PhD as research GitHub, and HashiCorp for their contributions to
advisor and reviewer for this paper. parts of the Developer Velocity research.
Developer Velocity: Lessons from Digital Leaders 41

Resources
Developer Velocity

Developer Velocity Assessment

McKinsey & Company Report: Developer Velocity: How software


excellence fuels business performance

McKinsey & Company Article: Developer Velocity at work:


Key lessons from industry digital leaders

© 2021 Microsoft Corporation. All rights reserved.


This document is provided “as is.” Information and views expressed in this document, including URL
and other internet website references, may change without notice. You bear the risk of using it. This
document does not provide you with any legal rights to any intellectual property in any Microsoft
product. You may copy and use this document for your internal, reference purposes.

You might also like