Module 7

Continuous Delivery

It is also referred as pipeline.

It represents workflows,activities and
automation need to provide continuous
release of value to end user.
Components of continuous delivery
pipeline
• Where the source code resides
Version Control • Include Git,SVN etc

System(VCS) • Developers commit change to repository &triggers CD

pipeline

Build stage • Code compiles without errors

Automated • After this automated tests are run

• Include unit test, integration tests and end to end test
testing • If tests fail pipeline stop alert developers fix problem

• It may be used to check code quality, adherence to coding

Quality checks standards, security vulnerabilities &potential bugs.
• It ensures that code meets predefined criteria before
&code analysis deployment

Deployment • Code passes all tests and quality check.

• It involve packaging the application &deploy to specific
stage environment(development, staging or production)
Components of continuous delivery pipeline
Mon • Post deployment, monitoring tools are used to track the performance
itori &behavior of the application in production environment
ng • Feedback loops help in identifying issues
&Fe
edb
ack
Fee • Feedback from monitoring inform developers about any issues
dba • Its feedback is crucial for continuous improvement
ck • Help in iterating &enhancing application
mec
hani
sm

• Failure detected in production pipeline may trigger rollback to previous stable

Roll version.
bac
k

Rep • Throughout pipeline various metrics &reports are generated

orti • It provide insights into pipeline efficiency, code quality ,test
ng coverage ,deployment frequency &other relevant aspect
&M
etri
cs
• Tools like Jenkins, GitLab CI/CD, Travis CI,CircleCI and many other help in
CI/ automating &orchestrating the stage, enabling creation of robust & efficient
CD CD pipeline
Tool
s
 Transition from continuous delivery to
continuous deployment
Continuous Delivery(CD) Continuous Deployment(CD)
• It involves automated process to • Extends continuous delivery by
deliver code change to staging or automatically deploying code
production environment but change to production without
requires manual approval before manual intervention
deployment
 Transition from continuous delivery to continuous
deployment

Evaluate Gradual rollout

Monitoring &feedback Security & Documentatio
current loops compliance n & training
processe
s

Feedback &
Staging Team iteration
Automate tests Incremental
environment collaboration
&quality assurance deployment
stability & culture
Key enhancements &considerations
Risk mitigation

Continuous deployment increase risk of introducing bugs

Robust testing &monitoring are crucial to mitigate risk

Cultural shift
Moving from manual approval to automated deployment
require cultural shift within team.
It involves building

Technical challenges
Legacy systems, complex architecture during the
transition
Address systematically possibly by breaking down
transition into smaller manageable steps
 Feature Flags
• Feature flags, also known as feature toggles are a
development technique used to enable or disable certain
features or functionalities in software applications at
runtime.

• These flags provide a way to control the release of new

features or changes, allowing developers to decouple
feature rollout from code deployment.
 Benefits of feature flags
Benefits of Using Feature Flags
[Link] Rollouts: Enable gradual releases of new features to specific user
segments or environments, reducing the risk of unexpected issues affecting all users.
2.A/B Testing and Experimentation: Conduct controlled experiments by enabling
features for specific user groups to gather data and assess user response before a full-
scale release.
[Link] Code Conflicts: Allow developers to merge code changes into the main
branch without activating the new features, reducing merge conflicts and enabling
continuous integration.
[Link] Management and Rollbacks: Quickly deactivate problematic features or
changes in real-time without the need for a code rollback, minimizing downtime and
user impact.
 Benefits of feature flags
[Link] of Deployment and Release: Decouple deployment from feature release, enabling
continuous deployment while controlling feature visibility.
[Link]-Centric Development: Tailor features to specific user groups, allowing for
personalized experiences and targeted feature adoption.
[Link] Collaboration: Facilitate collaboration between development, QA, and product teams
by enabling them to coordinate feature releases more effectively.
[Link]-Proofing: Prepare the codebase for future releases by incorporating features that might
not be ready for immediate deployment but are in active development.
[Link] and Compliance: Implement flags for security or compliance-related features, enabling
quick activation or deactivation in case of security vulnerabilities or regulatory changes.
 Implement Continuous Deployment with
Feature Flag
Establish feature flag Implement feature flag in Gradual rollouts
management system the development process &controlled activation
Flag integration in Code
Base: Feature Activation Control:
Integrate feature flags Use the feature flag
Choose a Feature Flag directly into application management system to
Tool: code to control the control the activation of
Select reliable feature flag behavior of specific specific features in different
management tool or features. environments.
Decouple Code
develop an in-house solution Deployment &Feature
to create ,manage, and Activation:
control feature flags within Ensure that deploying code Gradual Rollout:
your application. changes doesn’t Start by activating feature
automatically activate, new for small subset of users.
features,instead,feature
flags will control feature
visibility.
Implement Continuous Deployment with
Feature Flag
A/B Testing & Monitoring &Rollback Collaboration &
Experimentation Plans Communication
Monitoring & Team Collaboration:
Segmentation & Foster collaboration
Targeting: Utilize observability:
Implement monitoring between development,
feature flag to target QA &operations teams to
specific user groups for tools to track the
performance &health of effectively manage
A/B testing. feature flags
features in production.
&deployments.
Gathering Metrics: Rollback Strategies: Clear Communication:
Collect &analyze Establish clear rollback Ensure transparent
metrics to evaluate the strategies & ensure the communication regarding
performance &user problematic features can feature flag statues,
response to the new be quickly deactivated if activation &deactivation
features. issues arise. processes across teams.
 Implement Continuous Deployment with
Feature Flag
Security &Compliance Considerations Continuous Improvement

Security flags:
Feedback &
Utilize feature flags to enable or
Iteration:
disable security related features in
Gather feedback from users &internal
real time to address vulnerabilities.
teams to iterate on features &improve
the deployment process continuously.

Compliance Flags:
Implement flags for compliance Optimization:
related features to manage Continuously optimize the use of
regulatory changes effectively. feature flags based on performance
data and user feedback.
 Best Practices using Feature Flags
1. Granular Feature Flags:
Use fine grained feature flag to control individual feature or component
It provide more flexibility & control over the release process

[Link] Behavior:
Set a default state for each feature flag, defining whether feature is enabled or disabled by default.
It allows you to control exposure of new features to users.

[Link] Plans:
Plan feature rollouts carefully.
Gradually enable feature for small percentage of users.

[Link] configuration:
Make feature flag configurable at runtime to enable quick change without requiring redeployment
 Best Practices using Feature Flags
[Link] & Metrics:
Instrument feature flag with monitoring & logging to track their usage, performance impact & user
behavior.
The information is valuable to make informed decisions about feature toggles.

[Link] unused flags:

Regularly review and remove feature flag no longer need.
It help keep codebase clean & reduce complexity.

[Link] with Flags:

Use feature flags during development &testing to isolate new feature.
It allow for thorough testing without affecting entire user base.

[Link]:
Document the purpose &usage of feature flags, especially when they are introduced.
It help team understand the purpose & context of each toggle.
 Why Prioritize Security in CI/CD?
The efficiency of DevOps methodologies heavily relies on CI/CD pipeline.
Following code development & its repository commit the pipeline automates
code building, testing &readies it for production deployment.
Security of code entering production is directly tied to CI/CD pipeline
security.
If test cases are inaccurate,incomplete,or tampered with it could lead to
unnoticed vulnerabilities.
Additionally, the CI/CD process might inadvertently introduce malicious or
vulnerable code through third party dependencies.
CI/CD security measures are pivotal in addressing these & other potential
security threats encountered across the pipeline across pipeline stages.
 Securing the CI/CD Pipeline
1. Source Composition Analysis (SCA)
Purpose: Identifies third-party dependencies in an application and assesses their potential vulnerabilities.
Benefits: Helps mitigate risks associated with vulnerable third-party components and safeguards against attacks on
the software supply chain.
[Link] Code Scanning (SAST)
Purpose: Analyzes the source code of an application to detect and fix potential vulnerabilities.
Benefits: Allows DevOps teams to address security issues early in the SDLC, reducing the cost of remediation and
preventing vulnerabilities from reaching production.
[Link] Testing
Purpose: Conducts Dynamic Application Security Testing (DAST) on operational applications to identify
vulnerabilities that might not be detected by SAST.
Benefits: Catches security issues that may emerge during the later stages of the SDLC, ensuring a comprehensive
security assessment
[Link] Security
Purpose: Implements Runtime Application Self-Protection (RASP) to continuously monitor and protect
applications in production environments.
Benefits: Mitigates risks from vulnerabilities that may slip through testing or emerge post-production, providing an
additional layer of security.
 Why use CI/CD governance model?
Ensures efficiency: A well-defined governance model helps streamline the CI/CD
process, reducing manual effort and speeding up delivery times.
Enhances security: It establishes controls and checks to mitigate security risks and
ensure compliance with security standards.
Guarantees compliance: A governance model helps organizations adhere to industry
regulations and internal policies.

Key strategies for establishing a CI/CD governance framework:

Tailored to individual needs: The governance model should be customized to fit the
specific needs and processes of an organization.
Aligned with industry standards: It should adhere to relevant industry standards and
best practices to ensure quality and consistency.
End-to-end coverage: The framework should cover all aspects of the CI/CD pipeline,
from code development to deployment.
 CI/CD Governance Model
1. Mapping CI/CD Processes
•Understand the pipeline: Begin by mapping out your entire CI/CD pipeline to identify potential vulnerabilities and
areas that could be improved. This includes pinpointing handoff points between different teams or stages in the
process.
•Align with standards: Align the mapped pipeline with key controls, regulatory requirements, and industry standards
to ensure compliance and security.
2. Declarative Pipeline as Code
•Define the pipeline: Use declarative code (like YAML configuration files) to define your CI/CD pipeline. This
approach makes it easier to manage and version control your pipeline.
•Incorporate controls and gates: Include controls, gates, and governance practices within the declarative pipeline to
ensure consistency and enforce security measures across different environments.
3. Clarifying Roles and Responsibilities
•Define roles: Clearly define the roles and responsibilities of each team involved in the CI/CD process. This ensures
accountability and prevents confusion.
•Grant appropriate access: Grant appropriate access levels to different team members based on their roles and
responsibilities.
•Implement governance processes: Implement necessary governance processes to ensure compliance and security
throughout the CI/CD pipeline.
 CI/CD Governance Model
4. Access Control and Audits
•Centralized access management: Implement a centralized system to manage and control access permissions for
different team members.
•Regular audits: Conduct regular audits to monitor access permissions and identify any unauthorized access or
potential security risks.
5. Balancing Flexibility and Control
•Offer flexibility: Provide teams with the flexibility to choose their tools and processes while ensuring they align
with the overall governance framework.
•Establish guardrails: Set clear guardrails and guidelines to prevent unauthorized tools and processes from being
used.
•Regular auditing: Conduct regular audits to assess the level of flexibility and rigidity needed and make adjustments
accordingly.
[Link] Automated Testing
•Prioritize early testing: Prioritize automated tests early in the software development lifecycle to catch issues early
on.
•Scale complexity: As the development progresses, increase the complexity of automated tests to ensure thorough
testing and traceability of failures.
 CI/CD Governance Model
7. Standardized Code Review and Testing
•Enforce standardized practices: Implement standardized code review practices, such as the "four-eyes"
approach, to ensure multiple checks before code deployment.
•Improve code quality: Standardized code reviews help improve code quality and reduce defects.
[Link] Deployment Environment Rules
•Ensure consistency: Treat environments as code parameters to ensure consistency and traceability across
different deployment stages.
•Prevent errors: This approach helps prevent errors and inconsistencies that may arise during deployment.
[Link] Security Measures
•Employ least privilege access: Grant only the necessary permissions to individuals, limiting the potential for
unauthorized access.
•Encrypt sensitive data: Protect sensitive data by encrypting it both at rest and in transit.
•Automated security testing: Use automated tools to test dependencies for vulnerabilities and ensure security.
•Regular security audits: Conduct regular security audits to identify and address potential security risks.
 CI/CD Governance Model
10. Performance Monitoring
•Monitor pipeline performance: Utilize dedicated monitoring tools to track key performance indicators (KPIs) like
deployment frequency, lead time for changes, and failure rates.
•Identify bottlenecks: Analyze performance data to identify bottlenecks and areas for improvement in the CI/CD pipeline.

11. Regular Review and Updates

•Establish a review workflow: Create a process for regular reviews of the CI/CD governance model by a cross-functional
team.
•Update policies and procedures: Update policies and procedures based on feedback and changing organizational goals.
•Ensure alignment: Ensure that the CI/CD governance model aligns with the overall organizational strategy and
objectives.
•Communicate and measure: Communicate changes to the team and measure the impact of the updates to ensure
effectiveness.
 CI/CD Best Practices
1. Make it a team effort:
•CI/CD is not just a process; it's a culture.
•Get everyone on the team involved, from developers to testers to operations.
•This helps ensure everyone is on the same page and the CI/CD pipeline runs smoothly.
2. Commit early and commit often:
•This helps ensure your code is always in a deployable state.
•You can quickly identify and fix any problems.
3. Keep the builds green:
•Every build should pass all tests.
•If a build fails, fix it immediately before further development.
4. Build only once:
•Build your code only once per change.
•This improves the performance of your CI/CD pipeline.
5. Streamline your tests:
•Run only the necessary tests for each build.
•This reduces the time it takes to build and test your code.
6. Clean your environments:
•Clean your environment before starting a new build.
•This ensures a known state for your environment.
 CI/CD Best Practices
7. Make it the only way to deploy to production:
•Use CI/CD as the exclusive way to deploy to production.
•This ensures all changes are properly tested and deployed.
8. Monitor and measure your pipeline:
•Monitor and measure your pipeline's performance over time.
•This helps you identify and fix problems.
 Blue Green Deployment
Simultaneous Deployment:
Both the new and old versions of the application are deployed at the same time.
Active Version:
Only the older version of the application is active and receives production traffic.
 Deployment strategies to achieve zero
downtime in production
Advantages:
•Virtually Zero Downtime: The load balancer ensures a smooth switch to the new codebase,
minimizing disruptions for users.
•Testing in a Production-Like Environment: Allows for testing the new version in a live
setting, though caution is needed to avoid erroneous data generation.
•Easy Rollback: If issues arise with the new version, reverting to the old environment is
straightforward.

Disadvantages:
•Loss of Current Transactions and Sessions: Switching between environments may result in
the loss of ongoing user interactions.
•Potential Data Compatibility Issues: Changes in the database schema for the new version
might not be compatible with the old version, leading to production outages.
 Canary Deployment
Key Points:
•Gradual Release: A new feature is initially released to a small percentage of users.
•Expansion Based on Feedback: The release is gradually expanded based on user responses and
feedback.
•Purpose: This method helps identify and mitigate potential issues before a full-scale release.

How it works:
[Link]: The new feature is deployed to a small subset of production servers.
[Link] Routing: A small percentage of user traffic is directed to the servers with the new
feature.
[Link]: Close monitoring of user behavior, error logs, and performance metrics is
conducted.
[Link]: If the new feature performs well, the percentage of traffic directed to it is gradually
increased.
[Link] Rollout: Once the new feature is deemed stable and reliable, it's rolled out to all users.
 Deployment strategies to achieve zero
downtime in production
Advantages Disadvantages

Zero downtime during deployment. Possibility N-1 data compatibility issues.

Gathering user feedback before a Reliance on feature toggles, which can

widespread release. complicate the deployment process
 Rolling Deployment
Key Points:
•Gradual Replacement: Running instances of the application are gradually replaced with newer ones.
•Maintained Instances: A specific number of instances (N+1) are maintained at any given time.
•Phased Out: Older instances are phased out as new ones are introduced.

How it works:
[Link]: New instances of the application are deployed alongside the existing ones.
[Link] Routing: A portion of the traffic is gradually shifted to the new instances.
[Link]: Close monitoring of the new instances is conducted to ensure they are performing as expected.
[Link] Down: Once the new instances are stable, the old instances are decommissioned.

Advantages:
•Reduced Downtime: Minimal disruption to users as the transition is gradual.
•Easier Rollback: If issues arise with the new instances, traffic can be quickly switched back to the old ones.
•Simplified Management: The deployment process is relatively straightforward.

Disadvantages:
•Potential for Instability: Issues with the new instances can affect the overall system.
•Increased Complexity: Requires careful configuration and monitoring.
•Longer Deployment Time: The phased rollout can take longer than a traditional deployment.
 Preparation For Rollback
[Link] Control and Tagging
•Purpose: Helps track changes, identify specific versions, and enables easy rollback if needed.
•Implementation: Use version control systems like Git to tag releases with meaningful labels. This
allows for easy identification and rollback to specific versions.
[Link] Testing and Monitoring
•Purpose: Ensures the quality and stability of the deployed application.
•Implementation:
• Implement comprehensive automated testing procedures before deployment.
• Set up monitoring tools to closely observe the application's performance, system metrics, and
user interactions in real-time during and after deployment.
[Link] Systems and Data
•Purpose: Provides a safety net in case of deployment failures or data corruption.
•Implementation: Ensure you have backups of critical systems, databases, and configurations before
initiating a deployment. This facilitates restoring the system to its previous state if necessary.
 Preparation For Rollback
[Link] Plan
•Purpose: Outlines the steps and criteria for initiating a rollback.
•Implementation: Develop a detailed rollback plan outlining the steps and criteria
for initiating a rollback. Clearly define the triggers that indicate when a rollback is
necessary, such as increased error rates, system crashes, or user complaints.
[Link] Plan
•Purpose: Ensures smooth communication and coordination during deployment and
rollback scenarios.
•Implementation: Establish a communication plan within the team and
stakeholders, outlining the roles and responsibilities during a rollback scenario.
Ensure everyone knows the rollback procedure and the communication channels to
use in case of emergencies.
 Execution For Rollback
Identify Issues:
•Closely monitor: After deployment, keep a close watch on system performance, user feedback, and
error logs.
•Identify root cause: If problems arise, analyze these sources to pinpoint the root cause of the issue.

Assess Impact:
•Evaluate impact: Assess the severity of the issue and its impact on users, system stability, and
overall functionality.
•Determine rollback necessity: Based on predefined criteria, decide if a rollback is necessary to
mitigate the issue.

Initiate Rollback:
•Follow rollback plan: If a rollback is necessary, execute the pre-defined rollback plan meticulously.
•Revert to previous version: This might involve reverting to the previous version of the software,
restoring backed-up data, or reconfiguring systems to their earlier state.
 Implement Rolling Updates in CI/CD
Pipelines

1. Automated Deployment:
•Consistency and Reduced Human Error: Automated deployment tools ensure consistency and reduce the risk of
human error during the deployment process.
•Rolling Updates: Tools like Jenkins, GitLab CI/CD, Travis CI, and others can be configured to support rolling
updates, which minimize downtime and risk.

2. Incremental Rollouts:
•Gradual Deployment: This strategy involves gradually rolling out updates to a small subset of servers before
applying changes to the entire infrastructure.
•Early Issue Detection: This allows for early identification and resolution of issues, minimizing their impact on users.
 Implement Rolling Updates in CI/CD Pipelines
3. Health Checks and Monitoring:
•Instance Monitoring: Implement health checks to monitor the state of each instance during and after the update. If a
new version is causing issues, the deployment can be automatically rolled back.
•Performance Monitoring: Integrate monitoring tools to track key performance metrics and detect anomalies. This
enables proactive identification and resolution of potential problems.

4. Rollback Strategy:
•Well-Defined Rollback Plan: Have a well-defined rollback strategy in case issues arise. This might involve re-
deploying the previous version or rolling back specific changes that caused problems.

[Link] Deployments:
•Reduced Rollout Time: If possible, deploy multiple versions of your application in parallel to reduce the time it
takes to roll out updates. This can be achieved by using techniques like blue-green deployments or canary releases.

6. Feature Toggles:
•Controlled Feature Release: Implement feature toggles to enable or disable specific features at runtime. This allows
you to decouple deployment and release, giving you more control over when features are exposed to users.
 Implement Rolling Updates in CI/CD
Pipelines
[Link] Migrations:
•Separate Schema Changes: Consider database schema changes
separately from application updates. This helps isolate potential issues
and makes the rollback process easier.
•Backward Compatibility: Implement backward-compatible database
changes to avoid downtime during updates. This ensures that the old
application version can still function with the new database schema.

6. Immutable Infrastructure:
•Instance Replacement: Consider using immutable infrastructure
principles, where instances are replaced rather than updated in-place. This
ensures consistency and reduces the chance of configuration drift.
THANK YOU