WIP

_pages/security/incident-response-drills.md

@@ -6,7 +6,9 @@ sticky_sidenav: true
 
 ## Why do Incident Response drills?
 
-You don't want to be creating or testing recovery processes while things are on fire. Preparing and practicing ahead of time is a good idea.
+You don't want to be creating or testing recovery processes while things are on fire. 🔥
+
+Preparing and practicing ahead of time is a good idea.
 
 ## Preparing for the Drill
 
@@ -28,12 +30,12 @@ It is likely that your Agency or OCIO has existing policies around reporting for
 
 Scenarios worth practicing for a web app include:
 
-* [A deploy goes wrong](#a-deploy-goes-wrong)
-* [PII exposed](#pii-exposed)
-* [API Keys or passwords exposed](#api-keys-or-passwords-exposed)
-* [Oops, I deleted the database](#oops-i-deleted-the-database)
+* [Scenario: A Deploy Goes Wrong](#a-deploy-goes-wrong)
+* [Scenario: API Keys or Passwords Exposed](#api-keys-or-passwords-exposed)
+* [Scenario: Site Defacement](#site-defacement)
+* [Scenario: Oops, I Deleted the Database](#oops-i-deleted-the-database)
+* [Scenario: PII Exposed](#pii-exposed)
 * [S3 buckets are erased](#s3-buckets-are-erased)
-* [Site defacement](#site-defacement)
 * [Denial of Service](#denial-of-service)
 * [Service Downtime](#service-downtime)
 
@@ -43,50 +45,116 @@ These examples a web application hosted on [Cloud.gov](https://cloud.gov) that g
 
 Please adjust for your infrastructure.
 
-## A Deploy Goes Wrong
+## Scenario: A Deploy Goes Wrong
+
+It turns out, the new release doesn't deploy properly. It has successfully deployed in all the other environments. Let's re-deploy.
 
-It turns out, the new release doesn't deploy properly. It has successfully deployed in all the other environments.
+![Rerun job workflow in CircleCI]({{site.baseurl}}/assets/images/rerun-workflow-circleci-screenshot.png)
+<caption>
+  <i>Screenshot of how to re-run a workflow in a CI/CD tool (in this case, CircleCI)</i>
+</caption>
 
-### Mitigation:
+### Example mitigation steps:
 
 Re-deploy last successful release from  your CI/CD pipeline. (You are deploying from a CI/CD pipeline, right?)
 
-### Step-by-step template:
+1. Go to `<<Insert CI/CD URL>>` to view recent deploys.
+1. Rerun the deploy step for the last known-good deploy.
+1. If necessary, roll back the database to the correct version.
+
+### Example drill:
+
+Follow the mitigation steps above in a development environment.
+
+## Scenario: API Keys or Passwords Exposed
+
+An API Key for an AWS service was accidentally committed to our public code repository! (Use tools like [caulking](https://github.com/cloud-gov/caulking) to prevent issues like this from happening in the first place.)
+
+### Example mitigation steps:
+
+1. Contact `<<Insert email of POC laid out in Agency policies>>` and inform them of a breach.
+1. Write down which keys and services were exposed.
+1. Rotate all exposed keys.
+1. Remove any exposed keys from the commit history.
+
+### Example drill steps:
+
+1. Acknolwedge that the first step would be to inform points of contact; establish that everyone knows who to inform in the event of an incident.
+1. To simulate the real thing, push up a file to GitHub or whichever code repository use with a fake service key. (No using real keys for drills, please.)
+1. Practice rotating the keys for that service in a development context.
+1. Practice scrubbing the fake key from the commit history.
+
+## Scenario: Site Defacement
+
+The website has been hacked due to a compromised key! Now instead of our link to submit a report, we have a cute image of a cat and a spam link to follow cute cats on instagram.
+
+![Screenshot of Engineering Practices Guide homepage with cute cat photo in the middle of it]({{site.baseurl}}/assets/images/screenshot-fake-epg-hacked.png)
+<caption>
+  <i>Oh no! Who added this cute cat photo to our website?!?</i>
+</caption>
+
+### Example mitigation steps:
+
+TK
 
-1. Go to `<<Insert CI/CD URL>>` to view recent deploys
-1. Rerun the deploy step for the last known-good deploy
-1. If necessary, roll back the database to the correct version
+### Example drill steps:
 
-## PII Exposed
+TK
+
+## Scenario: Oops, I Deleted the Database
+
+The database needs to be restored from a backup.
+
+### Example mitigation steps:
+
+1. If you're using Cloud.gov, follow [Cloud.gov database backup procedures](https://cloud.gov/docs/services/relational-database/#backups).
+
+### Example drill steps:
+
+Assuming you have a staging database using a dedicated Cloud.gov database plan:
+
+1. Delete some data from your staging database. (No deleting data from a production database, please.)
+2. Reach out to Cloud.gov using the [the non-emergency email address provided in thir docs](https://cloud.gov/docs/services/relational-database/#backups); request a backup.
+3. Practice restoring the staging database to the point in time before you deleted the data.
+
+## Scenario: PII Exposed
 
 It's discovered that PII is being leaked to unauthorized users through the site.
 
-### Mitigation
+### Example mitigation steps:
 
-1. Contact `<<Insert email of POC laid out in Agency policies>>` and inform them of a breach
-1. Stop the exposure
-  1. Assess the severity and impact of the potential leak
-  1. Decide if the site needs to be set into a maintenance mode to stop further exposure. If yes, then bring up the maintenance page
-  1. If you are able to isolate the section of the site where the issue is occurring and remove/hide the page
-1. If this is in code version control history, be sure to scrub references to the PII from all versions.
-1. Identify root cause of the issue and deploy a hotfix
+1. Contact `<<Insert email of POC laid out in Agency policies>>` and inform them of a breach.
+1. Stop the exposure.
+  * Assess the severity and impact of the potential leak.
+  * Decide if the site needs to be set into a maintenance mode to stop further exposure. If yes, then bring up the maintenance page.
+  * If you are able to isolate the section of the site where the issue is occurring and remove/hide the page.
+1. Identify root cause of the issue and deploy a hotfix.
 1. Take necessary corrective action as directed by your agency security team. If there are corrective actions that the PO is able to handle in terms of contacting the affected users, do so.
 
-## API Keys or Passwords Exposed
+### Example drill steps:
 
-An API Key for an AWS service was accidentally committed to our public code repository!
+1. Acknolwedge that the first step would be to inform points of contact; establish that everyone knows who to inform in the event of an incident.
+1. In a development environment, practice putting the site into a maintenance mode or removing/hiding a page on the site, whichever would be most relevant to your project.
+1. Review any relevant corrective action / affected user notification procedures.
 
-### Mitigation
+## S3 buckets are erased
 
-1. Contact `<<Insert email of POC laid out in Agency policies>>` and inform them of a breach
-1. Rotate keys
+### Example mitigation steps:
 
-## Oops, I deleted the database
+TK
 
-## S3 buckets are erased
+### Example drill steps:
+
+TK
 
-## Site defacement
 
 ## Denial of Service
 
-## Service Downtime
+### Example mitigation steps:
+
+TK
+
+### Example drill steps:
+
+TK
+