Merge pull request #11 from timoguin/feat/terraform-state-modules

feat (partial): Inital spec and Terraform modules

Author: timoguin

README.md

@@ -20,6 +20,20 @@ are owed NOTHING! But we're going to follow semver once things are rolling.
 You can view the doc on [GitHub Pages]. The Markdown sources are all in the
 [docs] directory.
 
+## Building the Docs
+
+Requirements:
+
+- Poetry: This project uses poetry for managing Python requirements
+- make: Used for convenience in building/deploying the docs
+
+A static site is generated from the Markdown sources by using mkdocs. To build,
+a simple `make build` should do.
+
+To run a local server that will auto-generate and reload the static site when
+the Markdown sources or mkdocs configs are modified. Use `make serve` to run
+it, and access it on https://localhost:3000
+
 ## Contributing
 
 View the [Contributing Guide] if you're interested in. . . contributing. Check

docs/code/index.md

@@ -0,0 +1,3 @@
+# Infrastructure Code
+
+Document infrastructure code.

docs/index.md

@@ -7,5 +7,4 @@ Baseline essentials for an AWS account: documented and codified.
 <br /><br /><br />
 Used in a sentence:
 
-_I am trying to implement some new AWS accounts using
-the aws-baseline spec_.
+_I am trying to implement some new AWS accounts using the aws-baseline spec_.

docs/spec/cloudtrail/index.md

@@ -0,0 +1,24 @@
+# CloudTrail
+
+CloudTrail is an AWS service that records API calls and other events that occur within
+an account. Accounts should have a mechanism in place to quickly and easy analyze the
+large variety of API calls and events that flow through the system.
+
+## Recommended
+
+- Create a single trail for global events (IAM) in us-east-1
+- Create a trail in each region for read events
+- Create a trail in each region for write events
+- Create a trail in each region for S3 Object events
+- Create a trail in each region for Lambda events
+
+## Service Events
+
+https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html
+
+## AWS Organizations SCPs
+
+If your account is part of an AWS Organization, Service Control Policies can be used to
+ensure use of CloudTrail, as well as ensure critical resources cannot be changed.
+
+TODO: Add example SCP for CloudTrail

docs/spec/config/index.md

@@ -0,0 +1,35 @@
+# Config
+
+Config is an AWS service that tracks the configuration of resources. Any modifications
+to resources supported by Config will be captured and presented on a timeline, along
+with a diff of config changes and details about what principal modified the resource.
+
+Config Rules allow tracking compliance requirements. There is a list of built-in rules
+that can be enabled, and custom rules can be created via Lambda functions. 
+
+Config delivers logs to S3 and sends a notification to SNS upon delivery. Rules can be
+configured to send notifications, perform remediation, or perform other customizable
+actions.
+
+## Recommended
+
+-
+
+### Built-in Rules
+
+TODO: Build list of recommended Config Rules
+
+- 
+
+### Custom Rules
+
+TODO: Add recommendations for how to use custom rules, including any that should be
+enabled by default.
+
+## AWS Organizations SCPs
+
+If your account is part of an AWS Organization, Service Control Policies can be used to
+ensure the usage of Config and Config Rules, as well as ensure critical resources
+cannot be changed.
+
+TODO: Add example SCP for Config

terraform/modules/terraform-state/dynamodb-appautoscaling.tf docs/spec/iam/access-analyzer.md

@@ -0,0 +1,38 @@
+# Logging
+
+All accounts should have multiple logging pipelines available that can be used with
+little thought. Accounts should all have the tooling necessary to retain and analyze
+logs for a set retention period. All logging buckets should be replicated to another
+account for longer-term retention and analysis.
+
+See the [s3](s3/index.md) spec for details on bucket configurations necessary to
+support log collection from various sources.
+
+## AWS Service Logs
+
+- All logs should be delivered to S3
+- Use SNS to receive log delivery events
+- Send events to SQS queues
+- Subscribe Lambdas to the queues (invoked in parallel batches by SQS)
+
+## Processing
+
+Methods of log processing and delivery will vary depending on the volume of data there
+is to process. Things like CloudTrail and ELB logs can become quite large with active
+accounts and services.
+
+- Lambdas should process the messages, build a list of files to process, route the list
+  to other destinations (if necessary), and deliver the processed log data to another
+  S3 bucket
+- Use Athena's "Create Table As" query to convert logs to parquet and manage the
+  associated Glue catalogs.
+- Replicate processed logs between regions
+- Replicate processed logs to another account
+
+## Controlling Cost
+
+## Monitoring
+
+- Use S3 Inventory reports to periodically compare the processed S3 Events with the
+  actual objects in the bucket. This ensures that we still process any objects in the
+  event that S3 fails to properly fire and/or delivery an event.

terraform/modules/terraform-state/outputs.tf docs/spec/iam/permission-boundaries.md

@@ -0,0 +1,4 @@
+# Route53
+
+- Enable query logging to CloudWatch Logs
+- Steam logs to Kinesis Firehose for further processing

terraform/modules/terraform-state/variables-dynamodb.tf docs/spec/iam/policies.md

@@ -0,0 +1,83 @@
+# S3
+
+- Buckets for AWS service logs in each region (w/ S3 Object Lock)
+- Bucket for AWS S3 access logs in each region (w/ S3 Object Lock)
+- Replicate logging buckets to another account for longer retention and auditing
+
+S3 access logs are an outlier in that the object ownership and ACLs are special for the
+S3 service. Cross-account access requires role assumption and changes to object
+ownership.
+
+## Patterns
+
+### Single Bucket
+
+Bucket name:         <account_id>-<environment>-aws-service-logs-<region>
+Bucket name example: 123456789012-dev-aws-service-logs-us-east-1
+
+Log types and prefixes:
+
+- CloudTrail: cloudtrail/
+- Config: config/
+- ALB/ELB/NLB: lb/, alb/, elb/, nlb/ (pull from any and process the same)
+
+### Multiple Per-Service Buckets
+
+- CloudTrail: <account_id>-<environment>-cloudtrail-logs-<region>
+- Config: <account_id>-<environment>-config-logs-<region>
+- ELB: <account_id>-<environment>-elb-access-logs-<region>
+- S3: <account_id>-<environment>-s3-access-logs-<region>
+
+### Delivery Notications
+
+- CloudTrail: send to SNS
+- Config: send to SNS
+- ELB: S3 events to SNS
+- S3: S3 events to SNS
+
+#### Plumbing
+
+- Use S3 Events (optional) -> SNS -> SQS to retain a queue of events to process
+- SNS can also fan out to other destinations
+- SNS Event Forking to SAM applications (parallel processing and filtering)
+- SNS topics and SQS queues send failures to deadletter queues
+- Lambda to process the deadletter queues
+
+### Security
+
+- Each bucket only allows access from specific AWS services in the same region
+- All buckets encrypted with KMS, forced via the bucket policy
+- Public access blocked
+- Object Lock enabled (use compliance mode)
+- VPC Endpoint for S3
+ 
+#### Special Notes for S3 Access Logs
+
+Logging buckets for S3 access logs _cannot_ be encrypted. The bucket owner for the
+objects is also specific to the logging service. Objects cannot be accessed from
+another account without assuming a role into the account with the bucket.
+
+To allow these logs to be processed in the same manner as other service logs, use a
+Lambda (or some other form of automation) to copy the logs to another bucket and update
+the object ownerships.
+
+NOTE: Research S3 bucket replication in the context of S3 access logs. Could enable
+getting around the need for Lambda processing.
+
+#### Bucket Policies
+
+Specific AWS services vary on how to grant permissions to use the buckets:
+
+- CloudTrail uses a service principal (cloudtrail.amazonaws.com) to put logs and check bucket ACLs
+- Config uses an IAM role to do the same
+- ELB uses AWS account IDs (root) for the ELB service, different ones in each region
+
+### Durability
+
+- Replication (replicate buckets to another region and/or account for longer-term storage and analysis)
+- Versioning enabled (required for replication and object lock)
+
+### Cost Controls
+
+- Lifecycle policies to transition objects into cheaper storage (and/or eventually delete)
+- 

docs/spec/iam/roles.md

@@ -0,0 +1 @@
+# Tagging Standards

docs/spec.md docs/spec/index.md

@@ -24,8 +24,13 @@ remote_name: null
 nav:
    - Overview: overview.md
    - Goals: goals.md
-   - Specification: spec.md
-   - Code: code.md
+   - Spec:
+     - Index: spec/index.md
+     - CloudTrail: spec/cloudtrail/index.md 
+     - Config: spec/config/index.md 
+     - S3: spec/s3/index.md 
+     - Logging: spec/logging.md
+   - Code: code/index.md
    # - Advanced: advanced.md
    # - Costs: costs.md
    # - Issue Tracker: https://github.com/timoguin/aws-baseline/issues