Implmented node key model for image encryption

Signed-off-by: Brandon Lum <[email protected]>

docs/encryption.md

@@ -0,0 +1,71 @@
+# Configure Image Encryption
+This document describes the method to configure image encryption for `containerd` for use with the `cri` plugin.
+
+
+## Encrypted Container Images
+
+Encrypted container images are OCI images which contain encrypted blobs. An example of how these encrypted images can be created through the use of [containerd/imgcrypt project](https://github.com/containerd/imgcrypt). In order for the containerd runtime to be able to decrypt these images, the `cri` has to pass the correct options in its calls to containerd. This includes material such as keys and encryption metadata required by the runtime.
+
+## Key Models
+
+
+Encryption ties trust to an entity based on the model in which a key is associated with it. We call this the key model. There are two currently supported key models in which encrypted containers can be used. These is based on two main usecases. 
+
+1. "node" Key Model - In this model encryption is tied to workers. The usecase here revolves around the idea that an image should be only decryptable only on trusted host. Although the granularity of access is more relaxed (per node), it is beneficial because there various node based technologies which help bootstrap trust in worker nodes and perform secure key distribution (i.e. TPM, host attestation, secure/measured boot). In this scenario, runtimes are capable of fetching the necessary decryption keys. An example of this is using the [`--decryption-keys-path` flag in imgcrypt](https://github.com/containerd/imgcrypt).
+
+2. "multitenant" Key Model - In this model, the trust of encryption is tied to the cluster or users within a cluster. This allows multi-tenancy of users, and is useful in the case where multiple users of kubernetes each want to bring their own encrypted images. This is based on the [KEP that introduces `ImageDecryptSecrets`](https://github.com/kubernetes/enhancements/pull/1066/).
+
+
+## Configuring image encryption "node" key model
+
+The default configuration does not handle encrypted image. 
+
+In order to set up image encryption, create/modify `/etc/containerd/config.toml` as follows:
+```toml
+[plugins.cri.image_encryption]
+  key_model = "node"
+
+[stream_processors]
+  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
+    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
+    returns = "application/vnd.oci.image.layer.v1.tar+gzip"
+    path = "/usr/local/bin/ctd-decoder"
+    args = ["--decryption-keys-path", "/keys"]
+  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
+    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
+    returns = "application/vnd.oci.image.layer.v1.tar"
+    path = "/usr/local/bin/ctd-decoder"
+    args = ["--decryption-keys-path", "/keys"]
+```
+
+This will enable support of `cri` for handling encrypted images. The configuration here sets the key 
+model to that of "node". In addition, the decryption `stream_processors` are configured as specified in 
+[containerd/imgcrypt project](https://github.com/containerd/imgcrypt), and have an additional field `--decryption-keys-path` 
+configured to specify where decryption keys are located locally in the node.
+
+After modify this config, you need restart the `containerd` service.
+
+## Configuring image encryption "multitenant" key model
+
+```toml
+[plugins.cri.image_encryption]
+  key_model = "multitenant"
+
+[stream_processors]
+  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
+    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
+    returns = "application/vnd.oci.image.layer.v1.tar+gzip"
+    path = "/usr/local/bin/ctd-decoder"
+  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
+    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
+    returns = "application/vnd.oci.image.layer.v1.tar"
+    path = "/usr/local/bin/ctd-decoder"
+```
+
+
+To use the multitenant key model, the field should be set to `key_mode = "multitenant"`.
+We note that "multitenant" key modle requires a kubernetes version that supports the `ImageDecryptSecrets` 
+feature. The `stream_processors` need to be configured to handle decryption as specified in [containerd/imgcrypt project](https://github.com/containerd/imgcrypt). Since keys are passed through
+kubernetes, there is no need to specify a way in which the decoder needs to locally obtain keys.
+
+After modify this config, you need restart the `containerd` service.

pkg/config/config.go

@@ -153,6 +153,11 @@ type RegistryConfig struct {
 	TLS *TLSConfig `toml:"tls" json:"tls"`
 }
 
+type EncryptedImagesConfig struct {
+	// KeyModel specifies the model of where keys should reside
+	KeyModel string `toml:"key_model" json:"keyModel"`
+}
+
 // PluginConfig contains toml config related to CRI plugin,
 // it is a subset of Config.
 type PluginConfig struct {
@@ -162,6 +167,8 @@ type PluginConfig struct {
 	CniConfig `toml:"cni" json:"cni"`
 	// Registry contains config related to the registry
 	Registry Registry `toml:"registry" json:"registry"`
+	// EncryptedImagesConfig contains config related to handling of encrypted images
+	EncryptedImagesConfig `toml:"image_encryption" json:"imageEncryption"`
 	// DisableTCPService disables serving CRI on the TCP server.
 	DisableTCPService bool `toml:"disable_tcp_service" json:"disableTCPService"`
 	// StreamServerAddress is the ip address streaming server is listening on.
@@ -236,6 +243,12 @@ const (
 	RuntimeUntrusted = "untrusted"
 	// RuntimeDefault is the implicit runtime defined for ContainerdConfig.DefaultRuntime
 	RuntimeDefault = "default"
+	// EncryptionKeyModelMultitenant is the key model where keys are obtained from
+	// kubernetes ImageDecryptSecrets
+	EncryptionKeyModelMultitenant = "multitenant"
+	// EncryptionKeyModelNode is the key model where key for encrypted images reside
+	// on the worker nodes
+	EncryptionKeyModelNode = "node"
 )
 
 // ValidatePluginConfig validates the given plugin configuration.

pkg/server/container_create.go

@@ -165,6 +165,7 @@ func (c *criService) CreateContainer(ctx context.Context, r *runtime.CreateConta
 		// rootfs readonly (requested by spec.Root.Readonly).
 		customopts.WithNewSnapshot(id, containerdImage),
 	}
+
 	if len(volumeMounts) > 0 {
 		mountMap := make(map[string]string)
 		for _, v := range volumeMounts {

pkg/server/image_pull.go

@@ -32,6 +32,8 @@ import (
 	containerdimages "github.com/containerd/containerd/images"
 	"github.com/containerd/containerd/log"
 	"github.com/containerd/containerd/remotes/docker"
+	"github.com/containerd/imgcrypt"
+	"github.com/containerd/imgcrypt/images/encryption"
 	distribution "github.com/docker/distribution/reference"
 	imagespec "github.com/opencontainers/image-spec/specs-go/v1"
 	"github.com/pkg/errors"
@@ -106,15 +108,24 @@ func (c *criService) PullImage(ctx context.Context, r *runtime.PullImageRequest)
 			return nil, nil
 		}
 	)
-	image, err := c.client.Pull(ctx, ref,
+
+	pullOpts := []containerd.RemoteOpt{
 		containerd.WithSchema1Conversion,
 		containerd.WithResolver(resolver),
 		containerd.WithPullSnapshotter(c.config.ContainerdConfig.Snapshotter),
 		containerd.WithPullUnpack,
 		containerd.WithPullLabel(imageLabelKey, imageLabelValue),
 		containerd.WithMaxConcurrentDownloads(c.config.MaxConcurrentDownloads),
 		containerd.WithImageHandler(imageHandler),
-	)
+	}
+
+	if c.config.EncryptedImagesConfig.KeyModel == criconfig.EncryptionKeyModelNode {
+		ltdd := imgcrypt.Payload{}
+		decUnpackOpt := encryption.WithUnpackConfigApplyOpts(encryption.WithDecryptedUnpack(&ltdd))
+		pullOpts = append(pullOpts, encryption.WithUnpackOpts([]containerd.UnpackOpt{decUnpackOpt}))
+	}
+
+	image, err := c.client.Pull(ctx, ref, pullOpts...)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to pull and unpack image %q", ref)
 	}

vendor.conf

@@ -87,3 +87,9 @@ github.com/davecgh/go-spew 8991bc29aa16c548c550c7ff78260e27b9ab7c73 # v1.1.1
 github.com/containernetworking/plugins 9f96827c7cabb03f21d86326000c00f61e181f6a # v0.7.6
 github.com/containernetworking/cni 4cfb7b568922a3c79a23e438dc52fe537fc9687e # v0.7.1
 github.com/containerd/go-cni 0d360c50b10b350b6bb23863fd4dfb1c232b01c9
+
+# image decrypt depedencies
+github.com/containerd/imgcrypt 60475d2a2a95344ebcef9a456a4c9a1c7fcf4169
+github.com/containers/ocicrypt 142388cb70de0fe8c7edd921df79e477ab8b3051
+gopkg.in/square/go-jose.v2 v2.3.1 https://github.com/square/go-jose.git
+github.com/fullsailor/pkcs7 8306686428a5fe132eac8cb7c4848af725098bd4