Initial basic bootstrap-checkpoint support #50984

timothysc · 2017-08-20T19:28:01Z

What this PR does / why we need it:
Adds initial support for Pod checkpointing to allow for controlled recovery of the control plane during self host failure conditions.

fixes #49236
xref kubernetes/enhancements#378

Special notes for your reviewer:

Proposal is here: https://docs.google.com/document/d/1hhrCa_nv0Sg4O_zJYOnelE8a5ClieyewEsQM6c7-5-o/edit?ts=5988fba8#

Controlled tests work, but I have not tested the self hosted api-server recovery, that requires validation and logs. /cc @luxas
In adding hooks for checkpoint manager much of the tests around basicpodmanager appears to be stub'd. This has become an anti-pattern in the code and should be avoided.
I need a node-e2e to ensure consistency of behavior.

Release note:

Add basic bootstrap checkpointing support to the kubelet for control plane recovery

/cc @kubernetes/sig-cluster-lifecycle-misc @kubernetes/sig-node-pr-reviews

timothysc · 2017-08-20T19:29:00Z

I will rebase post review and conversation. I will also post the design proposal writeup from https://docs.google.com/document/d/1hhrCa_nv0Sg4O_zJYOnelE8a5ClieyewEsQM6c7-5-o/edit?ts=5988fba8# today.

luxas

Did a quick initial pass.
As we said earlier, the naming of this feature will be crucial to not cause unneeded confusion.

I didn't see anything that'd remove the Static Pod once the pivot is done; WIP, right?
Also I didn't review the overall flow in detail, but am starting the conversation with these basic comments.

luxas · 2017-08-20T19:34:22Z

cmd/kubelet/app/options/options.go

@@ -232,6 +232,8 @@ func AddKubeletConfigFlags(fs *pflag.FlagSet, c *componentconfig.KubeletConfigur
 	fs.MarkDeprecated("experimental-fail-swap-on", "This flag is deprecated and will be removed in future releases. please use --fail-swap-on instead.")

 	fs.StringVar(&c.PodManifestPath, "pod-manifest-path", c.PodManifestPath, "Path to to the directory containing pod manifest files to run, or the path to a single pod manifest file. Files starting with dots will be ignored.")
+	fs.StringVar(&c.CheckpointPath, "checkpoint-path", c.CheckpointPath, "<Warning: Alpha feature> Path to to the directory where the checkpoints are stored")


--experimental-BOOTSTRAPPING-checkpoint-path 😉?
We talked about scoping this down a lot and being clear about the functionality at this time.

@yujuhong ?

I'm ok with @smarterclayton's suggestion for the flag experimental-pod-preservation-path

I'm changing to --bootstrap-checkpoint-path per: ' https://github.com/kubernetes/community/pull/1241/files '

Putting 'experimental' or 'alpha' into names has been known to cause issues during promotion and we've moved to updating descriptions and documentation. The name change should suffice.

luxas · 2017-08-20T19:36:11Z

pkg/api/annotation_key_constants.go

@@ -72,6 +72,10 @@ const (
 	// This annotation can be attached to node.
 	ObjectTTLAnnotationKey string = "node.alpha.kubernetes.io/ttl"

+	// CheckpointAnnotationKey represents a Resource (Pod) that should be checkpointed by
+	// the kubelet prior to running
+	CheckpointAnnotationKey string = "node.kubernetes.io/checkpoint"


alpha.node.kubernetes.io/bootstrapping-checkpointing?
node.alpha.kubernetes.io/bootstrapping-checkpointing?

I guess that when this gets graduated we'll use an alpha/beta/whatever field using the grand new proposal we have for that, right?

Nope alpha has been removed from the naming conventions per api-machinery folks.

Cool! That's a great step forwards 👍

I'm keeping the annotation name the same for the purpose of potential re-use. The documentation update will call out the details and limitations.

luxas · 2017-08-20T19:36:45Z

pkg/generated/openapi/BUILD

@@ -9,6 +9,8 @@ openapi_library(
    name = "go_default_library",
    srcs = ["doc.go"],
    openapi_targets = [
+        "_output/dockerized/go/src/github.com/miekg/coredns/vendor/k8s.io/client-go/1.5/pkg/runtime",


luxas · 2017-08-20T19:37:02Z

pkg/apis/componentconfig/v1alpha1/types.go

@@ -271,6 +271,9 @@ type KubeletConfiguration struct {
 	// podManifestPath is the path to the directory containing pod manifests to
 	// run, or the path to a single manifest file
 	PodManifestPath string `json:"podManifestPath"`
+	// checkpointPath is the path to the directory where the kubelet will checkpoint
+	// annotated pod manifests and configmaps before they are run
+	CheckpointPath string `json:"checkpointPath"`


BootstrappingCheckpointPath?

luxas · 2017-08-20T19:37:15Z

pkg/kubelet/checkpoint/checkpoint.go

+)
+
+const (
+	delimiter = "_"


luxas · 2017-08-20T19:38:40Z

pkg/kubelet/checkpoint/checkpoint.go

+}
+
+// loadCheckpoint will load 'Resource_Name' Checkpoint yaml file.
+func (fcp *fileCheckPointManager) loadPod(file string) (*v1.Pod, error) {


use "k8s.io/kubernetes/pkg/volume/util".LoadPodFromFile(filename string) for this instead?

Ohh hey, that is handy, I'll update.

luxas · 2017-08-20T19:39:36Z

pkg/kubelet/checkpoint/checkpoint.go

+	return pod, nil
+}
+
+// checkAnnotations will validate the annotations are there.


nit: checkAnnotations will check whether the bootstrapping checkpointing annotation is set on the pod object

luxas · 2017-08-20T19:40:27Z

pkg/kubelet/checkpoint/checkpoint.go

+
+// NewCheckpointManager will create a NewCheckpointManager that points to the following path
+func NewCheckpointManager(path string) Manager {
+	return &fileCheckPointManager{path: path}


This makes something like var _ Manager = &fileCheckPointManager{} not needed, right?

Yeah, and it's consistent with the rest of the Kubelet.

luxas · 2017-08-20T19:44:00Z

pkg/apis/componentconfig/types.go

@@ -195,6 +195,9 @@ type KubeletConfiguration struct {
 	// podManifestPath is the path to the directory containing pod manifests to
 	// run, or the path to a single manifest file
 	PodManifestPath string
+	// checkpointPath is the path to the directory where the kubelet will checkpoint
+	// annotated pod manifests and configmaps before they are run


Not implemented yet, right?
Should we consider adding this to the comment once we actually add this feature?

I'll nix the configmaps bit there.

rphillips · 2017-08-21T15:11:08Z

hack/local-up-cluster.sh

@@ -646,6 +646,7 @@ function start_kubelet {
      sudo -E "${GO_OUT}/hyperkube" kubelet ${priv_arg}\
        --v=${LOG_LEVEL} \
        --vmodule="${LOG_SPEC}" \
+	--fail-swap-on=false \


nit. spacing

timothysc · 2017-08-23T13:55:32Z

@yujuhong - I'm pending on your feedback before I rebase and address other comments.

/cc @roberthbailey

timothysc · 2017-08-24T15:52:40Z

/cc @liggitt - Did you have opinions on the startup ordering. This is a PoC and comments are solicited.

liggitt · 2017-08-24T16:57:01Z

pkg/kubelet/kubelet.go

+			return nil, nil, err
+		}
+		if len(pods) > 0 {
+			c <- kubetypes.PodUpdate{Pods: pods, Op: kubetypes.SET, Source: kubetypes.ApiserverSource}


won't this make the kubelet kill any currently running containers not in the checkpointed list?

nope, it only seeds the cache which will need to run through the checks.

are you sure? this will add the ApiserverSource to the sourcesReady list before we've actually gotten a full list from the API server. That allows AllReady() to return true, which allows housekeeping to run HandlePodCleanups() prior to us knowing what we should be running from the API server. Won't that delete running containers?

@liggitt is right. Kubelet assumes the first update from a source includes the complete list of pods. Once all sources are ready, GC will kick in to delete containers that do not belong to a known pod.

Wiring this so that kubelet considers the source is not ready is required.

smarterclayton · 2017-08-24T16:58:03Z

Can we rename the flag and pr so it's clear that this isn't actually checkpointing (as discussed on the meeting)? I haven't seen a checkpointing design, I've seen something much more limited with no design consideration given to the long term problems.

liggitt · 2017-08-24T16:58:49Z

pkg/kubelet/checkpoint/checkpoint.go

+// Writes a checkpoint to a file on disk if annotation is present
+func (fcp *fileCheckPointManager) WritePod(pod *v1.Pod) error {
+	var err error
+	if fcp.checkAnnotations(pod) {


this means that removing the checkpoint annotation from a pod will never clean it up (here or in DeletePod)

Good point, I'll create a UID cache so updates that remove cause a checkpoint delete.

timothysc · 2017-08-24T18:09:38Z

Can we rename the flag and pr so it's clear that this isn't actually checkpointing (as discussed on the meeting)?

Gotta name?

I haven't seen a checkpointing design, I've seen something much more limited with no design consideration given to the long term problems.

Per the meeting, it was explicitly stated we would defer future use of such facilities would require their own proposals, but I'll distill the 2 docs down into concise language stating it explicitly and push today.

smarterclayton · 2017-08-24T21:59:56Z

Maybe experimental-pod-preservation-path for the flag and call it Pod Preservation or Kubelet Pod Memory or something that implies recall without the more precise term.

yujuhong

There are some tricky cases to consider for implementation. Will need to think about it more...

yujuhong · 2017-08-25T01:14:08Z

hack/local-up-cluster.sh

@@ -646,6 +646,7 @@ function start_kubelet {
      sudo -E "${GO_OUT}/hyperkube" kubelet ${priv_arg}\
        --v=${LOG_LEVEL} \
        --vmodule="${LOG_SPEC}" \
+	--fail-swap-on=false \


Why is this needed in this PR?

yujuhong · 2017-08-25T01:22:53Z

pkg/kubelet/kubelet.go

+			return nil, nil, err
+		}
+		if len(pods) > 0 {
+			c <- kubetypes.PodUpdate{Pods: pods, Op: kubetypes.SET, Source: kubetypes.ApiserverSource}


@liggitt is right. Kubelet assumes the first update from a source includes the complete list of pods. Once all sources are ready, GC will kick in to delete containers that do not belong to a known pod.

Wiring this so that kubelet considers the source is not ready is required.

yujuhong · 2017-08-25T01:31:28Z

pkg/kubelet/pod/pod_manager.go

@@ -154,6 +159,11 @@ func (pm *basicManager) UpdatePod(pod *v1.Pod) {
 	pm.lock.Lock()
 	defer pm.lock.Unlock()
 	pm.updatePodsInternal(pod)
+	if pm.checkpointManager != nil {
+		if err := pm.checkpointManager.WritePod(pod); err != nil {
+			glog.Errorf("Error writing checkpoint for pod: %v", pod.GetName())


What's the expected behavior if this happens? kubelet simply won't restore the pod at startup?

Yes, at some point intervention will be required, and given this is for bootstrapping this is ok.

yujuhong · 2017-08-25T01:37:38Z

pkg/kubelet/kubelet.go

+			for _, pod := range pods {
+				glog.Infof("Starting Checkpoint %v - Name (%v)", pod.GetUID(), pod.GetName())
+			}
+		}


Another problem is that if the pod gets deleted from the apiserver while kubelet is down, you would NOT receive a DELETE event from pkg/kubelet/config/apiserver.go through the regular channel because the cache in the pkg/kubelet/config would be cold. That means you will keep the pods restored from the checkpoint forever.

That's one of the reasons why I suggested also looking into storing the checkpoints there (pkg/kubelet/config).

Fixed following your example.

yujuhong · 2017-08-25T01:41:08Z

pkg/kubelet/checkpoint/checkpoint.go

+	var err error
+	if fcp.checkAnnotations(pod) {
+		if blob, err := yaml.Marshal(pod); err == nil {
+			err = ioutil.WriteFile(fcp.getPodPath(pod), blob, 0644)


ioutil.WriteFile doesn't call f.Sync(). You'll likely end up with corrupted files, which we may not be able to detect due to the lack of checksum?

Also, is the write atomic? Do we need to write to a temp file and move it here?

yes. we need to write a temp file, fsync it and rename.

Fixed, using safefile instead.

yujuhong · 2017-08-25T01:48:50Z

cmd/kubelet/app/options/options.go

@@ -232,6 +232,8 @@ func AddKubeletConfigFlags(fs *pflag.FlagSet, c *componentconfig.KubeletConfigur
 	fs.MarkDeprecated("experimental-fail-swap-on", "This flag is deprecated and will be removed in future releases. please use --fail-swap-on instead.")

 	fs.StringVar(&c.PodManifestPath, "pod-manifest-path", c.PodManifestPath, "Path to to the directory containing pod manifest files to run, or the path to a single pod manifest file. Files starting with dots will be ignored.")
+	fs.StringVar(&c.CheckpointPath, "checkpoint-path", c.CheckpointPath, "<Warning: Alpha feature> Path to to the directory where the checkpoints are stored")


I'm ok with @smarterclayton's suggestion for the flag experimental-pod-preservation-path

timothysc · 2017-11-15T15:05:17Z

/test pull-kubernetes-unit

yujuhong

The PR mostly looks good.

Besides the basic checkpoint write/restore testing (which should be a node e2e test suite), I'd like to see testing for the following cases:

Update the pods in the apiserver while kubelet is down. Ensure kubelet restores the pods from the checkpoints, AND gets the updates from the apiserver correctly (i.e., eventually gets the updated pods).
Delete the pods from the apiserver while kubelet is down. Ensure kubelet properly deletes pods (and the checkpoints).
Change annotation of a pod and observe checkpoints are handled properly.

As we agreed in our last meeting, manual testing is fine for these corner cases for now. Eventually, it'd be nice to have some test coverage in our CI.

yujuhong · 2017-11-16T19:41:01Z

pkg/kubelet/apis/kubeletconfig/v1alpha1/types.go

@@ -54,6 +54,9 @@ type KubeletConfiguration struct {
 	// podManifestPath is the path to the directory containing pod manifests to
 	// run, or the path to a single manifest file
 	PodManifestPath string `json:"podManifestPath"`
+	// bootstrapCheckpointPath is the path to the directory containing pod checkpoints to
+	// run on restore
+	BootstrapCheckpointPath string `json:"checkpointPath"`


s/json:"checkpointPath/json:"bootstrapCheckpointPath

yujuhong · 2017-11-16T19:41:59Z

pkg/kubelet/apis/kubeletconfig/types.go

@@ -58,6 +58,9 @@ type KubeletConfiguration struct {
 	// podManifestPath is the path to the directory containing pod manifests to
 	// run, or the path to a single manifest file
 	PodManifestPath string
+	// bootstrapCheckpointPath is the path to the directory containing pod checkpoints to
+	// run on restore
+	BootstrapCheckpointPath string


Spoke to @mtaufen offline. He has removed all Alpha features from the kubeletconfig, unless they are feature-gated.
To make it consistent with other fields, the suggestion is either

Remove this from the configuration, but keep it as a flag, or

Feature gate this properly.

BTW, removing this from the configuration may make node e2e testing harder, just FYI.

Yup, and just to add some context:

Removing it from the configuration for now doesn't prevent you from adding it back later, once you're more confident in the stability of your API and behavior (e.g. no longer an alpha feature).

If you do want an alpha feature in the API, these are the guidelines regarding feature gating it properly: https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md#alpha-field-in-existing-api-version

He has removed all Alpha features from the kubeletconfig, unless they are feature-gated.

@yujuhong This is patently untrue.

NodeLabels, VolumePluginDir, CPUManagerPolicy, LockFilePath are all alpha labeled features that exist in the kubeletconfig. The usage of componentconfig is also alpha, from what I've seen.

I can do whatever, but your statement doesn't reconcile with what is there.

@timothysc CPUManagerPolicy is feature gated, which is why it remains.
NodeLabels, VolumePluginDir, and LockFilePath were all moved from KubeletConfiguration to KubeletFlags in #55562, because they are alpha but lack feature gates.
This is in preparation for moving KubeletConfiguration to beta.
Please feature gate your field if you want to leave it in KubeletConfiguration or otherwise move it to KubeletFlags.
If you have more questions, feel free to ping me on Slack.

Fixed, following #55562 as reference.

yujuhong · 2017-11-16T19:44:03Z

pkg/kubelet/checkpoint/checkpoint.go

+
+// NewCheckpointManager will create a Manager that points to the following path
+func NewCheckpointManager(path string) Manager {
+	mutex.Lock()


Suggestion: add a comment to indicate that this is just for precaution; current implementation do/should not run multiple checkpoint managers.

yujuhong · 2017-11-16T19:48:33Z

pkg/kubelet/checkpoint/checkpoint.go

+		// Get just the Resource from "Resource_Name"
+		fnfields := strings.Split(fname, delimiter)
+		switch fnfields[0] {
+		case "Pod":


+1 for making it a constant.

yujuhong · 2017-11-16T19:50:08Z

pkg/kubelet/checkpoint/checkpoint.go

+		fnfields := strings.Split(fname, delimiter)
+		switch fnfields[0] {
+		case "Pod":
+			pod, err := fcp.loadPod(fmt.Sprintf("%v/%v", fcp.path, f.Name()))


nit: I'd prefer %s too since it's more clear.

yujuhong · 2017-11-16T21:06:17Z

pkg/kubelet/checkpoint/checkpoint.go

+func (fcp *fileCheckPointManager) DeletePod(pod *v1.Pod) error {
+	var err error
+	podPath := fcp.getPodPath(pod)
+	if _, err := os.Stat(podPath); !os.IsNotExist(err) {


Don't think you need to Stat first.

if err := os.Remove(podPath); !os.IsNotExist(err) { return err } return nil

yujuhong · 2017-11-16T21:16:09Z

pkg/volume/util/util.go

@@ -233,7 +233,7 @@ func LoadPodFromFile(filePath string) (*v1.Pod, error) {
 	}
 	pod := &v1.Pod{}

-	codec := legacyscheme.Codecs.LegacyCodec(legacyscheme.Registry.GroupOrDie(v1.GroupName).GroupVersion)
+	codec := legacyscheme.Codecs.UniversalDecoder()


Why changing this? I'm not familiar with the codecs, so just a question.

In testing that f(n) it was a bug.

yujuhong · 2017-11-16T21:47:36Z

pkg/kubelet/kubelet.go

@@ -293,9 +293,20 @@ func makePodSourceConfig(kubeCfg *kubeletconfiginternal.KubeletConfiguration, ku
 		glog.Infof("Adding manifest url %q with HTTP header %v", kubeCfg.ManifestURL, manifestURLHeader)
 		config.NewSourceURL(kubeCfg.ManifestURL, manifestURLHeader, nodeName, kubeCfg.HTTPCheckFrequency.Duration, cfg.Channel(kubetypes.HTTPSource))
 	}
+
+	// Restore from the checkpoint path


Add a comment warning that this must happen before creating the apiserver source below, or the checkpoint would override the source of truth (apiserver).

yujuhong · 2017-11-16T22:03:33Z

pkg/kubelet/kubelet.go

+			// the pod checkpoints and may be incomplete. Do not mark the
+			// source as ready.
+
+			// Mark the source ready after receiving at least one update from the


Remove the original (now duplicated) comment below (line 1880)

yujuhong · 2017-11-16T22:15:26Z

pkg/kubelet/checkpoint/checkpoint.go

+	} else {
+		// This is to handle an edge where a pod update could remove
+		// an annotation and the checkpoint should then be removed.
+		err = fcp.DeletePod(pod)


IIUC, every update of a regular pod without the checkpoint annotation will trigger a deletion, which either calls os.Stat or os.Remove. That doesn't seem good. If you plan to optimize that in the future, please add a TODO.

The stat precedes any delete, and again this is an opt-in behavior on master nodes for control plane bootstrapping, not general purpose checkpointing, but I can add a TODO to optimize if more generally used.

I updated below.

k8s-github-robot · 2017-11-21T04:15:30Z

[MILESTONENOTIFIER] Milestone Pull Request Current

@luxas @timothysc @yujuhong

Note: If this pull request is not resolved or labeled as priority/critical-urgent by Wed, Nov 22 it will be moved out of the v1.9 milestone.

Pull Request Labels

sig/cluster-lifecycle sig/node: Pull Request will be escalated to these SIGs if needed.
priority/important-soon: Escalate to the pull request owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
kind/feature: New functionality.

Help

timothysc · 2017-11-21T14:09:57Z

/test pull-kubernetes-e2e-gce-device-plugin-gpu

yujuhong · 2017-11-21T19:00:46Z

pkg/kubelet/checkpoint/checkpoint.go

+	if err != nil {
+		return nil, err
+	}
+	for _, f := range files {


Would this possibly contain temporary files written by safefile.WriteFile? Could you check and ignore them?

I think the code would print an warning, but it's quite annoying. Okay for an alpha feature though, as long as those temporary files are properly ignored. Please do check if that's the case.

yujuhong · 2017-11-21T19:07:31Z

Looks good overall. Look forward to seeing the followup test PRs.

yujuhong · 2017-11-21T19:09:13Z

/lgtm

k8s-github-robot · 2017-11-21T19:09:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: timothysc, yujuhong
We suggest the following additional approver: thockin

Assign the PR to them by writing /assign @thockin in a comment when ready.

Associated issue: 49236

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-11-22T01:57:38Z

Automatic merge from submit-queue (batch tested with PRs 55812, 55752, 55447, 55848, 50984). If you want to cherry-pick this change to another branch, please follow the instructions here.

aoxn · 2017-12-13T06:42:44Z

Is that mean that i have to power up the right machine which has apiserver.pod on to survive cluster power failure? I cant bring up my failed cluster back after power up the right node. Am i right? @timothysc

luxas · 2017-12-13T09:49:23Z

@spacexnice Please discuss that in an issue or on Slack. Thanks

timothysc added this to the v1.8 milestone Aug 20, 2017

timothysc assigned yujuhong Aug 20, 2017

timothysc requested review from luxas and yujuhong August 20, 2017 19:28

k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. sig/node Categorizes an issue or PR as relevant to SIG Node. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 20, 2017

luxas reviewed Aug 20, 2017

View reviewed changes

luxas self-assigned this Aug 20, 2017

rphillips reviewed Aug 21, 2017

View reviewed changes

k8s-ci-robot requested a review from roberthbailey August 23, 2017 13:55

timothysc mentioned this pull request Aug 23, 2017

kubeadm: Enable self-hosting by default #50968

Closed

timothysc requested a review from liggitt August 24, 2017 15:54

liggitt reviewed Aug 24, 2017

View reviewed changes

yujuhong reviewed Aug 25, 2017

View reviewed changes

k8s-github-robot added milestone/needs-approval and removed milestone/incomplete-labels labels Nov 15, 2017

timothysc added status/approved-for-milestone and removed milestone/needs-approval labels Nov 15, 2017

timothysc changed the title ~~Initial basic checkpoint support~~ Initial basic bootstrap-checkpoint support Nov 15, 2017

yujuhong reviewed Nov 16, 2017

View reviewed changes

timothysc force-pushed the checkpoint branch from 0e0e9cd to de59ff2 Compare November 19, 2017 02:18

k8s-github-robot added the milestone/needs-attention label Nov 20, 2017

Addition of bootstrap checkpointing

ed4401c

timothysc added the status/in-progress label Nov 21, 2017

k8s-github-robot removed the milestone/needs-attention label Nov 21, 2017

Update generated and dep addition of safefile

763122a

timothysc force-pushed the checkpoint branch from de59ff2 to 763122a Compare November 21, 2017 04:24

yujuhong reviewed Nov 21, 2017

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 21, 2017

timothysc added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 21, 2017

k8s-github-robot merged commit 277d866 into kubernetes:master Nov 22, 2017

roboll mentioned this pull request Feb 2, 2018

kubelet manifest dir sync broken #59275

Closed

knabben mentioned this pull request May 31, 2020

kubelet: remove the --bootstrap-checkpoint-path feature #91577

Merged

Initial basic bootstrap-checkpoint support #50984

Initial basic bootstrap-checkpoint support #50984

Conversation

timothysc commented Aug 20, 2017 • edited Loading

timothysc commented Aug 20, 2017

luxas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Aug 23, 2017

timothysc commented Aug 24, 2017

Choose a reason for hiding this comment

timothysc Aug 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Aug 24, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Aug 24, 2017 • edited Loading

smarterclayton commented Aug 24, 2017

yujuhong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Nov 15, 2017

yujuhong left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Aug 20, 2017 •

edited

Loading

timothysc Aug 24, 2017 •

edited

Loading

smarterclayton commented Aug 24, 2017 •

edited

Loading

timothysc commented Aug 24, 2017 •

edited

Loading