Provision nodes with kubeadm

[mumoshu]

Probably after kubeadm starts supporting multi-master & dedicated-etcd-nodes setup.

Relevant PR(which may or may not be merged): kubernetes/kubernetes#44793

[cknowles]

I'm curious about what this will add to kube-aws given the docs about it. @mumoshu What are your thoughts on what kube-aws gains from this?

[mumoshu]

@c-knowles Thanks for chiming in 👍
My thought is written in #675 - In short, I expect the amount of code for configuring apiserver and kubelets and so on to be reduced when we completely moved to kubeadm.
What do you think?

[cknowles]

I think it seems aligned in some aspects like self hosting. If we are sticking with CloudFormation then kubeadm doesn't seems like 100% a natural fit based on what the docs say. It depends on your view around how much we bake in and how many moving parts on node initialisation.

[mumoshu]

Probably adopting kubeadm should prevent issues like #793 in longer term
cc @danielfm

[luxas]

Probably after kubeadm starts supporting multi-master & dedicated-etcd-nodes setup.

kubeadm has supported external etcd clusters pretty much from the get go.
There is no out-of-the-box HA, no, but in situations where you can copy files around and have a loadbalancer, that's not a problem.

Probably adopting kubeadm should prevent issues like #793 in longer term

That and much more is exactly why kubeadm should be used as the basic building block in a cluster.
See my blog post about this here: http://blog.kubernetes.io/2017/01/stronger-foundation-for-creating-and-managing-kubernetes-clusters.html

I think this should work fine, at least to start speccing out. Do you mind attending the kubeadm adoption working group meeting later today?
cc @justinsb

See:
https://groups.google.com/forum/#!topic/kubernetes-sig-cluster-lifecycle/HHr7WphU_xE
https://groups.google.com/forum/#!topic/kubernetes-sig-cluster-lifecycle/v2TzSHwYe9I
https://docs.google.com/document/d/1KdXsLYiJYJdiRbtgZsx6qbHF4g_K-gAScB9Zs4avgzg/edit#heading=h.aly6m9xjbivl
https://docs.google.com/document/d/1SAoA_GDIioQ1rAt8VhwWqPAgHTQpJ6TVnjJUKDqyVA4/edit#heading=h.mpsw4vcdaazw

[mumoshu]

Thanks a lot for the kind follow-up, @luxas!

but in situations where you can copy files around and have a loadbalancer

What files should be copied? Also, you meant that the files should be copied from the node running the kubeadm master to nodes running kubeadm followers(I'm not entirely sure if the terminology is correct)

Would you mind guiding me about one more thing: How kubeadm should be run in Container Linux?
Is there a statically-linked binary of kubeadm which can be run outside of a container in Container Linux? Or should be docker-run a kubeadm process within a privileged container, or anything else?

Do you mind attending the kubeadm adoption working group meeting later today?

I wish I could, but unfortunately no. It is 2-3 am in my timezone, which isn't acceptable to me(It is just impossible for me to wake up at such time - in an Nth non-rem sleep after I've finally put my son to sleep 😴

I'd really appreciate it if I could virtually attend WG meetings like that asynchronously via recorded videos, meeting notes, etc.

[mumoshu]

So, should we at least sync the following files among controller nodes?

• The config file provided to kubeadm when the --config flag is used instead of the configuration through flags
• /etc/kubernetes/admin.conf which contains the admin credentials generated on kubeadm init without the --token flag.
  • However, not specifying --token would require us to coordinate kubeadm init runs among multiple controller nodes. Otherwise, admin credentials would conflict among different instances of kubeadm masters among controller nodes.

Ref: https://kubernetes.io/docs/admin/kubeadm/

Would there be anything else?

[mumoshu]

Also, can we instruct kubeadm to run kubelets in rkt pods rather than in docker containers like we do currently? How? (I know apiserver and controller pods are run as static pods via cri configured for kubelets)

According to the "Use Kubeadm with other CRI runtimes" section of the kubeadm doc, it seems to support rkt through CRI. However, isn't it only for configuring kubelets? What is unclear to be is how we could configure runtimes of kubelets themselves.

Update:

4. Outputting a kubeconfig file for the kubelet to use to connect to the API server, as well as an additional kubeconfig file for administration.

Perhaps we must prepare a kubelet systemd unit per node before kubeadm init starts?

7. kubeadm installs add-on components via the API server. Right now this is the internal DNS server and the kube-proxy DaemonSet.

However, this sentence seems to imply that we would need a running kubelet which requires kubeconfig, before kubeadm init which writes kubeconfig in the first place.
Sounds like a chicken-and-egg problem.
I must be missing something.

[luxas]

@mumoshu Thanks for writing this up.
I think the most productive way here would be to schedule a meeting or something and talk this through, then follow up on this thread and post the TL;DR; results/outcomes.

Very unfortunate on the timezones! However, I'm in the GMT+2 timezone, so we should be able to sync 1:1 at least I think... I can act as a proxy to the others in the kubeadm adoption working group later.

[mumoshu]

@luxas Thanks for the suggestion 👍 I'm still catching things up so let's talk after that.

[mumoshu]

According to the info gathered until now, this issue can be solved with or without it, but anyway I've submitted a feature request to add dedicated etcd nodes support for kubeadm in kubernetes/kubeadm#491.

Also note that, at first glance, I took it as kubeadm would allow us to provision every kube-aws node after kubernetes/kubeadm#261 is addressed. However, it doesn't seem to include dedicated etcd nodes bootstrapped by kubeadm as of today.

[redbaron]

Isn't kubeadm model implies SSHing into already up and running nodes and installing packages/updating configs? It doesn't play well with autoscaling groups. It also need to support coreos.

[luxas]

@mumoshu kubeadm supports external etcd. That should probably work for your case (setting up etcd yourself, delegating k8s bootstrap to kubeadm). Regarding high availability -- it is totally possible to set up HA clusters with kubeadm if you can a) move the certificates for the cluster to all your masters b) set up a LB in front of the API servers. I think you have those capabilities, so you should be good to go.

@redbaron kubeadm handles bootstrapping of Kubernetes on a machine that exists. You can install / set up kubeadm in a boot script or afterwards by executing commands via ssh, or whatever. How do you currently set up k8s after the machines are created?

It also need to support coreos.

We just don't provide packages for CoreOS, but you can indeed use kubeadm on CoreOS.

Please read the design doc here about technical details: https://github.com/kubernetes/kubeadm/blob/master/docs/design/design_v1.8.md

[mumoshu]

@luxas Thanks for the clarifications!

I'm still looking forward to work on this soon.

I've studied a bit about kubeadm - it seems to just writes various files required for a master/worker node into well-known locations on the local filesystem, so that the kubelet on the node is able to read them to deploy static pods, deployments, daemonsets to form a k8s cluster.
Is my understanding correct?

How do you currently set up k8s after the machines are created?

1. On each controller node, we run kubelet via systemd.

2. apiservers and controller-managers are deployed as static pods and hence /etc/kubernetes/manifests contains static pod manifests for those.

3. We also deploy kube-dns, metrics-server, kube-proxy by running kubectl apply -f "from controller nodes", within a bash script invoked on each node's startup process via systemd.

I expect kubeadm to do the steps 2 and 3.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.