kube-dns-anti-affinity: kube-dns never-co-located-in-the-same-node

[StevenACoffman]

What this PR does / why we need it:

This is upstreaming the kubernetes/kops#2705 pull request by @jamesbucher that was originally against kops.
Please see kubernetes/kops#2705 for more details, including a lengthy discussion.

Briefly, given the constraints of how the system works today:

• if you need multiple DNS pods primarily for availability, then requiredDuringSchedulingIgnoredDuringExecution makes sense because putting more than one DNS pod on the same node isn't useful
• if you need multiple DNS pods primarily for performance, then
  preferredDuringScheduling IgnoredDuringExecution makes sense because it will allow the DNS pods to schedule even if they can't be spread across nodes

Which issue this PR fixes

fixes kubernetes/kops#2693

Release note:

Improve resilience by annotating kube-dns addon with podAntiAffinity to prefer scheduling on different nodes.

[k8s-ci-robot]

Hi @StevenACoffman. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-github-robot]

Adding do-not-merge/release-note-label-needed because the release note process has not been followed.
One of the following labels is required "release-note", "release-note-action-required", or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.

[chrislovecnm]

/ok-to-test

[StevenACoffman]

/retest

[StevenACoffman]

Not sure why this would have caused the test failure... perhaps in the test there's a single node?

[StevenACoffman]

@justinsb @chrislovecnm I'm not familiar enough with the test suites to understand why this change would cause these errors:

W0909 20:42:43.501] error: context "e2e-f8n-agent-pr-93-0" does not exist
W0909 21:04:47.369] Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy)
W0909 21:04:47.635] error: context "e2e-f8n-agent-pr-93-0" does not exist
W0909 21:06:54.694] 2017/09/09 21:06:54 main.go:233: Something went wrong: error starting federation: error during ./federation/cluster/federation-up.sh: exit status 124
I0909 21:06:55.796] Starting pull-kubernetes-federation-e2e-gce-26305...
E0909 21:06:55.796] Command failed
I0909 21:06:55.797] process 20344 exited with code 1 after 39.8m
E0909 21:06:55.797] FAIL: pull-kubernetes-federation-e2e-gce

[bowei]

@csbell @kubernetes/sig-federation-misc for federation e2e failures

[MrHohn]

We might not need these alpha annotations anymore. This PR will likely go in for 1.9, and 1.5 is already 4 major versions away.

[MrHohn]

/retest

Ref #40063, #41125

cc @thockin

[chrislovecnm]

The main question that was raised on the PR into kops was if we use required or perferred. @thockin et al any thoughts?

[johanneswuerbach]

Any updates on this? We also just lost cluster dns service as all dns pods were located on the same node.

Anything I could help?

[bowei]

There is a discussion on the kops repo about using "preferred" vs "required". kubernetes/kops#2705

It seems to me we should go with "preferred" for now as it is strictly better than what we have before and does not result in kube-dns instances not being scheduled due to constraints.

If the PR is updated to preferred, I can lgtm.

[StevenACoffman]

@bowei Done!

[bowei]

/lgtm

[StevenACoffman]

Looks like the tests passed, and the weight is corrected. How's this look to you now @MrHohn and @bowei ?

[MrHohn]

Thanks!
/lgtm

Could you update release note as well? Ref PULL_REQUEST_TEMPLATE.

[k8s-github-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, MrHohn, StevenACoffman

Associated issue: 2705

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• cluster/addons/dns/OWNERS [MrHohn,bowei]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[k8s-github-robot]

Automatic merge from submit-queue (batch tested with PRs 53106, 52193, 51250, 52449, 53861). If you want to cherry-pick this change to another branch, please follow the instructions here.

[evanj]

YAY! Thanks! I just noticed that one of our clusters was running both its instances of kube-dns on a single machine today, then found this issue while searching around for bugs or mentions. I was going to attempt this, but this is even better!

[luxas]

This got reverted now, but please do keep kubeadm in sync at all times. The code lives in cmd/kubeadm as @johanneswuerbach pointed out (thanks!)

[jonastl]

1. For what reason was this reverted?
2. Was Kube-dns pods coming up super slowly in large clusters #54164 referring to an actual production cluster, or a "toy test" setup?
3. What is the current plan for ensuring the intended behavior of this PR?

I just opened a ticket on this very issue (in the wrong project unfortunately), as result of the described behavior causing problems in one of our production clusters.

We have three small control nodes which service critical infrastructure pods, and a slew of pre-emptive nodes (on GKE). Without this patch we end up with poor scheduling decisions, where the scheduler stacks kube-dns pods on top of each other on the same nodes, causing our own workload pods to fail scheduling due to kube-dns having stolen the Allocatable resources.

We had hoped to be able to use the merged patch with node anti-affinity, to keep kube-dns away from the preemptive worker nodes, to avoid the problem faced and described in #41125 (spotty DNS service).

Without this patch we'll end up with dozens of kube-dns pods (created by kube-dns-autoscaler), all stacked onto 3 x 2 core machines, stealing all allocatable CPU and preventing our own critical backplane pods from being scheduled.

Right now it seems our only option is to buy more machines for the sole purpose of hosting kube-dns. The pod(s) must run somewhere in the cluster, but we don't want them on pre-emptive nodes, and we don't want more than at most one on each control node. Since the reverted merge prevents us from telling the scheduler "not more than one per node!", it seems to me we'll need a new set of machines dedicated to kube-dns hosting. This will increase our cost and there will be more moving parts to manage (additional node group).

[bowei]

Unfortunately the scheduling feature (anti-affinity) does not currently scale very well, it cannot be used for a system service such as this. When the scheduling team improves the performance of the feature, we can re-examine the scheduler constraint...

[jonastl]

Bowei, do you have a proposal / suggestion for how to cope with kubedns until such a fix is out?
Is a dedicated kube-dns "server farm" the only option?

Edit: If we had self-hosted k8s, we could of course have disabled the kube-dns autoscaler and set the deployment replica count manually, and revised the deployment to enable anti-affinity ourselves. Unfortunately we're using google and they have some reconciliation process that tend to overwrite / revert customer made changes in kube-system.
This sort of issue straddles k8s and Google and I'm not sure whether to carry this discussion in k8s context or through our google support channel. They tend to point us here anyway as soon as it even has a potential of being a k8s concern.

[bowei]

It possible to run a customized version of kube-dns on the cluster:

NOTE: after following these steps, you will be responsible for maintaining the configuration of the DNS system. Changes can always be reverted to return to the original configuration.

• Download your current cluster kube-dns yaml:
  • kubectl get -n kube-system -o yaml deployment/kube-dns > my-kube-dns.yaml
  • Modify the YAML with your custom settings.
  • Change the name of the deployment in the metadata section to my-kube-dns.
  • Remove the label kubernetes.io/cluster-service: "true" from my-kube-dns.yaml. (The label is used by the Kubernetes addon manager to distinguish user resources from those managed by the Kubernetes system itself)
  • Make sure the cluster domain is set properly in my-kube-dns.yaml. (Search for cluster.local)
• Scale the dns autoscaler replicas to 0. You can use kubectl scale --namespace kube-system deploy/kube-dns-autoscaler --replicas 0.
• The next steps will disrupt DNS temporarily in the cluster:
• Create the deployment my-kube-dns.yaml as constructed above. ClusterFirst requests will be sent to my-kube-dns as well as kube-dns because they have the same application label.
• Scale the existing kube-system:kube-dns replicas to 0 to remove traffic from kube-dns. You can do this by setting the DNS auto scaler to have 0 replicas.
• Scale up my-kube-dns by setting replicas to the appropriate value.

[jonastl]

Excellent. Just one question.
What about the label addonmanager.kubernetes.io/mode: Reconcile?
I take it that should be removed as well.

[bowei]

Yes, anything related to the addon manager will need to be removed.

[jonastl]

Perfect, thanks!

[StevenACoffman]

Unfortunately the scheduling feature (anti-affinity) does not currently scale very well, it cannot be used for a system service such as this. When the scheduling team improves the performance of the feature, we can re-examine the scheduler constraint...

@bowei Is there an issue tracking the anti-affinity performance/scaling problem we can link to this PR to remind us to revisit it?

[MrHohn]

bowei Is there an issue tracking the anti-affinity performance/scaling problem we can link to this PR to remind us to revisit it?

I believe that issue is #54189.