Document manual steps to setup monitoring for etcd

[surajssd]

Create a document to help users setup monitoring for etcd manually. Taking inspiration from the document create an update issue to automate those steps in #252.

[invidian]

I've looked how this can be done and actually it requires few things to be done:

• We need to generate multiple etcd client certificates with different CNs:
  • CN: root, this one is special, for administrative tasks, including enabling RBAC
  • CN: kube-apiserver (CN can be different), for kubernetes API server, should have readwrite access to entire etcd
  • CN: prometheus (CN can also be different), for monitoring, should have no role granted
• --client-cert-auth=true flag must be added to etcd: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/authentication.md#using-tls-common-name
• Following commands needs to be executed:

  #!/bin/sh
  # Setup connection
  export ETCDCTL_API=3
  export ETCDCTL_CACERT=<ca cert path>
  export ETCDCTL_CERT=<root user client cert path>
  export ETCDCTL_KEY=<root user client private key path>
  export ETCDCTL_ENDPOINTS=https://<etcd member IP>:2379
  # Create root user, role and role grant, so 'auth enable' does not print error.
  etcdctl user add --no-password=true root
  etcdctl role add root
  etcdctl user grant-role root root
  etcdctl auth enable
  etcdctl user add --no-password=true kube-apiserver
  etcdctl role add kube-apiserver
  etcdctl role grant-permission kube-apiserver readwrite --prefix=true /
  etcdctl user grant-role kube-apiserver kube-apiserver
  etcdctl user add --no-password=true prometheus
  # Until https://github.com/etcd-io/etcd/issues/8458 is resolved.
  etcdctl user grant-role kube-apiserver root
  

Then, it works just fine. kube-apiserver works as before and have access to everything and prometheus certificate can only list metrics securely, but cannot access any data.

[invidian]

See also etcd-io/etcd#8458.

[iaguis]

Is this still relevant after #493?

[invidian]

No, but there should be a follow up issue created to scrape etcd metrics securely. Let me create it.

[invidian]

This is now done in #493, though insecurely, so let's follow up on that in #553.