Refactor secrets and auto-generate password + ssh_key

[cognifloyd]

Refactor secrets values to implement #14 and #16. Since these changes overlap, I combined them into one PR.

BREAKING CHANGE:
Move secrets.st2.{username,password,ssh_key,datastore_crypto_key} values into st2.* (implements #14).

Plus, auto-generate password and ssh_key secrets (implements #16), so that the stackstorm-ha installation is a bit more secure by default. On upgrade, use lookup to pull the current secrets from k8s APIs.

Resolves: #14
Resolves: #16
Closes: #62

[cognifloyd]

This change was triggered by: https://app.circleci.com/pipelines/github/StackStorm/stackstorm-ha/1192/workflows/9f1ccdab-bfef-4017-a98f-4b298bb49c75/jobs/4020

I'm using helm 3.5.4. e2e tests use the latest helm version, 3.6.0.

[cognifloyd]

rebased on master

[arm4b]

I hadn't chance to explore deeper the new lookup functionality.

How this thing will behave if user needs to override the previous password set? (whether it's auto-generated or custom)

Any edge cases?

[cognifloyd]

Hmm. We might need a flag like resetPassword which would make it skip the lookup. Something like this:

{{- if and .Release.IsUpgrade (not .Values.st2.resetPassword) }}

Wdyt?

[cognifloyd]

One edge case might be trying to update the password while upgrading from an existing install without setting that flag. I can't think of any other edge cases. You?

[cognifloyd]

fixed

[arm4b]

Ah, replied in another thread: https://github.com/StackStorm/stackstorm-ha/pull/203/files#r660046629

[arm4b]

Thanks! Looks great at a high level.

Based on the CI run (https://app.circleci.com/pipelines/github/StackStorm/stackstorm-ha/1251/workflows/dbc5fbc0-7557-4eb1-987e-a6d480366c3a/jobs/4191), the st2 password is not showed up in the NOTES when doing the initial Helm chart install:

2. Login with the following credentials:
username: st2admin
password:

This is a high-profile feature that'll affect everyone and so a few more cycles of testing won't hurt, especially around the corner cases we mentioned.

[cognifloyd]

Yeah. That's very sad - the lookup works on upgrade but not install.
Also, lookup always returns nil when using helm template to debug because helm is not supposed to contact the cluster during template.
I'll do a bit more research to see if there's a good way to grab the auto-generated value in multiple places. If you come across anything please let me know.

[cognifloyd]

Helm templates all of the files at once, before applying the resources, so we cannot lookup in NOTES what was generated in another file. For that matter, during the same helm run, we can't lookup anything generated in one template file from another template file.

Also, NOTES is stored with the other release information. By default helm 3 uses secret storage for that, but also allows using configmap and postgresql storage backends. It makes sense to minimize the number of places that expose the plain secrets, without regard to how the cluster is [mis-]configured. So, I think using kubectl to retrieve the password instead of embedding it directly in the NOTES marginally improves security for the default st2 ha installation.

So, I changed the NOTES to explain how to retrieve the password with kubectl. Wdyt?

[ericreeves]

I agree that it's a step forward to not just print the password in the NOTES. Providing the kubectl command to retrieve it is far safer and nearly as simple from a user experience perspective.

Printing passwords at every upgrade when they weren't requested is a bad thing. They could get caught in log files and terminal output all over the place, and this is the admin password!

Great idea for a PR, and I think the implementation looks nice!

[arm4b]

Looks good 👍 Thanks @cognifloyd!

WIll run a few tests though before merging.
Would appreciate if @ericreeves you could help with more manual testing around possible edge cases.