Support private helm (deps) repositories

[romuduck]

Hello,

looking at the code below, I assume private repositories are not supported

func (h helmProcessor) repoEnsure(path string, name string, url string) error {
	_, err := h.helmDo(path, `repo`, `add`, name, url)
	return err
}

I was wondering if we could provide a limited support by leveraging ArgoCD env variables like: GIT_USERNAME and GIT_PASSWORD, assuming the same credentials for source repo, and all other repo found underneath.
It would work in my case but maybe too limited for community.

++
thanks

[Joibel]

I'll have a look at this. We should work out how to support it.
GIT_USERNAME and GIT_PASSWORD wouldn't get through Argo CD 2.4's allowed plugin names: https://github.com/argoproj/argo-cd/blob/master/docs/operator-manual/upgrading/2.3-2.4.md#update-plugins-to-use-newly-prefixed-environment-variables

I could make ARGOCD_ENV_GIT_ work as alternatives for 2.4.

If you can token authenticate with your repo, you could define your repo as https://@/path and I'm pretty sure that would work. It isn't ideal.

I will add user+password support though.

[romuduck]

Thanks @Joibel, I ll try your workaround.
Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Joibel]

Would still like to fix this.

[paveq]

This would be really useful feature to have.

[Tyrion85]

If you can token authenticate with your repo, you could define your repo as https://@/path and I'm pretty sure that would work. It isn't ideal.

@Joibel could you give a more detailed example? I've been trying to configure a remote base for kustomize, to no avail (because local bases don't work).

Given a private GitHub repo https://github.com/MyOrg/my-repo and a base path k8s/base/namespaces, what should a kustomization.yaml look like? This repo is added to ArgoCD with access token (or with ssh key, both work for me)

I've tried a number of different combinations of https with access token and ssh with ssh key, none of them seem to work. Some examples that don't work:

git::https://github.com/MyOrg/my-repo/k8s/base/namespaces?ref=main
https://github.com/MyOrg/my-repo/k8s/base/namespaces?ref=main
https://@/github.com/MyOrg/my-repo/k8s/base/namespaces
https://@/github.com/MyOrg/my-repo/k8s/base/namespaces?ref=main
https://[email protected]/MyOrg/my-repo/k8s/base/namespaces
https://[email protected]/MyOrg/my-repo/k8s/base/namespaces?ref=main
ssh://github.com/MyOrg/my-repo.git//k8s/base/namespaces
ssh://[email protected]/MyOrg/my-repo.git//k8s/base/namespaces
.......

Weirdly enough, ssh://github.com/MyOrg/my-repo.git//k8s/base/namespaces works when a new test case is added in test directory here in source code, but not when run in ArgoCD with reposerver having an ssh config to repo.

Thanks! 🙏🏼

[Joibel]

https://[email protected]/MyOrg/my-repo/k8s/base/namespaces is what I was thinking of.

[Tyrion85]

https://[email protected]/MyOrg/my-repo/k8s/base/namespaces is what I was thinking of.

oh, that won't work for obvious reasons out of the box :( can't really store a token in git, and there is no CI or any scripting in front of it. Maybe some preprocessing would work, but that sounds complicated. Don't you guys have requirements for inclusion of bases? Curious what others are doing and how they're solving this issue.

[Joibel]

I don't use bases myself, no.
I use applicationsets with LOVELY_KUSTOMIZE_MERGE/PATCH or sometimes just applications with same for variations. But the differences are minor between instances.
I'd like what you're doing to work. This ticket here should really be solved by reading and using the ArgoCD secrets that already exist. This is non-trivial to implement, but it is here to be done. Solving #66 for you is easier, so more likely to get done in the short term.
If you're using hashicorp vault our sister project argocd-vault-replacer can easily plug your token in at argocd level, but setting any of that up is hard work, so I don't really recommend it as a solution to your problem.

[astephanh]

@Tyrion85 One workaround for now would be to download the dependency chart and put in the charts folder.

[Joibel]

The problem with using ArgoCD secrets that it has for access to private repos is that lovely doesn't implement projects, and access to those secrets is guarded by project membership. I'll investigate project detection, but I really suspect this won't work.

As a workaround, given that only admins should have access to the application object itself, is to provide to lovely a list of secrets that it may use as secrets for repositories, then we can use argocd's secrets instead of implementing our own.

[astephanh]

Hi,

i couldn't find any information about a plugin could access the repository settings. But - just an idea here: maybe it would be possible to use the Argocd "internal" helm integration to pull the charts via the repository settings.

[wmiller112]

Also running into this trying to use remote resource bases in a private github repo with kustomize. For other plugins, I've been able to work around this by using a git_askpass script as described here. Basically have a script in a configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: git-askpass
data:
  git_askpass.sh: |
    #!/usr/bin/env bash
    set -euo pipefail
    username="${GIT_USERNAME:-${GIT_USER:-}}"
    password="${GIT_TOKEN:-${GIT_PASSWORD:-}}"
    if [ "${1:-}" = get ]; then
      echo "username=$username"
      echo "password=$password"
    elif [[ "$*" =~ Username ]]; then
      echo "$username"
    elif [[ "$*" =~ Password ]]; then
      echo "$password"
    fi

and apply patches to the repo server:

- op: add
  path: /spec/template/spec/containers/0/env
  value:
    - name: GIT_USER
      valueFrom:
        secretKeyRef:
          name: github-creds
          key: username
    - name: GIT_TOKEN
      valueFrom:
        secretKeyRef:
          name: github-creds
          key: password

- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    name: git-askpass
    mountPath: /usr/local/bin/git_askpass.sh
    subPath: git_askpass.sh

- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: git-askpass
    configMap:
      name: git-askpass
      defaultMode: 0555
      items:
        - key: git_askpass.sh
          path: git_askpass.sh

And an example cmp that then is able to successfully use kustomize with remote private bases:

- name: kustomize-envsubst
   generate:
     command: ["sh", "-c"]
     args: ["export GIT_ASKPASS=/usr/local/bin/git_askpass.sh && kustomize build --enable-helm --load-restrictor=LoadRestrictionsNone . | envsubst"]

I tried throwing the export GIT_ASKPASS=/usr/local/bin/git_askpass.sh in a dedicated plugin, and then calling that in preprocessor, but that doesn't seem to work. I think probably needs to be built into the docker image to run before kustomize/helm, otherwise its probably getting overwritten as discussed in that linked argo issue. Going to fork and test a bit.

[wmiller112]

So this of course is just a quick change to test with what I already had configured - github creds in a kubernetes secret, askpass script in configmap, all mounted to the repo server. I found that setting GIT_ASKPASS within the plugin (here) enabled private git access

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Joibel]

Poke stalebot

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jainsrbh]

How can I connect the plugin to ECR? In my organisation, helm charts and docker images are mirrored in ECR. We also have our own custom charts and docker images.

[Joibel]

Public ECR repositories work. This issue is still open as there is no support for private repos in lovely of any kind.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Joibel]

There is discussion around a credentials service becoming part of ArgoCD, which would enable this to be implemented.

[br0nwe]

We are facing the same issue. We cannot access our Nexus Helm Repositories with this as the credentials are not used

[aguckenber-chwy]

My use case is to connect to private Artifactory which is the same as everyone above. One thing lovely could possibly do is read from a standard named Kubernetes Secret (Ex: argocd-lovely-creds) and when deploying a helm chart we can possible tell lovely what keys to use from the secret for credentials to get the helm chart. Any ideas to even hack around this issue would be great.

TBH this private repository stuff has been the bane of my existence for the last week. Vanilla applications work but can't be customized (which this plugin does) and kustomization helmCharts can be customzied but don't work with private repositories because credentials can't be set.

[br0nwe]

@aguckenber-chwy
i feel with you.
Before we used https://github.com/crumbhole/argocd-vault-replacer directly as an ArgoCD Plugin and it did everything we needed. With the new argocd version we are forced to use it via the lovely plugin
Looking at #339 means imho that they do not plan to get the lost feature back
It is a step back :( I do not say its lovelys fault. it is just a feature that dissapears for us in the argocd ecosystem

For our usecase we managed to get rid of lovely for this feature by integrating https://external-secrets.io

[MaksimKlepikov]

As a side note to future passersby.

Came here after argocd-vault-plugin which can't work with builtin helm and force you to do all the helm heavy lifting
But it turns out that Lovely doesn't support private repositories.
So far, there doesn't seem to be an easy way to make your private Helm chart values fill up with vault secrets.
This proposal should make life easier , but it's stalled

Now looking at https://external-secrets.io/ too.

[borisovano]

Sadly due to CMP plugin approach, we ended up giving up in favor of external-secrets.

[Joibel]

External secrets is the right answer long term anyway.