[BUG] Getting error "Couldn't get account Id" when trying to assume role into another account

[lrobles-iterable]

Is there an existing issue for this?

• I have searched the existing issues

YACE version

└─$ k exec -ti -n monitoring yace-exporter-746c78f7cd-tddgf -- sh                                                                                       1 ⨯
~ $ yace version
v0.56.0

Config file

    apiVersion: v1alpha1
    sts-region: us-west-2
    discovery:
      exportedTagsOnMetrics:
        acm:
          - CertificateArn
      jobs:
      - type: acm
        regions:
          - us-west-2
        roles:
          - roleArn: "arn:aws:iam::main_account:role/cloudwatch-exporter-sandbox"
          - roleArn: "arn:aws:iam::second_account:role/cloudwatch-exporter-stg"
        metrics:
          - name: DaysToExpiry
            statistics:
            - Maximum
            period: 3600
            length: 86400

Current Behavior

End goal:

Running yace in main_account, we want to be able to assume a role in second_account and get cloudwatch metrics

Before specifying role arns in the config file, we were getting metrics from main_account just fine, using the following definition in the helm chart values. This part is working

  replicaCount: 1
  image:
    repository: ghcr.io/nerdswords/yet-another-cloudwatch-exporter
    pullPolicy: IfNotPresent
    tag: ""

  fullnameOverride: "yace-exporter"

  serviceAccount:
    create: true
    labels: {}
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::main_account:role/cloudwatch-exporter-sandbox
    name: "yace-exporter"

the contents of config.yaml working with these values

  config: |-
    apiVersion: v1alpha1
    sts-region: us-west-2
    discovery:
      exportedTagsOnMetrics:
        acm:
          - CertificateArn
      jobs:
      - type: acm
        regions:
          - us-west-2
        metrics:
          - name: DaysToExpiry
            statistics:
            - Maximum
            period: 3600
            length: 86400

and the corresponding iam role trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::main_account:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/xxxxxxxx"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-2.amazonaws.com/id/xxxxxx:sub": "system:serviceaccount:monitoring:yace-exporter"
                }
            }
        },

After creating an iam role in second_account with the following trust policy, which allows assume_role from main_account's cloudwatch exporter role, the container fails to scrape any metrics and spits out errors

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::main_account:role/cloudwatch-exporter-sandbox"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

the config.yaml we're using to try and scrape metrics from both main_account and second_account

  config: |-
    apiVersion: v1alpha1
    sts-region: us-west-2
    discovery:
      exportedTagsOnMetrics:
        acm:
          - CertificateArn
      jobs:
      - type: acm
        regions:
          - us-west-2
        roles:
          - roleArn: "arn:aws:iam::main_account:role/cloudwatch-exporter-sandbox"
          - roleArn: "arn:aws:iam::second_account:role/cloudwatch-exporter-stg"
        metrics:
          - name: DaysToExpiry
            statistics:
            - Maximum
            period: 3600
            length: 86400

{
    "arn": "arn:aws:iam::second_account:role/cloudwatch-exporter-stg",
    "caller": "scrape.go:37",
    "err": "AccessDenied: User: arn:aws:sts::main_account:assumed-role/cloudwatch-exporter-sandbox/xxxxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::second_account:role/cloudwatch-exporter-stg\n\tstatus code: 403, request id: 31f9696f-1913-4516-9efc-49c2a6a581f9",
    "job_type": "acm",
    "level": "error",
    "msg": "Couldn't get account Id",
    "region": "us-west-2",
    "ts": "2024-02-13T21:42:37.682596075Z",
    "version": "v0.56.0"
}

Expected Behavior

the IAM role bound to the service account in main_account should be able to assume role into second_account

Steps To Reproduce

1. use the config file pasted earlier, with two roleArn definitons
2. create necessary IAM roles in both accounts, one with the AssumeRole trust policy
3. deploy helm chart to cluster

Anything else?

No response

[Flowlance]

I had the same issue. It worked for me when I added the command line feature flag -enable-feature=aws-sdk-v2