[FEATURE?] How to remove all the prometheus operator auto relables from your metrics

[WesselAtWork]

Is there an existing issue for this?

• I have searched the existing issues

Feature description

It took me an ENTIRE day to figure this out so I just want to drop it here in hopes it reaches someone in need.

We are running Prometheus operator so it automatically adds a bunch of kubernetes labels to the metrics that are scraped from the pod.
Stuff like the instance's ip address, the pod name etc.

It doesn't really make sense to have the pod's endpoint on a s3 metric or the node the ec2 metrics.

Eventually (after a LOT of googling) I figured out that you need to do:

relabelings:
  - action: labeldrop
    regex: (container|endpoint|namespace|pod|service|job)
metricRelabelings:
  - action: labeldrop
    regex: instance

And you SHOULD be left with only the labels that are scraped

Note:
This might be useful if you have multiple scrape sources but we are just running one.

What might the configuration look like?

# Set relabelings for the ServiceMonitor, use to apply to samples before scraping
# relabelings: []
# Set metricRelabelings for the ServiceMonitor, use to apply to samples for ingestion
# metricRelabelings: []
#
# Example config. Will strip all the auto 
# relabelings:
#   - action: labeldrop
#     regex: (container|endpoint|namespace|pod|service|job)
# metricRelabelings:
#   - action: labeldrop
#     regex: instance

Anything else?

Maybe add this in values.yaml?
Just as an example.

Hope this was the right place to put this, let me know if there is a better spot to put it.