Bugfixes and features
Services:
- Add Auto Discovery Support For Sagemaker by @charleschangdp
- Add support for AWS/TrustedAdvisor by @cristiangreco
Bugs:
- fix(kafkaconnect): update resource filter by @cgowthaman
- Validate should fail when no roles are configured by @thepalbi
- Fix default value for nilToZero and addCloudwatchTimestamp in static job by @cristiangreco
- ddos protection: Discover resources outside us-east-1
Dependencies
- Bump github.com/aws/aws-sdk-go from 1.44.284 to 1.44.290
- Bump github.com/aws/aws-sdk-go-v2/service/amp from 1.16.12 to 1.16.13
- Bump github.com/aws/aws-sdk-go-v2/service/apigatewayv2 from 1.13.12 to 1.13.13
- Bump github.com/aws/aws-sdk-go-v2/service/cloudwatch from 1.26.1 to 1.26.2
- Bump github.com/aws/aws-sdk-go-v2/service/ec2 from 1.100.0 to 1.102.0
- Bump github.com/prometheus/client_golang from 1.15.1 to 1.16.0
- Bump github.com/prometheus/common from 0.43.0 to 0.44.0
- Bump github.com/urfave/cli/v2 from 2.25.6 to 2.25.7
Full Changelog: https://github.com/nerdswords/yet-another-cloudwatch-exporter/compare/v0.52.0...v0.53.0
Important news and breaking changes
This releases introduces the feature flag aws-sdk-v2 (by @kgeckhart), which changes YACE networking layer to use the AWS sdk v2 package. Read on for more details and considerations.
- The main benefit of sdk v2 is deserialization/serialization is done via code generation vs reflection which drastically lowers memory/cpu usage for large scrape jobs
- Considerations before enabling sdk v2:
- FIPS is not supported in v2 as v2 delegates all URL resolution to the sdk and AWS does not have FIPS compliant endpoints for AutoScaling API and Tagging API. The v1 implementation worked around this by hard coding FIPS URLs where they existed and using non-FIPS URLs otherwise. This work around was not ported to v2 and is unlikely to be ported.
- sdk v2 uses regional sts endpoints by default vs global sts which is considered legacy by aws. The
sts-regionjob configuration is still respected when setting the region for sts and will be used if provided. If you still require global sts instead of regional set thests-regiontoaws-global.
Bugfixes and features
Features:
- Discovery jobs support
recentlyActiveOnlyparameter to reduce number of old metrics returned by CloudWatch API by @PerGon - Feature flag
aws-sdk-v2: use the more performant AWS sdk v2 (see above section) by @kgeckhart
Services:
- Add support for API Gateway V2 by @matej-g
- Add support for MediaConvert by @theunissenne
- Add support for CWAgent by @cristiangreco
- Add support for memorydb by @glebpom
Docs:
- ALB example: use Average for ConsumedLCUs by @cristiangreco
- Update configuration.md: deprecated custom namespace jobs by @wimsymons
- Update permissions examples and docs in readme by @kgeckhart
- Add example for ElastiCache by @cristiangreco
- Update mixin readme by @cristiangreco
Bugs:
- Fix AmazonMQ Broker name dimension match by @cristiangreco
- Fix invalid GH action file and broken test case by @cristiangreco
- Fix namespace case in metrics conversion by @cristiangreco
- Make exporter options a non-global type by @kgeckhart
- Fix debug logging in discovery jobs by @cristiangreco
Refactoring:
- Refactor AWS sdk client usage to hide behind new ClientCache by @kgeckhart
- Introduce model types to replace sdk types in cloudwatch client by @kgeckhart
Dependencies
New dependencies:
- github.com/aws/aws-sdk-go-v2/config 1.18.27
- github.com/aws/aws-sdk-go-v2/service/amp 1.16.11
- github.com/aws/aws-sdk-go-v2/service/apigateway 1.13.13
- github.com/aws/aws-sdk-go-v2/service/autoscaling 1.28.9
- github.com/aws/aws-sdk-go-v2/service/cloudwatch 1.26.1
- github.com/aws/aws-sdk-go-v2/service/databasemigrationservice 1.25.7
- github.com/aws/aws-sdk-go-v2/service/ec2 1.100.0
- github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi 1.14.14
- github.com/aws/aws-sdk-go-v2/service/storagegateway 1.18.14
Updates:
- Bump alpine from 3.17.3 to 3.18.2
- Bump github.com/aws/aws-sdk-go from 1.44.249 to 1.44.284
- Bump github.com/prometheus/common from 0.42.0 to 0.43.0
- Bump github.com/sirupsen/logrus from 1.9.0 to 1.9.3
- Bump github.com/stretchr/testify from 1.8.2 to 1.8.4
- Bump github.com/urfave/cli/v2 from 2.25.1 to 2.25.6
- Bump golang.org/x/sync from 0.1.0 to 0.3.0
- Bump golangci/golangci-lint-action from 3.4.0 to 3.6.0
Full Changelog: https://github.com/nerdswords/yet-another-cloudwatch-exporter/compare/v0.51.0...v0.52.0
Important breaking changes
- Jobs of type
customNamespaceare deprecated and might be removed in a future release (please reach out if you're still using this feature)
Bugfixes and features
Features:
- Add feature flags support by @thepalbi
- Feature flag
max-dimensions-associator: new resource-matching algorithm for discovery jobs. It fixes metrics attribution for ECS. Please test it out and report any issue! - Feature flag
list-metrics-callback: reduce memory usage of ListMetrics API requests
Services:
- Add support for AWS/Usage namespace by @cristiangreco
- Fix ECS regexes by @cristiangreco
Docs:
- Add docker compose support for easier development by @thepalbi
- Add more config examples by @cristiangreco
- Review docs about embedding yace by @cristiangreco
Bugs:
- Fix for Dockerfile smell DL3007 by @grosa1
Refactoring:
- Refactor Tagging/CloudWatch clients by @cristiangreco
- CloudWatch client: split out input builders into separate file by @cristiangreco
- Refactor promutils migrate functions by @cristiangreco
- Use grafana/regexp by @cristiangreco
- Refactor implementation of getFilteredMetricDatas by @cristiangreco
- Remove uneeded Describe implementation by @kgeckhart
- Add counter to see if duplicate metrics are still a problem by @kgeckhart
- Refactor label consistency and duplicates by @kgeckhart
- Refactor GetMetricData calls in discovery jobs by @cristiangreco
Dependencies
- Bump github.com/aws/aws-sdk-go from 1.44.235 to 1.44.249
- Bump github.com/prometheus/common from 0.41.0 to 0.42.0
Full Changelog: https://github.com/nerdswords/yet-another-cloudwatch-exporter/compare/v0.50.0...v0.51.0
Important breaking changes
- Change
UpdateMetricssignature to accept options and return error by @cristiangreco -- if you embed YACE as a Go library this is a breaking change.
Bugfixes and features Features:
- Refactor API clients concurrency handling by @cristiangreco
- Add feature flags support by @thepalbi
- Allow discovery jobs to return result even if there are no resources by @kgeckhart
- Add flag to enable pprof profiling endpoints by @cristiangreco
Services:
- Add a ResourceFilter to ElasticBeanstalk by @benbridts
Docs:
- Update config docs format by @cristiangreco
Refactoring:
- Linting: fix revive issues by @cristiangreco
- Remove extra error log when no resources are found by @kgeckhart
- Wrap debug logging in FilterMetricData by @cristiangreco
- Minor internal refactorings by @cristiangreco
Dependencies
- Bump actions/setup-go from 3 to 4
- Bump github.com/aws/aws-sdk-go from 1.44.215 to 1.44.235
- Bump github.com/urfave/cli/v2 from 2.25.0 to 2.25.1
Full Changelog: https://github.com/nerdswords/yet-another-cloudwatch-exporter/compare/v0.49.2...v0.50.0
- Update release action to use goreleaser docker image v1.16.0
- Update release action to use Go 1.20
- From now on we're dropping the
-alphasuffix from the version number. YACE will be considered alpha quality until v1.0.0. - The helm chart is now hosted at https://github.com/nerdswords/helm-charts, please refer to the instructions in the new repo.
Helm chart:
- Move helm chart out of this repo by @cristiangreco
- Update helm repo link in README.md by @cristiangreco
New services:
- Add support for Container, queue, and database metrics for MWAA by @millin
- Add support for acm-pca service by @jutley
Docs updates:
- Docs review: move "install" and "configuration" in separate docs by @cristiangreco
- Docs: Fix example config link by @matej-g
- Add example config files by @cristiangreco
Internal refactoring:
- Code refactoring: split out job and api code by @cristiangreco
- Minor refactoring of pkg/apicloudwatch and pkg/apitagging by @cristiangreco
- Refactor CW metrics to resource association logic and add tests by @thepalbi
- Wrap service filter errors by @kgeckhart
- Bump github.com/aws/aws-sdk-go from 1.44.194 to 1.44.215
- Bump github.com/prometheus/common from 0.37.0 to 0.41.0
- Bump github.com/stretchr/testify from 1.8.1 to 1.8.2
- Bump github.com/urfave/cli/v2 from 2.24.3 to 2.25.0
- Bump golang.org/x/sync from 0.0.0-20220722155255-886fb9371eb4 to 0.1.0
Bugfixes and features:
- Revert "Publish helm chart before releasing binaries".
Bugfixes and features:
- Add Elemental MediaLive, MediaConnect to supported services by @davemt
- Add support for OpenSearch Serverless by @Hussainoxious
- Makefile: always add build version ldflags by @cristiangreco
- Publish helm chart before releasing binaries by @cristiangreco
- Build with Go 1.20 by @cristiangreco
Dependencies:
- Bump github.com/aws/aws-sdk-go from 1.44.192 to 1.44.194
- Bump github.com/urfave/cli/v2 from 2.24.2 to 2.24.3
Breaking changes:
- If you use Yace as a library: this release changes the package
name
pkg/loggertopkg/logging.
Bugfixes and features:
- Fix to set logging level correctly by @cristiangreco
- ct: disable validate-maintainers by @cristiangreco
Dependencies:
- Bump github.com/aws/aws-sdk-go from 1.44.189 to 1.44.192
Breaking changes:
- Note if you use Yace as a library: this release changes the signature
of
config.Loadmethod.
Bugfixes and features:
- Helm chart update to customize port name by @nikosmeds
- Clear up docs and re-organize sections by @thepalbi
- Helm: add README file template by @cristiangreco
- Config parsing: emit warning messages for invalid configs by @cristiangreco
- Pre-compile dimensions regexps for supported services by @cristiangreco
- AWS/DX: add more dimension regexps by @cristiangreco
Dependencies:
- Bump github.com/aws/aws-sdk-go from 1.44.182 to 1.44.189
- Bump github.com/urfave/cli/v2 from 2.23.7 to 2.24.2
- Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0
Breaking changes:
- Note if you use Yace as a library: this release changes the packages and funcs exported publicly, you will need to review the imports (although signatures are mostly unchanged)
Bugfixes and features:
- Refactor code into separate packages by @cristiangreco
- Refactor list of supported services and filter funcs by @cristiangreco
- Wrap debug logging to avoid expensive operations by @cristiangreco
- Fix to use length of metrics level on customNamespace by @masshash
- feat: bump helm chart by @rasta-rocket
- feat: release helm chart when Chart.yml is updated by @rasta-rocket
- Add test for configuration of services list by @cristiangreco
- GolangCI: review linters settings by @cristiangreco
Dependencies:
- Bump azure/setup-helm from 1 to 3
- Bump docker/setup-buildx-action from 1 to 2
- Bump docker/setup-qemu-action from 1 to 2
- Bump github.com/aws/aws-sdk-go from 1.44.175 to 1.44.182
- Bump github.com/prometheus/client_golang from 1.13.0 to 1.14.0
- Bump helm/chart-releaser-action from 1.4.1 to 1.5.0
- Bump helm/kind-action from 1.2.0 to 1.5.0
- add support to custom namespaces with their dimensions (by @arielly-parussulo)
- Optimise support for custom namespaces to use GetMetricData API (by @code-haven)
- GH workflows: run "publish" workflows only in this repo. (by @cristiangreco)
- Bump Go version to 1.19 for CI and docker image. (by @cristiangreco)
- Fix not to refer to loop variable in a goroutine (by @masshash)
- Validate tags when converting to prometheus labels (by @cristiangreco)
- Bump github.com/aws/aws-sdk-go from 1.44.127 to 1.44.167
- Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1
- Bump github.com/urfave/cli/v2 from 2.23.0 to 2.23.7
- Resolve logging issue (@datsabk)
- MediaTailor - Correct dimension regex for MT (@scott-mccracken)
- Helm chart update for optional test-connection pod (@nikosmeds)
- Helm chart update to set priorityClassName (@nikosmeds)
- Bump github.com/aws/aws-sdk-go from 1.44.122 to 1.44.127
- Bump github.com/urfave/cli/v2 from 2.20.3 to 2.23.0
- Clean up unused variables. (@cristiangreco)
- Fix typo: sts-endpoint should be sts-region. (@cristiangreco)
- Enabled Managed prometheus metrics (@datsabk)
- Add support for AWS Kafka Connect (@cgowthaman)
- Import CloudWatch mixin. (@jeschkies)
- main.go refactoring: define cmd action as a separate func. (@cristiangreco)
- Add support for EMR Serverless (@cgowthaman)
- Fix typo in Charts.yml (@yasharne)
- Subcommand
verify-configactually validates the config file. (@cristiangreco) - Add dimensions regex for AmazonMQ. (@cristiangreco)
- Fix metrics with additional dimensions being not being scraped. (@cristiangreco)
- Remove unused code, add test for RemoveDuplicateMetrics. (@cristiangreco)
- Bump github.com/sirupsen/logrus
- Bump github.com/urfave/cli/v2
- Bump github.com/aws/aws-sdk-go
- Bump actions/setup-python
- Improve code quality and unblock this release (cristiangreco)
- Add helm chart (vkobets)
- Fix DX metrics (paulojmdias)
- Fix searchTags and bad dimension name (femiagbabiaka)
- Handle empty list in filter metric tests (mtt88)
- Add AWS Elemental MediaTailor support (scott-mccracken)
- Support storagegateway metrics (sedan07)
- Filter api gateway resources to skip "stages" (ch4rms)
- Bump aws-sdk, urfave/cli, prometheus/client_golang
- Set max page size for tagging API requests (#617)
- Build with Go 1.18
- New config
dimensionNameRequirementsallows autodiscovery jobs to only fetch metrics that include specified dimensions (jutley) - Update deps
- Cost Reduction - Use less API requests if no tagged resources are found (cristiangreco)
- Update deps
- Use structured logs for logging interface (kgeckhart)
- BREAKING CHANGE FOR LIBRARY USERS Major refactoring of usage of logging library (kgeckhart)
- Minor update of deps and security patches (urfave/cli/v2, golangci/golangci-lint-action, github.com/prometheus/client_golang, github.com/stretchr/testify, github.com/aws/aws-sdk-go
- Updates of Readme (markwallsgrove)
- Update dependencies
- Improve / Document way how to use the exporter as external library (kgeckhart)
- Refactor label consistency (kgeckhart)
- Add suppot for vpc-endpoint (AWS/PrivateLinkEndpoints) (aleslash)
- Add support for vpc-endpoint-service (AWS/PrivateLinkServices) (aleslash)
- Update dependencies
- Add weekly dependabot updates (jylitalo)
- Add support for regional sts endpoints (matt-mercer)
- Add multi-arch docker build (charlie-haley)
New services
- Add global accelerator support (charlie-haley)
- Add AppStream support (jhuesemann)
- Add Managed Apache Airflow support (sdenham)
- Add KinesisAnalytics support (gumpt)
Bug Fixes
- Fix targetgroup arn lookup (domcyrus)
- Fix WorkGroup Dimension are not showing in Athena Metrics (sahajavidya)
- Improve regex performance (kgeckhart)
- Fix prometheus reload causing a goroutine leak (gumpt / cristiangreco)
Docs
- Added help for new contributors (aleslash)
- Add /healthz route which allows to deploy more secure with helm (aleslash)
- Read DMS replication instance identifier from the DMS API (nhinds)
- [BREAKING] Fix the calculation of start and end times for GetMetricData (csquire)
floating-time-window is now replaced with roundingPeriod
Specifies how the current time is rounded before calculating start/end times for CloudWatch GetMetricData requests. This rounding is optimize performance of the CloudWatch request. This setting only makes sense to use if, for example, you specify a very long period (such as 1 day) but want your times rounded to a shorter time (such as 5 minutes). to For example, a value of 300 will round the current time to the nearest 5 minutes. If not specified, the roundingPeriod defaults to the same value as shortest period in the job.
- Improve testing / linting (cristiangreco)
- Verify cli parameters and improve cli parsing (a0s)
- Allow to configure yace cli parameters via env variables (a0s)
- Improve error handling of cloudwatch (matthewnolf)
- Add support for directconnect and route53 health checks
- Improve throttling handling to AWS APIs (anilkun)
- Add issue templates to improve support (NickLarsenNZ)
- Allow setting default values for statistics (surminus)
- Fix apigateway method and resouce dimension bug (aleslash)
Thanks a lot to all contributors! - Lovely to see so much efforts especially in testing to get this project more and more stable. - I know we are far away from a nice tested code base but we are improving in the right direction and I really love to see all of your efforts there. It is really appreciated from my side.
I just contacted AWS to get some open source credits so we can build some kind of end to end tests. This shoud allow us to find tricky bugs earlier and not only when we ship things.
Love to all of you, Thomas!
- [BREAKING] Decoupled scraping is now default. Removed code which allowed to use scraper without it.
# Those flags are just ignored
-decoupled-scraping=false
-decoupled-scraping=true
- [BREAKING] Small timeframes of scraping can be used again now. In the past yace decided the scraping interval based on config. This magic was removed for simplicity.
# In the past this would have in some cases still set --scraping-interval 600
--scraping-interval 10
# Now it really would scrape every 10 seconds which could introduce big API costs. So please watch
# your API requests!
--scraping-interval 10
- Fix problems with start/endtime of scrapes (klarrio-dlamb)
- Add support for Database Migration Service metrics
- Allow to hotreload config via /reload (antoniomerlin)
- SECURITY Fix issue with building binaries. Please update to mitigate (https://nvd.nist.gov/vuln/detail/CVE-2020-14039)
- Thanks jeason81 for reporting this security incident!
- BREAKING Introduce new version field to config file (jylitalo)
# Before
discovery:
jobs:
# After
apiVersion: v1alpha1
discovery:
jobs:
- [BUG] Fix issues with nilToZero (eminugurkenar)
- [BUG] Fix race condition setting end time for discovery jobs (cristiangreco)
- Simplify session creation code (jylitalo)
- Major improvement of aws discovery code (jylitalo)
- Major rewrite of the async scraping logic (rabunkosar-dd)
- Add support for AWS/ElasticBeanstalk (andyzasl)
- Upgrade golang to 1.17
- Upgrade golang libraries to newest versions
Okay, private things settled. We have a new organisation for the project. Lets boost it and get the open PRs merged! This version is like 0.28.0-alpha but docker images hosted on ghcr.io and published via new github organisation nerdswords. Find details here.
Thanks to all there waiting and using the product! :)
- BREAKING CHANGE Using a new docker registry / organisation:
# Before
quay.io/invisionag/yet-another-cloudwatch-exporter:v0.29.0-alpha
# Now
ghcr.io/nerdswords/yet-another-cloudwatch-exporter:v0.29.0-alphaSorry folks, I currently struggle a little bit to get things merged fast due to a lot of private stuff. Really appreciate all your PRs and hope to get the bigger ones (which are sadly still not merged yet) into next release.
Really appreciate any person working on this project! - Have a nice day :)
- BREAKING CHANGE Added support for specifying an External ID with IAM role Arns (cristiangreco)
# Before
discovery:
jobs:
- type: rds
roleArns:
- "arn:aws:iam::123456789012:role/Prometheus"
# After
discovery:
jobs:
- type: rds
roles:
- roleArn: "arn:aws:iam::123456789012:role/Prometheus"
externalId: "shared-external-identifier" # optional- Add alias for AWS/Cognito service (tohjustin)
- Fix logic in dimensions for Transit Gateway Attachments (rhys-evans)
- Fix bug with scraping intervals (boazreicher)
- Support arm64 builds (alias-dev)
- Fix IgnoreLength logic (dctrwatson)
- Simplify code base (jylitalo)
- Simplify k8s deployments for new users (mahmoud-abdelhafez)
- Handle metrics with '%' in their name (darora)
- Fix classic elb name (nhinds)
- Skip metrics in edge cases (arvidsnet)
Freshly shipped new integrations:
- Certificate Manager (mksh)
- WorkSpaces (kl4w)
- DDoSProtection / Shield (arvidsnet)
- Make exporter a library. (jeschkies)
- Add CLI option to validate config file (zswanson)
- Fix multidimensional static metric (nmiculinic)
- Fix scrapes running in EKS fail after first scrape (rrusso1982)
- Fix Docker build (jeschkies)
- Allow to use this project in China (insectme)
- Fix error retrieving kafka metrics (friedrichg)
Freshly integrated:
- Add AWS/NetworkFirewall (rhys-evans)
- Add AWS/Cassandra (bjhaid)
- Add AWS/AmazonMQ (saez0pub)
- Add AWS/Athena (haarchri)
- Add AWS/Neptune (benjaminaaron)
Thanks to doc fixes: calvinbui
- Fix CI issue
- BREAKING CHANGE Removed a need to use static dimensions in dynamic jobs in cases, when they cannot be parsed from ARNs (AndrewChubatiuk)
# Before metrics: - name: NumberOfObjects statistics: - Average additionalDimensions: - name: StorageType value: AllStorageTypes # After metrics: - name: NumberOfObjects statistics: - Average
- BREAKING CHANGE Use small case for searchTags config option (AndrewChubatiuk)
# Before searchTags: - Key: type Value: public # After searchTags: - key: type value: public ``` - BREAKING CHANGE CloudFront renamed from
cftocloudfront# Before - type: cf # After - type: cloudfront ```
- Added regular expressions to parse dimensions from resources (AndrewChubatiuk)
- Added option to use floating time windows (zqad)
- Added CLI option to validate config file (zswanson)
- Added AWS network Firewall (rhys-evans)
- Fixed multidimensional static metric (nmiculinic)
- Tidy up code (jylitalo)
- BREAKING CHANGE Use NaN as default if AWS returns nil (arnitolog)
- Add autodiscovery for AWS/EC2Spot (singhjagmohan1000)
- Add autodiscovery for DocumentDB (haarchri)
- Add autodiscovery for GameLift (jp)
- Added support for fips compliant endpoints (smcavallo)
- Update deps and build with golang 1.15 (smcavallo)
- Add API Gateway IAM info to README (Botono)
- Fix sorting of datapoints, add test util functions (Botono)
- Fix missing DataPoints and improve yace in various ways (vishalraina)
- Added Github action file to basic validation of incoming PR (vishalraina)
- Fix info metrics missing (goya)
- Add rds db clusters (goya)
- Fix missing labels (goya)
- Add sampleCount statistics (udhos)
- Add WAFv2 support (mksh)
- Fix alb issues (reddoggad)
- Add nlb support (reddoggad)
- Big tidy up of code, remove old methods and refactor used ones (jylitalo)
- Fix crashes where labels are not collected correctly (rrusso1982)
- Fix pointer bug causing metrics to be missing (jylitalo)
- Allow more then 25 apigateways to be discovered (udhos)
- Add api-gateway support (smcavallo)
- Improve metrics validation (jylitalo)
- Fix metrics with '<', '>' chars
- Remove error during build
Wow what a release. Thanks to all contributors. This is our biggest release and it made me a lot of fun to see all those contributions. From small doc changes (love those) to major rewrites of big components or new complex features. Thanks!
- BREAKING CHANGE Add support for multiple roleArns (jylitalo)
# Before
---
discovery:
jobs:
- type: rds
roleArn: "arn:aws:iam::123456789012:role/Prometheus"
# After
discovery:
jobs:
- type: rds
roleArns:
- "arn:aws:iam::123456789012:role/Prometheus"- Upgrade golang from 1.12 to 1.14
- Major linting of code and improving global code quality. (jylitalo)
- Improve logging (jylitalo)
- Add config validation. (daviddetorres)
- Added support for tags with '@' char included (afroschauer )
- Added Transit Gateway Attachment Metrics (rhys-evans)
- Fix information gathering if no data is retrieved by cloudwatch (daviddetorres)
- Improve docs (calvinbui)
- Add redshift support (smcavallo)
- Allow easier configuration through adding period / addCloudwatchTimestamp setting additionally to job level. (rrusso1982)
- Add initial unit tests (smcavallo)
- Add new configuration to allow snake case labels (rrusso1982)
- Fix complex metric dimension bug (rrusso1982)
- Upgrade golang packages (smcavallo)
- Set up correct partition for ASG for AWS China and GovCloud Regions (smcavallo)
- Add ability to set custom tags to discovery job metrics (goya)
- BREAKING CHANGE Add support for multiple regions (goya)
# Before
---
discovery:
jobs:
- type: rds
region: eu-west-1
# After
discovery:
jobs:
- type: rds
regions:
- eu-west-1- Fix missing alb target group metrics (abhi4890 )
- Added support for step functions (smcavallo)
- Added support for sns / firehose (rhys-evans)
- Added support for fsx / appsync (arnitolog)
- Hugh rewrite: Decouple scraping and serving metrics. Thanks so much daviddetorres!
- BREAKING CHANGE Decoupled scraping and set scraping interval to 5 minutes.
The flag 'decoupled-scraping' makes the exporter to scrape Cloudwatch metrics in background in fixed intervals, in stead of each time that the '/metrics' endpoint is fetched. This protects from the abuse of API requests that can cause extra billing in AWS account. This flag is activated by default.
If the flag 'decoupled-scraping' is activated, the flag 'scraping-interval' defines the seconds between scrapes. Its default value is 300.
- Hugh rewrite: Rewrite of metric gathering to reduce API Limit problems. Thanks so much daviddetorres!
- Improvment of ALB data gathering and filtering (daviddetorres)
- Detect and fix bug after merge (deanrock)
- Add cloudfront support (mentos1386)
- Fixed docker run command in README.md (daviddetorres)
- Added support for Nat Gateway / Transit Gateway / Route 53 Resolver (j-nix)
- Added support for ECS/ContainerInsights (daviddetorres)
- Fix pagination for getMetricList (eminugurkenar)
- Change logging to json format (bheight-Zymergen)
- Add support for kafka (eminugurkenar)
- Add structured json logging (bheight-Zymergen)
- Increase code readability (bheight-Zymergen)
- Fix ecs scraping bug (rabunkosar-dd)
- Fix aws cloudwatch period bug (rabunkosar-dd)
- Fix sts api calls without specifying a region (nhinds)
- Update aws-sdk to v1.25.21 (nhinds)
- Fix github actions (nhinds)
- Update aws-sdk-go (deanrock)
- Avoid appending to a shared dimensions variable from inside a loop (nhinds)
- Remove hardcoded StorageType dimension from S3 metric (nhinds)
- Fix problems and crashes with ALBs and ELBs (Deepak1100)
- BREAKING Changing user in Docker image to be non root to adhere to potential security requirements. (whitlekx)
- Fix prometheus metric bug with new services with '-' e.g. ecs-svc.
- Was accidentally with code from 01.14.0-alpha released.
- BREAKING Default command in Dockerfile is changed to yace. This removes the need to add yace as command.
# Before
command:
- "yace"
- "--config.file=/tmp/config.yml"
# After
args:
- "--config.file=/tmp/config.yml"- Add support for Elastic MapReduce (nhinds)
- Add support for SQS - (alext)
- Add support for ECS Services as ecs-svc
- Add support for NLB
- Add retries to cloudwatch api calls (Deepak1100)
- Fix dimension labels for static jobs (alext)
- Add region as exported label to metrics
- Fix errors with "=" in tags (cdchris12)
- Add curl to container for easy debugging (cdchris12)
- Limit concurrency of aws calls
- Add Autoscaling group support (wjam)
- Fix strange AWS namespace bug for static exports (AWS/EC2/API)
- Add warning if metric length of less than 300s is configured / Interminent metrics
- Fix ALB problems. Target Group metrics are now exported as aws_albtg
aws_albtg_request_count_sum{dimension_LoadBalancer="app/Test-ALB/fec38de4cf0cacb1",dimension_TargetGroup="targetgroup/Test/708ecba11979327b",name="arn:aws:elasticloadbalancing:eu-west-1:237935892384916:targetgroup/Test/708dcba119793234"} 0
- CI problem
- BREAKING For some metrics
cloudwatch:ListMetricsiam permissions are needed. Please update your role! - BREAKING Add 'v' to indicate it is a version number in docker tag / version output
# Before
image: quay.io/invisionag/yet-another-cloudwatch-exporter:0.13.0
# After
image: quay.io/invisionag/yet-another-cloudwatch-exporter:v0.13.0
- Use golang 1.12.0 to build
- Use goreleaser to release
- Update aws dependencies
- Use github actions as CI
- Migrate dependency management to golang modules
- BREAKING For some metrics
cloudwatch:ListMetricsiam permissions are needed. Please update your role! - BREAKING As adding cloudwatch timestamp breaks some metrics I decided to not set it as default anymore.
This should make it easier for new users to have fun with this project.
It fixes for some users
non-histogram and non-summary metrics should not have "_sum" suffixbug.
# Before
metrics:
- name: FreeStorageSpace
disableTimestamp: true
# After
metrics:
- name: FreeStorageSpace
# Before
metrics:
- name: FreeStorageSpace
# After
metrics:
- name: FreeStorageSpace
useCloudwatchTimestamp: true- Add ability to specify additional dimensions on discovery jobs e.g. for BucketSizeBytes metrics on S3 (abuchananTW)
- Fix incorrect dimension value in case of alb in discovery config (GeeksWine)
- Add CLI command to debug output
- Add DynamoDB support
- BREAKING Add the exact timestamps from CloudWatch to the exporter Prometheus metrics (LeePorte)
- Add a new option
disableTimestampto not include a timestamp for a specific metric (it can be useful for sparse metrics, e.g. from S3) (LeePorte) - Add support for kinesis (AndrewChubatiuk)
- BREAKING Add snake_case to prometheus metrics (sanchezpaco)
# Before
aws_elb_requestcount_sum
# After
aws_elb_request_count_sum- Add optional delay setting to scraping (Deepak1100)
period: 60
length: 900
delay: 300- Reduce usage of listMetrics calls (nhinds)
- Add support of iam roles (nhinds)
- Add optional roleArn setting, which allows scraping with different roles e.g. pull data from mulitple AWS accounts using cross-acount roles (nhinds)
metrics:
- name: FreeStorageSpace
roleArn: xxx
statistics:
- 'Sum'
period: 600
length: 60- Add lambda support (nhinds)
- Fix support for listing multiple statistics per metric (nhinds)
- Add tag labels on metrics for easy querying (nhinds)
# Before
aws_ec2_cpuutilization_average + on (name) group_left(tag_Name) aws_ec2_info
# After, now name tags are on metrics and no grouping needed
aws_ec2_cpuutilization_average
- BREAKING Change config syntax. Now you can define tags which are exported as labels on metrics. Before:
discovery:
- region: eu-west-1
type: "es"
searchTags:
- Key: type
Value: ^(easteregg|k8s)$
metrics:
- name: FreeStorageSpace
statistics:
- 'Sum'
period: 600
length: 60New Syntax with optional exportedTagsOnMetrics:
discovery:
exportedTagsOnMetrics:
ec2:
- Name
jobs:
- region: eu-west-1
type: "es"
searchTags:
- Key: type
Value: ^(easteregg|k8s)$
metrics:
- name: FreeStorageSpace
statistics:
- 'Sum'
period: 600
length: 60- Added VPN connection metrics (AndrewChubatiuk)
- Added ExtendedStatistics / percentiles (linefeedse)
- Added Average Statistic (AndrewChubatiuk)
- ALB Support (linefeedse)
- Custom lables for static metrics
Example
static:
- namespace: AWS/AutoScaling
region: eu-west-1
dimensions:
- name: AutoScalingGroupName
value: Test
customTags:
- Key: CustomTag
Value: CustomValue
metrics:
- name: GroupInServiceInstances
statistics:
- 'Minimum'
period: 60
length: 300- Sanitize colons in tags (linefeedse)
- BREAKING: Period/length uses now seconds instead of minutes
- BREAKING: Config file uses new syntax to support static
- Support of --debug flag which outputs some dev debug informations
- Support of metrics who are not included in tags api (e.g. autoscaling metrics)
Before
jobs:
- discovery:
region: eu-west-1
metrics:
- name: HealthyHostCount
statistics:
- 'Minimum'
period: 60
length: 300New Syntax:
discovery:
- region: eu-west-1
type: elb
searchTags:
- Key: KubernetesCluster
Value: production
metrics:
- name: HealthyHostCount
statistics:
- 'Minimum'
period: 60
length: 300
static:
- namespace: AWS/AutoScaling
region: eu-west-1
dimensions:
- name: AutoScalingGroupName
value: Test
metrics:
- name: GroupInServiceInstances
statistics:
- 'Minimum'
period: 60
length: 300- Support of EFS - Elastic File System
- Support of EBS - Elastic Block Storage
- BREAKING: Config file uses list as statistics config option, this should reduce api calls for more than one statistics.
Before:
jobs:
- discovery:
metrics:
statistics: 'Maximum'After:
jobs:
- discovery:
metrics:
statistics:
- 'Maximum'- Start to track changes in CHANGELOG.md
- Better error handling (discordianfish)
- Increase speed, not only each jobs threaded but now each metric
- Add s3 support
- Fix potential race condition during cloudwatch access
- Fix bug ignoring period in cloudwatch config
- Use interfaces for aws access and prepare code for unit tests
- Implement minimum, average, maximum, sum for cloudwatch api
- Implement way to handle multiple data returned by cloudwatch
- Update go dependencies