Release notes on Grafana Labs

Grafana Enterprise Metrics v2.17 release notes

Fri, 16 Oct 2026 14:22:29 +0000

Version 2.17 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.17, which is built on top of Grafana Mimir 2.17.

GEM 2.17 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.17 release. Given this, it’s best to start with the Grafana Mimir 2.17 release notes.

Features and enhancements

GEM 2.17 contains the following new features and enhancements:

Federation-frontend: With cross-cluster query federation, allow automatic retries of individual requests that return retryable errors in queries. You can configure the total number of allowed attempts in the federation.max-attempts-per-request configuration parameter.
Federation-frontend: Add HTTP client cluster validation support through the federation.proxy_targets.cluster_validation_label configuration parameter.
Federation-frontend: Add support for Prometheus remote read API request.
Federation-frontend: You can set the query-frontend.extra-propagated-headers option to specify extra headers allowed to pass through to the rest of the query path.
Gateway: Add optional HTTP load balancing to the cortex.query and cortex.distributor proxies via the [proxy].load-balancing and [proxy].load-balancing-policy configuration parameters.

Upgrade considerations

This release also inherits the upgrade considerations from the Grafana Mimir 2.17 release.

After you upgrade to GEM 2.17, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bug fixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.17.0

Cross-cluster query federation: Fix unreported partial results when the remote-read stream breaks after the first response.
Graphite querier: Improve error messages for render errors.
Ruler: Allow different remotes to re-register wal_watcher metrics without panicking.

v2.17.1

Update root CA certificates in Docker images.
Update to Go v1.24.6 to address CVE-2025-4674, CVE-2025-47907.
Federation-frontend: Correct GEM version requirements in docs.
Ingester: Fix a bug where ingesters would get stuck in read-only mode after compactions.

v2.17.2

Update to Go v1.24.9 to address CVE’s.
Update distroless images to latest digest to address CVE’s.
Distributor: Fix a bug where no NHCB passes validation when native histogram bucket limit -validation.max-native-histogram-buckets is set.
Ingest: Fix a data corruption issue caused by memory pooling when a newer version is rolled out and the experimental -distributor.otel-created-timestamp-zero-ingestion-enabled feature is enabled.
Memcached: Ignore invalid responses when discovering cache servers using dnssrv+ or dnssrvnoa+ service discovery prefixes.

v2.17.3

Update opencontainers/selinux dependency to v1.13.0 to address CVE-2025-52881.
Update Go crypto dependency to v0.43.0 to address CVE-2025-47913.

v2.17.4

Update to Go 1.24.11 to address CVE-2025-61729 and CVE-2025-61727.
Update Go crypto dependency to v0.45.0 to address CVE-2025-47914 and CVE-2025-58181.

v2.17.5

Update Mimir to 2.17.5. For list of fixes, see related release notes in Mimir.
Update to Go v1.25.5 to align with Mimir 2.17 releases.

v2.17.6

Update to Go v1.25.7 to address CVE-2025-61726.
Ruler: Add path traversal checks when parsing namespaces and groups, which prevents namespace and group name from containing non-local paths.

v2.17.7

Federation: Fix nil pointer dereference when querying a non-existent __cluster__ with partial queries enabled.
Federation: Stop emitting cortex_federation_frontend_partial_results metric when partial queries are disabled. Previously when partial queries were disabled, the metric would always report a 0% failure rate, even when cluster requests failed.
Security: Update module golang.org/x/net to v0.45 to address CVE-2025-58190 and CVE-2025-47911.

v2.17.8

Skipping this patch to match the mimir version.

v2.17.9

Distributor: Add OTLP usage reporter that tracks OTel scope occurrences and periodically exports them to the stats server. Enabled via -otlp-usage.enabled.
Ruler: Propagate GCS object mutation rate limit for rule group uploads.
Querier: Fix not_aggregated_wrapper incorrectly return errors for metric with no sample when MQE is in use.
Security: Update to Go v1.25.8 to address CVE-2026-27142, CVE-2026-27139, CVE-2026-25679, CVE-2026-27138, CVE-2026-27137.

v2.17.10

Ruler: Fix parsing of rule expressions with leading newlines.

v2.17.11

Admin API: Fix follower-to-leader proxy failing on IPv6-only clusters due to unbracketed IPv6 address in ring registration.

v2.17.12

Security: Upgrade Mimir to 2.17.10 to address CVE-2026-39883, GHSA-xmrv-pmrh-hhx2.
Security: Upgrade to Go 1.25.9 to address CVE-2026-32283, CVE-2026-32282, CVE-2026-27144, CVE-2026-27143, CVE-2026-27140, CVE-2026-32289, CVE-2026-32288, CVE-2026-32280, CVE-2026-32281.

v2.17.13

Upgrade github.com/go-jose/go-jose/v3 to v3.0.5 to address CVE-2026-34986.

v2.17.14

Security: Upgrade Mimir to 2.17.11 to address CVE-2026-27141, CVE-2026-33814, CVE-2026-39882.
Security: Use Go 1.25.10 to address CVE-2026-33814, CVE-2026-33811, CVE-2026-39836, CVE-2026-42499, CVE-2026-39820, CVE-2026-42501, CVE-2026-39826, CVE-2026-39823, CVE-2026-39817, CVE-2026-39825, CVE-2026-39819.
Build: Use Go 1.26.3 to align with upstream Mimir.
Security: Upgrade Mimir to address CVE-2026-42506, CVE-2026-39833.

v2.17.15

Security: Use Go 1.26.4 to address CVE-2026-42504, CVE-2026-27145 and CVE-2026-42507.
Security: Upgrade base image used for boringcrypto image to address CVE-2026-0915, CVE-2026-4046, CVE-2026-4437, CVE-2025-15281, CVE-2026-0861 and CVE-2026-4438.

v2.17.16

Security: Use Go 1.26.5 to address GO-2026-4970 and GO-2026-5856.
Security: Upgrade go.mongodb.org/mongo-driver to v1.17.7 to address CVE-2026-2303 and GHSA-cp6g-7hqx-qxhp.
Security: Upgrade golang.org/x/net to v0.56.0 and golang.org/x/text to v0.39.0 to address CVE-2026-46600, GO-2026-5942, CVE-2026-56852, and GO-2026-5970.
Security: Upgrade google.golang.org/grpc to v1.82.1 to address GHSA-hrxh-6v49-42gf.

v2.17.17

Security: Upgrade go.opentelemetry.io/otel to v1.44.0 to address CVE-2026-41178.
Ingester: Fix out-of-order queries that span the in-order head mint blocking TSDB head truncation and compaction.

v2.17.18

Ingester: Optimize TSDB head-chunk access during compaction to avoid O(n²) cost in PopulateBlock.

v2.17.19

Security: Use Go 1.26.7 to address GO-2026-5026, GO-2026-5942, GO-2026-5972, GO-2026-6088, GO-2026-6089, GO-2026-6090, GO-2026-6091, and GO-2026-6218.
Security: Upgrade Mimir to address CVE-2026-56865, GHSA-259r-337f-4rfw, and CVE-2026-73500.

v2.17.20

Security: Fix an issue where configuration that referenced files could load local files. Resolves CVE-2026-81938.

v2.17.21

Security: Use Go 1.26.9 to address GO-2026-6599, GO-2026-6600, GO-2026-6603, GO-2026-6605, GO-2026-6607, GO-2026-6608, GO-2026-6609, GO-2026-6610, GO-2026-6611, GO-2026-6612, GO-2026-6613, and GO-2026-6617.
Security: Upgrade Mimir to address CVE-2026-81872, CVE-2026-81871, and CVE-2026-81870.
Fix an issue where configuring S3 with signature_version2: true would still result in using AWS Signature Version 4.

Grafana Enterprise Metrics v2.16 release notes

Tue, 28 Oct 2025 18:04:58 +0000

Version 2.16 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.16, which is built on top of Grafana Mimir 2.16.

GEM 2.16 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.16 release. Given this, it’s best to start with the Grafana Mimir 2.16 release notes.

Features and enhancements

Grafana Enterprise Metrics 2.16 adds support for partial queries to cross-cluster query federation. It also including the following changes:

Cross-cluster query federation: increase default proxy target timeout to two minutes. This brings it into line with other similar timeouts such as -querier.timeout.

Upgrade considerations

This release also inherits the upgrade considerations from the Grafana Mimir 2.16 release.

After you upgrade to GEM 2.16, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bug fixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Deprecations

Admin API: Deprecate -admin-api.limits.enabled and default to false in preparation for removal. You should change this to true if you want to continue using the Admin API for limits management.
Remove experimental Datadog proxy.

Bug fixes

Cross-cluster query federation: querying exemplars would fail with a misleading no cluster name provided error. It will now return cross-cluster query federation does not support querying exemplars.

v2.16.0

Update base image for grafana/enterprise-metrics-alpine image to alpine:3.21.2.
Remove the images grafana/enterprise-metrics-alpine, grafana/enterprise-metrics-boringcrypto-alpine.

v2.16.1

Update to Go v1.23.9 to address CVE-2025-22871.
Update golang.org/x/net to v0.38.0 to address CVE-2025-22872.
Query-frontend: Fix a panic in monolithic mode caused by a clash in labels of the cortex_client_invalid_cluster_validation_label_requests_total metric definition.

v2.16.2

Update Go from v1.23.9 to v1.23.12 to address CVEs.
Update Alpine from v3.21.3 to v3.21.5.

Grafana Enterprise Metrics v2.15 release notes

Tue, 28 Oct 2025 18:04:58 +0000

Version 2.15 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.15, which is built on top of Grafana Mimir 2.15.

GEM 2.15 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.15 release. Given this, it’s best to start with the Grafana Mimir 2.15 release notes.

Features and enhancements

This release introduces changes to the federation-frontend, which now requires GEM clusters as proxy targets and mandates upgrading to GEM 2.15+ before updating the frontend. It also updates the Ruler to handle missing rule groups gracefully and resolves several bugs, including fixes for cross-cluster query federation label handling, token creation ETag responses, and critical security vulnerabilities in go-jose and go-oidc.

Upgrade considerations

This release also inherits the upgrade considerations from the Grafana Mimir 2.15 release.

After you upgrade to GEM 2.15, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bug fixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.15.0

Include the initial ETag version in the token creation response when using the Admin API
When using cross-cluster query federation, series with the label name __cluster__ were incorrectly handled in shardable queries, which are queries that can be divided up and sent to different clusters and then combined later. This fixes that issue and allows users to use original___cluster__ as a label matcher to match that label and __cluster__ to match the cluster they defined, making it consistent with the behavior of other kinds of queries. This applies to aggregation grouping labels as well.
Upgrade dependencies to address CVE-2024-28180.
Upgrade dependencies to address CVE-2024-28180.

v2.15.1

Upgrade Go base image to 1.23.6.
Update base image to alpine:3.20.5.
Upgrade to github.com/golang/glog to v1.2.4 to resolve CVE-2024-45339.

v2.15.2

Update module golang.org/x/net to v0.36.0 to address CVE-2025-22870.
Update module github.com/golang-jwt/jwt/v5 to v5.2.2 to address CVE-2025-30204.
Update module github.com/golang-jwt/jwt/v4 to v4.2.2 to address CVE-2025-30204.
Update module github.com/golang-jwt/jwt/v3 to v3.0.4 to address CVE-2025-30204.
Update module golang.org/x/oauth2 to v0.27.0 to address CVE-2025-22868.
Update module golang.org/x/crypto to v0.35.0 to address CVE-2025-22869.
Update module github.com/go-jose/go-jose/v4 to v4.0.5 to address CVE-2025-27144.
Upgrade Go base image to 1.23.7.

Grafana Enterprise Metrics v2.14 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.14 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.14, which is built on top of Grafana Mimir 2.14.

GEM 2.14 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.14 release. Given this, it’s best to start with the Grafana Mimir 2.14 release notes.

Features and enhancements

Grafana Enterprise Metrics 2.14 adds the following new configurations:

Configuration to set timeout for query request and label names and values requests in Cross Cluster query federation.
Internal gateway gRPC proxy clients can now have their maximum send and receive size limits configured via proxy.grpc-max-recv-msg-size and proxy.grpc-max-send-msg-size config flags.
Two new command line options, -auth.pass-access-policy-name and -auth.pass-token-name, that enable sending the query author’s access policy name and token name to the downstream query services with request headers. The GEM query-frontend can choose to log the X-Access-Policy-Name and X-Token-Name headers in the query statistics logs, via its query-frontend.log-query-request-headers config flag.

Upgrade considerations

This release also inherits the upgrade considerations from the Grafana Mimir 2.14 release.

After you upgrade to GEM 2.14, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Deprecations

The experimental support for Redis caching is now deprecated and set to be removed in the next major release. You are encouraged to switch to Memcached.

Bug fixes

v2.14.0

Fix "name" tag setting in various functions, including aliasSub, aliasByNode in Graphite querier.
Return 404 when fetching unknown tokens via admin API instead of 500.

Grafana Enterprise Metrics v2.13 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.13 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.13, which is built on top of Grafana Mimir 2.13.

GEM 2.13 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.13 release. Given this, it’s best to start with the Grafana Mimir 2.13 release notes.

Features and enhancements

GEM has implemented enhanced usage statistics for more accurate reporting.

GEM has followed suit with Mimir 2.13 by limiting Go’s GOMAXPROCS setting to be no more than NumCPU to reduce scheduler overhead.

The grafana/enterprise-metrics docker image is now based on gcr.io/distroless/static-debian12 image. The Alpine-based docker image is still available as grafana/enterprise-metrics-alpine.

Upgrade considerations

With Grafana Mimir 2.13, mimir-continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The configuration option -log.buffered, deprecated in Grafana Mimir 2.11, has now been removed from Mimir 2.13. Logs are now always written in a buffered fashion.

Bug fixes

v2.13.3

Update build Docker image to use Go 1.22.10.
Upgrade to github.com/golang-jwt/jwt/v4 v4.5.1 to resolve CVE-2024-51744.
Upgrade to github.com/gorilla/schema v1.4.1 to resolve CVE-2024-37298.
Upgrade to github.com/go-jose/go-jose/v3 v3.0.3 to resolve CVE-2024-28180.
Upgrade to github.com/docker/cli v27.3.1 to resolve CVE-2024-41110.

v2.13.2

Updated base image for grafana/enterprise-metrics-alpine image to alpine:3.20.3.

v2.13.1

Cross-cluster query federation: fix issue where time range for label names and label values requests was not propagated to remote targets.
Cross-cluster query federation: log failed requests to remote targets.
Active series: the default value for active series idle timeout (-ingester.active-series-metrics-idle-timeout) increased from 10m to 20m.
Active series: active series metrics must now be enabled and their update interval must be set to 20m; otherwise, GEM fails to start.

Grafana Enterprise Metrics v2.12 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.12 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.12, which is built on top of Grafana Mimir 2.12.

GEM 2.12 inherits all the features, enhancements, and bug fixes that are in the Grafana Mimir 2.12 release. Given this, it’s best to start with the Grafana Mimir 2.12 release notes.

Features and enhancements

Distroless images have been introduced to minimize the amount of software that may require security patches in release images. The distroless images are based on gcr.io/distroless/static-debian12 and can be found at grafana/enterprise-metrics-distroless and grafana/enterprise-metrics-boringcrypto-distroless.

Graphite query support includes the following improvements:

You can disable the graphite querier rate limits by setting the value for the option to 0. The options you can set are:
- -graphite.querier.rate-limit-qps
- -graphite.querier.tenant-rate-limit-qps
- -graphite.querier.heavy-rate-limit-qps
You can use an experimental flag -graphite.querier.deduplicate-samples-in-raw-interval to keep only the last sample in the raw interval of a series. This attempts to mimic Graphite’s behavior of overwriting samples of the raw series, instead of aggregating them together.

Upgrade considerations

In Grafana Enterprise Metrics 2.12 the following behavior has changed:

Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed in Grafana Enterprise Metrics 2.12:

-blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
-query-frontend.max-cache-freshness from 1m to 10m.
-distributor.write-requests-buffer-pooling-enabled from false to true.
-locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
-memberlist.stream-timeout from 10s to 2s.
-server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Enterprise Metrics 2.12:

The YAML setting frontend.cache_unaligned_requests.
Experimental CLI flag -querier.prefer-streaming-chunks-from-ingesters.

The following configuration options are deprecated and will be removed in Grafana Enterprise Metrics 2.14:

The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter. It now defaults to true.
The CLI flag -ingester.return-only-grpc-errors. It now defaults to true.

Warning
To guarantee backwards compatibility when migrating from a version prior to 2.11, it’s necessary to first migrate to version 2.11, and then to version 2.12. Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won’t be recognized, and the corresponding request will be repeated.
The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled. It now defaults to true.
The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter. It now defaults to true.
The CLI flag -distributor.enable-otlp-metadata-storage. It now defaults to true.
The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated in Grafana Enterprise Metrics 2.12:

cortex_bucket_store_blocks_loaded_by_duration has been removed.
cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Grafana Enterprise Metrics 2.14.

This release also inherits the upgrade considerations from the Grafana Mimir 2.12 release.

After you upgrade to GEM 2.12, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.12.0

Update Alpine to 3.18.5 to address CVE-2023-5363.
Update Golang to 1.21.5 to address CVE-2023-39326.
LBAC: caching for /prometheus/api/v1/label/* endpoints was not respecting LBAC policies leading to sharing cached responses for different LBAC policies within the same tenant.

v2.12.1

Update Alpine to 3.18.8

Grafana Enterprise Metrics v2.11 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.11 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.11, which is built on top of Grafana Mimir 2.11.

GEM 2.11 inherits all the features, enhancements, and bugfixes that are in the Grafana Mimir 2.11 release. Given this, it’s best to start with the Grafana Mimir 2.11 release notes.

Features and enhancements

Audit log sampling can now be set to a decimal value, which allows sampling rates below 1%. For example, setting admin-api.auditlogging.sample-rate to 0.1 means sampling of 0.1%.

Authentication can now be disabled for debugging endpoints via the configuration flag -auth.required-for-debug. The default behavior of requiring auth has not changed.

Graphite Query support has been enhanced to include the following:

The tag details endpoint now supports the from and to parameters to control the queried time range.
The new flag -graphite.querier.max-query-length limits the largest time range allowed for queries. This is useful to prevent out of memory errors when processing very large queries.
The value -1 is now supported for maxDataPoints in order to disable maxDataPoints consolidation in the /render endpoint. This brings the behavior closer in line with native Graphite.

Upgrade considerations

In Grafana Enterprise Metrics 2.11 the following behavior has changed:

Support for AWS Marketplace licenses has been removed along with the cli option -license.type=aws-marketplace.
The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Enterprise Metrics 2.11:

The CLI flag -querier.iterators.
The CLI flag -query.batch-iterators.
The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Enterprise Metrics 2.13:

The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

This release also inherits the upgrade considerations from the Grafana Mimir 2.11 release.

After you upgrade to GEM 2.11, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.11.0

Failed release process. See v2.11.1.

v2.11.1

Network errors are now responded with 598 status code instead of 504 (originally 408).
Update base image to alpine:3.18.4
Fix federation-frontend panic because of nil ExemplarQueryable
Fix execution: attempted to read series at index X from ingester chunks stream, but the stream has series with index Y errors when multiple LBAC label selectors apply to a request and ingester to querier or store-gateway to querier chunks streaming is enabled.

v2.11.2

Update Alpine to 3.18.5 to address CVE-2023-5363.
Built using the latest version of Go (1.21.8) to address:

Grafana Enterprise Metrics v2.10 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.10 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.10, which is built on top of Grafana Mimir 2.10.

GEM 2.10 inherits all the features, enhancements, and bugfixes that are in the Grafana Mimir 2.9 release. Given this, it’s best to start with the Grafana Mimir 2.10 release notes.

Features and enhancements

Major enhancement in this release is the optimization of internal representation of labels in memory, which can lead to 15% reduction of memory usage by GEM ingesters, depending on the pattern of labels used, number of tenants, etc.

Upgrade considerations

There are no breaking changes in this release of GEM.

This release also inherits the upgrade considerations from the Grafana Mimir 2.10 release.

After you upgrade to GEM 2.10, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.10.0

Failed release process. See v2.10.1.

v2.10.1

Graphite querier: return 499s instead of 500s when clients cancel requests in certain scenarios.
Graphite querier: correctly support setting SplitQueriesByInterval to 0.
Graphite querier: fixes a JSON serialization bug when a returned value is infinity.
Graphite querier: fixes handling of holtWinters functions, that are now proxied to Graphite Web.
Graphite querier: Updated rollup aggregation to aggregate NaN or empty sample lists to NaN instead of zero.
Graphite querier: wildcard matchers in patterns are now correctly honored, and series with less nodes than queried are no longer returned as part of the result.
Graphite querier: CarbonAPI’s groupByNode function implementation now returns the expected number of series.
Graphite querier: CarbonAPI’s holtWintersConfidenceBands and holtWintersForecast functions implementation now return correct results.
Graphite querier: CarbonAPI’s moving* functions implementation now returns correct results.
Graphite querier: CarbonAPI’s exponentialMovingAverage functions implementation now returns correct results.
Graphite querier: filter out series with fewer nodes than queried when path expressions end with glob patterns.
Graphite querier: handle remote querier disconnections more gracefully, avoiding internal errors in certain scenarios.
Admin client: Fix not invalidating in-memory cache sooner. If admin client cache is in-memory, then revert to the original maximum 1 minute cache expiration before -admin.client.cache.expiration was introduced.
Admin client: Fix ignoring -admin-api.leader-election.client configuration parameters.

v2.10.2

GEM: update dependencies to fix CVE-2023-44487.

v2.10.3

GEM: update dependencies to fix concurrency issue introduced in v2.10.2.

v2.10.4

GEM: update dependencies to fix CVE-2023-45142.
Admin client: Fix memory leak.

v2.10.5

Update Alpine to 3.18.5 to address CVE-2023-5363.
Update Golang to 1.21.5 to address CVE-2023-39326.

v2.10.6

Built using the latest version of Go (1.21.8) to address:

Grafana Enterprise Metrics v2.9 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.9 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.9, which is built on top of Grafana Mimir 2.9.

GEM 2.9 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.9 release. Given this, it’s best to start with the Grafana Mimir 2.9 release notes.

Features and enhancements

The admin API now has experimental support for audit logging of requests. More information is available in the documentation.
We’ve added arm64 Docker images for GEM to enable organizations to run GEM on ARM-based machines. These images are available on the Enterprise Metrics Docker Registry, as described on the Grafana Enterprise Metrics Downloads page. In our own initial internal tests, highlighted in our GrafanaCon 2023 presentation at 15:55, we saw a cost reduction of 20% when comparing GEM on ARM to GEM on x86 with no noticeable performance difference.

Upgrade considerations

There are no breaking changes in this release of GEM.

This release also inherits the upgrade considerations from the Grafana Mimir 2.9 release.

After you upgrade to GEM 2.9, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.9.0

Graphite querier: count function in summarize now correctly only counts non-NaN values.
Graphite querier: smartSummarize is now correctly processed by CarbonAPI and not proxied to stock Graphite Web.
Graphite querier: hitcount is now correctly processed by CarbonAPI and not proxied to stock Graphite Web.
Graphite querier: certain bad requests no longer return 500 errors.
If receiving the request body times out, the gRPC proxy will fail the write request immediately and not update the cortex_request_downstream_duration_seconds metric.
Ruler: fix performance of max rule group enforcement when per-tenant max rule group limits are disabled.

v2.9.1

GEM: update dependencies to fix CVE-2023-44487.

v2.9.2

GEM: update dependencies to fix concurrency issue introduced in v2.9.1.

v2.9.3

GEM: update dependencies to fix CVE-2023-45142.
Admin client: Fix memory leak.

v2.9.4

Update Alpine to 3.18.5 to address CVE-2023-5363.
Update Golang to 1.20.12 to address CVE-2023-39326.

Grafana Enterprise Metrics v2.8 release notes

Mon, 18 Aug 2025 18:19:44 +0000

Version 2.8 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.8, which is built on top of Grafana Mimir 2.8.

GEM 2.8 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.8 release. Given this, it’s best to start with the Grafana Mimir 2.8 release notes.

Features and enhancements

All base Docker images were updated from alpine:3.17.2 to alpine:3.17.3.

Upgrade considerations

There are no breaking changes in this release of GEM.

This release also inherits the upgrade considerations from the Grafana Mimir 2.8 release.

After you upgrade to GEM 2.8, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.8.0

Fixed a bug where setting location in the alertmanager configuration caused the loading of the configuration to fail.
Fixed an error parsing ruler.query_frontend configuration block for GEM configuration.
Updated Go version to 1.20.4 to fix CVE-2023-24538, CVE-2023-24539, CVE-2023-24540, CVE-2023-29400.

v2.8.1

Ruler: fix performance of max rule group enforcement when per-tenant max rule group limits are disabled.

Grafana Enterprise Metrics v2.7 release notes

Fri, 17 Mar 2023 00:00:00 +0000

Version 2.7 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.7, which is built on top of Grafana Mimir 2.7.

GEM 2.7 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.7 release. Given this, it’s best to start with the Grafana Mimir 2.7 release notes.

Features and enhancements

Remote cache support for the object storage client was added. New flags (with respective YAML configuration options):
- -admin.client.cache.expiration
- -admin.client.cache.type
- -admin.client.cache.memcached.addresses
- -admin.client.cache.memcached.max-async-buffer-size
- -admin.client.cache.memcached.max-async-concurrency
- -admin.client.cache.memcached.max-get-multi-batch-size
- -admin.client.cache.memcached.max-get-multi-concurrency
- -admin.client.cache.memcached.max-idle-connections
- -admin.client.cache.memcached.max-item-size
- -admin.client.cache.memcached.timeout
- -admin.client.cache.redis.endpoint
- -admin.client.cache.redis.username
- -admin.client.cache.redis.password
- -admin.client.cache.redis.db
- -admin.client.cache.redis.master-name
- -admin.client.cache.redis.connection-pool-size
- -admin.client.cache.redis.dial-timeout
- -admin.client.cache.redis.idle-timeout
- -admin.client.cache.redis.read-timeout
- -admin.client.cache.redis.write-timeout
- -admin.client.cache.redis.min-idle-connections
- -admin.client.cache.redis.max-connection-age
- -admin.client.cache.redis.max-async-buffer-size
- -admin.client.cache.redis.max-async-concurrency
- -admin.client.cache.redis.max-get-multi-batch-size
- -admin.client.cache.redis.max-get-multi-concurrency
- -admin.client.cache.redis.max-item-size
- -admin.client.cache.redis.tls-enabled
- -admin.client.cache.redis.tls-ca-path
- -admin.client.cache.redis.tls-cert-path
- -admin.client.cache.redis.tls-key-path
- -admin.client.cache.redis.tls-server-name
- -admin.client.cache.redis.tls-insecure-skip-verify
- -admin.client.cache.redis.tls-min-version
- -admin.client.cache.redis.tls-cipher-suites
- -admin.client.cache.refresh-interval
All base Docker images were updated from alpine:3.17.1 to alpine:3.17.2.

Upgrade considerations

There are no breaking changes in this release of GEM.

This release also inherits the upgrade considerations from the Grafana Mimir 2.7 release.

After you upgrade to GEM 2.7, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Grafana Enterprise Metrics v2.6 release notes

Wed, 22 Feb 2023 00:00:00 +0000

Version 2.6 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.6, which is built on top of Grafana Mimir 2.6.

GEM 2.6 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.6 release. Given this, it’s best to start with the Grafana Mimir 2.6 release notes.

Features and enhancements

The Graphite Querier has experimental support for optional rate limiting at the subquery level. This can be set by the .max-concurrent-sub-queries-per-request flag.
The Graphite Querier storage aggregation method set in storage-aggregation.conf can no longer be overridden during runtime using consolidateBy when metrictank is used as a render engine for remote queries.
The Graphite Querier endpoint /tags/autoComplete/values now takes input time range into account. Previously only tag values in the last hour were returned.
Similarly, the Graphite Querier now respects to/until parameters for /find and /tags endpoints. Though if to/until is greater than the current time, it iss adjusted to the current time. Previously this value was always overwritten by the current time
The Graphite Querier now supports the /metrics/expand endpoint.
The number of requests to object storage has been improved in the versioned bucket client.
All base container images have been updated from alpine:3.16.2 to alpine:3.17.1.

Helm

The mimir-distributed Helm chart is now release separately from GEM. See the Grafana mimir-distributed Helm chart documentation for more information including release notes for when 2.6 is included.

Upgrade considerations

There are no breaking changes in this release of GEM

This release also inherits the upgrade considerations from the Grafana Mimir 2.6 release.

After you upgrade to GEM 2.6, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Grafana Enterprise Metrics v2.5 release notes

Thu, 15 Dec 2022 00:00:00 +0000

Version 2.5 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.5, which is built on top of Grafana Mimir 2.5.

GEM 2.5 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.5 release. Given this, it’s best to start with the Grafana Mimir 2.5 release notes.

Features and enhancements

TLS is now more configurable for both internal and external connections
- Added -*.tls-min-version to configure the minimum TLS version
- Added -*.tls-cipher-suites to allow overriding the default cipher suites provided by the Go runtime
GEM Self Monitoring cpu metrics now include a clustername label to distinguish multiple installations of Grafana Enterprise Metrics, Grafana Enterprise Logs and Grafana Enterprise Traces.
- cortex_quota_cpu_count
- cortex_quota_gomaxprocs
- cortex_quota_cgroup_cpu_max
- cortex_quota_cgroup_cpu_period
GEM Self Monitoring bundles 16 additional recording rules from the Grafana Mimir monitoring mixin.
- target:cortex_ingester_queried_exemplars:99quantile
- target:cortex_ingester_queried_exemplars:50quantile
- target:cortex_ingester_queried_exemplars:avg
- target:cortex_ingester_queried_exemplars_bucket:sum_rate
- target:cortex_ingester_queried_exemplars_sum:sum_rate
- target:cortex_ingester_queried_exemplars_count:sum_rate
- target_instance:cortex_alertmanager_alerts:sum
- target_instance:cortex_alertmanager_silences:sum
- target:cortex_alertmanager_state_replication_total:rate5m
- target:cortex_alertmanager_state_replication_failed_total:rate5m
- cortex_alertmanager_alerts_invalid_total:rate5m
- target:cortex_alertmanager_alerts_received_total:rate5m
- target:cortex_alertmanager_partial_state_merges_total:rate5m
- target:cortex_alertmanager_partial_state_merges_failed_total:rate5m
- target_integration:cortex_alertmanager_notifications_total:rate5m
- target_integration:cortex_alertmanager_notifications_failed_total:rate5m
The object storage parameters -*.azure.msi-resource are now ignored, and will be removed in GEM 2.7. This setting is now chosen automatically by Azure.
The Graphite Querier’s default cache TTL has been lowered to 10 minutes. This ensures consistent query results when out-of-order ingestion is enabled.
The storage aggregation method set for the Graphite Querier in storage-aggregation.conf can no longer be overridden during runtime using consolidateBy when metrictank is used as a render engine. This matches Graphite-Web’s behavior.
Fixed a bug in the Graphite Querier where render requests that failed to be processed by the native engine were not being proxied to Graphite-Web.

Helm changes

The mimir-distributed Helm chart is the best way to install GEM on Kubernetes. Notable changes follow. For the full list of changes, see the Helm chart changelog.

Zone aware replication Helm now supports deploying the ingesters and store-gateways as different availability zones. The replication is also zone-aware, therefore multiple instances of one zone can fail without any service interruption and roll outs can be performed faster because many instances of each zone can be restarted together, as opposed to them all restarting in sequence.

This is a breaking change, for details on how to upgrade please review the Helm changelog.

This release also inherits the helm changes from the Grafana Mimir 2.5 release.

Upgrade considerations

There are no breaking changes in this release of GEM
If you are deploying via Helm, note the Helm changes above.

This release also inherits the upgrade considerations from the Grafana Mimir 2.5 release.

After you upgrade to GEM 2.5, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.5.1

Fixed empty buildinfo in GEM binary

v2.5.2

Fix issue where authentication caches were not sized correctly resulting in poor performance

Grafana Enterprise Metrics v2.4 release notes

Mon, 14 Nov 2022 00:00:00 +0000

Version 2.4 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.4, which is built on top of Grafana Mimir 2.4.

GEM 2.4 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.4 release. Given this, it’s best to start with the Grafana Mimir 2.4 release notes.

Note: Mimir 2.4, and by extension GEM 2.4, includes a system that optionally and anonymously reports non-sensitive, non-personally identifiable information about the running GEM cluster to a remote statistics server. This feature is turned on by default. For more information about the type of data that is collected, refer to Which information is collected. To opt out, refer to Disable the anonymous usage statistics reporting.

Features and enhancements

Fixed CVE-2022-44643
CarbonAPI is now being used instead of MetricTank as the default native query engine for the Graphite querier. This can be changed by setting -graphite.querier.querier-carbonapi-rollout-fraction=0.
Added optional rate limiting capabilities to the Graphite querier. This can be configured using the flags -graphite.querier.rate-limit-enabled, -graphite.querier.rate-limit-qps, -graphite.querier.tenant-rate-limit-qps and -graphite.querier.heavy-rate-limit-qps.
Added a new flag -graphite.querier.cache-ttl to the Graphite querier to configure the TTL of cached metric names and aggregation configs.

Helm changes

The mimir-distributed Helm chart is the best way to install GEM on Kubernetes. Notable changes follow. For the full list of changes, see the Helm chart changelog.

Ingresses for the GEM gateway and nginx will no longer render on Kubernetes versions <1.19
Added support for topologySpreadConstraints to all components, including the GEM gateway, admin-api, and alertmanager, which did not have podAntiAffinity previously.

Upgrade considerations

The GEM docker image no longer requests the CAP_NET_BIND_SERVICE capability because the default HTTP port was changed from 80 to 8080.
- If you set -server.http-listen-port or -server.grpc-listen-port to a value that is lower than 1024, then you need to modify your configuration:
  - When using Docker, provide the flag --cap-add net_bind_service.
  - When using the mimir-distributed Helm chart, make sure that all the GEM components have the following additional securityContext setting in their respective values file sections:
    YAML
    securityContext: sysctls: - name: net.ipv4.ip_unprivileged_port_start value: "0" # or lowest http/grpc listen port number

This release also inherits the upgrade considerations from the Grafana Mimir 2.4 release.

After you upgrade to GEM 2.4, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.4.1

Fix issue where authentication caches were not sized correctly resulting in poor performance

Grafana Enterprise Metrics v2.3 release notes

Wed, 28 Sep 2022 00:00:00 +0000

Version 2.3 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.3, which is built on top of Grafana Mimir 2.3.

GEM 2.3 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.3 release. Given this, it’s best to start with the Grafana Mimir 2.3 release notes.

Features and enhancements

The Gateway component now exposes TSDB block upload routes to facilitate the experimental backfilling feature introduced in Grafana Mimir 2.3. To learn more about the backfill feature, please see mimirtool backfill and Configure TSDB block upload.
The new common YAML config option key (or -common.storage.* CLI flags) for setting object storage for multiple components now also apply to the Admin Client.

Helm changes

It is now possible to enable the Graphite Proxy.
The gateway service now opens up port 80 as well as 8080. Port 8080 is deprecated and will be removed by default in version 6.0 of the Helm chart.

Upgrade considerations

The Gateway component’s default dial timeout has been changed to 5s from 30s. This is configurable with the new CLI flags of the form -gateway.proxy.*.dial-timeout (and their respective dial_timeout YAML options).

This release also inherits the breaking changes made in the Grafana Mimir 2.3 release. See the Upgrade considerations section of the Grafana Mimir 2.3 release notes for details.

After you upgrade to GEM 2.3, upgrade your GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes in the GEM plugin, see the Grafana Enterprise Metrics: Changelog.

Bug fixes

v2.3.1

Fixed CVE-2022-44643

v2.3.2

Fix issue where authentication caches were not sized correctly resulting in poor performance

Grafana Enterprise Metrics v2.2 release notes

Thu, 21 Jul 2022 00:00:00 +0000

Version 2.2 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.2, which is built on top of Grafana Mimir 2.2. This means that GEM 2.2 inherits all of the features, enhancements, and bugfixes that are in the Grafana Mimir 2.2 release. Given this, it’s best to start with the Grafana Mimir 2.2 release notes.

For the full list of changes in GEM 2.2, see the Changelog.

Upgrade considerations

All API endpoints that are under /api/v1/rules* and /prometheus/rules* have been removed from the GEM ruler component in favor of identical endpoints that use the prefix /prometheus/config/v1/rules*. For Mimir, this is a breaking change; if you use GEM with the GEM gateway, this is not a breaking change because the gateway has been updated to reroute requests from the deleted endpoints to the new endpoints.
If you are using the Graphite querier, the remote subquerier is no longer optional. Consequently, the following CLI flags (and their respective YAML configuration options) have been removed: -graphite.querier.enable-remote-subquerier and -graphite.querier.use-remote-results.

We’ve released a new version of our Helm chart at the same time as the GEM 2.2 release. There are breaking changes associated with the Helm chart upgrade. All of these breaking changes are also relevant to you if you use the Helm chart to install Grafana Mimir. Rather than duplicating the documentation about the breaking changes here, see the Upgrade considerations section of the Mimir release notes for details. Alternatively, refer directly to Upgrade the Grafana Mimir Helm chart from version 2.1 to 3.0.

After you upgrade to GEM 2.2, upgrade your Grafana Enterprise GEM plugin to the latest version. For more information about the most recent enhancements and bugfixes, see the Grafana Enterprise Metrics: Changelog.

Grafana Enterprise Metrics v2.1 release notes

Thu, 02 Jun 2022 00:00:00 +0000

Version 2.1 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.1. GEM 2.1 is built on top of Grafana Mimir 2.1, which means it inherits all its features, enhancements, and bugfixes. Given this, we recommend you start by reading the Grafana Mimir 2.1 release notes.

Below, we highlight the most important features, enhancements, and bugfixes specific to GEM 2.1. For the full list of changes, see the Changelog.

Features and enhancements

The following list highlights the features and enhancements in Grafana Enterprise Metrics 2.1:

Remote ruler mode for improved rule evaluation performance: This experimental feature is inherited from Grafana Mimir 2.1 and is explained in more detail in the Mimir 2.1 release notes. GEM 2.1 adds an additional configuration parameter, -ruler.query-frontend.auth-token, which is required for configuring the remote ruler with GEM enterprise authentication. See Remote rule evaluation for more information.
Page fault metrics for self-monitoring: GEM now collects and stores metrics tracking the number of page faults and out-of-memory kills it experiences, making it easier to debug performance issues.

Upgrade considerations

We’ve released a new version of our Helm chart in tandem with the GEM 2.1 release. The upgrade for users of the Helm chart has no breaking changes.

Once you upgrade to GEM 2.1, we recommend you upgrade your Grafana Enterprise GEM plugin to the latest version. For more information on the most recent enhancements and fixes, see the plugin’s changelog page.

Bug fixes

v2.1.0

Fixed a bug where ruler limits (e.g., maximum number of rules in a rule group or maximum number of rule groups per tenant) were not enforced when creating a rule group via GEM’s ruler API. These limits are now enforced.
GEM now returns a generic error message to clients (e.g., the Grafana Agent) upon internal errors during authentication. The detailed error message is now written to the GEM logs rather than being returned to the client.

Grafana Enterprise Metrics v2.0 release notes

Wed, 13 Apr 2022 00:00:00 +0000

Version 2.0 release notes

Grafana Labs is excited to announce the release of Grafana Enterprise Metrics (GEM) 2.0. GEM 2.0 is built on top of Grafana Mimir 2.0, which is the most scalable, most performant open source time series database in the world. To learn more about the Grafana Mimir project, see our announcement blog post as well as the Grafana Mimir documentation.

GEM 2.0 inherits all of the highlights of Grafana Mimir 2.0, including easy deployment, native multi-tenancy, high availability, durable long-term metrics storage, and exceptional query performance. GEM includes additional functionality that enterprise users need to operate a time series database at their organization, including simplified tenant management, fine-grained access controls, and out-of-the box self-monitoring and usage data.

GEM 2.0 is a major version upgrade from the previous version of GEM, 1.7, and represents a major improvement in terms of usability and functionality over its predecessor. However, some of these improvements introduce breaking changes for those upgrading from GEM 1.7. We did our best to weigh the tradeoffs between the one-time pain and long-term benefits, such as a simplified configuration experience, and reduced maintenance burden, when deciding which breaking changes to introduce. For more information about upgrading, see Upgrade Considerations.

Although GEM releases have historically been based on an arbitrary commit hash of Cortex, going forward we will base GEM releases off of a named release of Grafana Mimir. Doing so simplifies the process of understanding what changes from Grafana Mimir are present in GEM.

Features and enhancements

The majority of features and enhancements in GEM 2.0 are inherited from Grafana Mimir 2.0. Therefore, start by reading about the features and enhancements in the Grafana Mimir 2.0 release notes, as well as the Grafana Mimir 2.0 documentation.

The following list highlights the features and enhancements in Grafana Enterprise Metrics 2.0:

The following features were released in Grafana Mimir 2.0, but have some GEM 2.0 specific implications:
- GEM 2.0 inherits Grafana Mimir 2.0’s experimental support for cross-tenant alerting and recording rules. GEM 2.0 seamlessly integrates with GEM’s access control model to ensure that recording and alerting rules can only be created and executed after proper authorization. For more information, see Cross tenant alerting and recording rule federation.
- In Grafana Mimir 2.0, we open sourced and promoted to stable the split-and-merge compaction algorithm that was initially released as an experimental closed-source feature in GEM 1.6. Both Grafana Mimir 2.0 and GEM 2.0 utilize the split-and-merge compaction strategy by default. The closed-source time-sharding compaction strategy introduced in GEM 1.4 has been removed because it is redundant. The split-and-merge compactor contains the vertical scaling capability the time-sharding algorithm introduced and adds horizontal scaling as well. For more information on how to transition from the time-sharded compactor to the split-and-merge compactor, see the migration guide. To learn more about the new split-and-merge compactor, see Compaction in the Grafana Mimir 2.0 docs.
- In Grafana Mimir 2.0, we open sourced the experimental query-sharding feature that was introduced in GEM 1.6. This feature has been promoted to stable, and new documentation about how to enable and configure it is in the Query sharding section of the Grafana Mimir 2.0 docs.
We’ve released a version 3 of the admin-api that removes the ability to hard delete tenants, access policies, and tokens via the admin-api. Instead, when users no longer want a tenant, access policy, or token, they are expected to soft delete it by marking it “inactive”. Moving to soft deletes allows us to eliminate several race conditions and cache invalidation problems that caused unexpected behavior with hard deletes. For more information about the v3 API, see the Admin-API documentation.
For those running GEM on Kubernetes, we’ve released a new version of the GEM Helm chart alongside our GEM 2.0 release. The chart itself includes multiple changes including most notably that the chart was renamed to mimir-distributed. There will now be 1 Helm chart for installing both OSS Grafana Mimir and Grafana Enterprise Metrics, with users using the enterprise.enabled flag to choose between the two. For more detail, see the Helm chart README.
We’ve made two improvements to GEM’s self-monitoring functionality:
- We added several panels to the Compactor dashboard. They display the time since the last successful compaction run and indicate when too much time has elapsed. This can help flag a malfunctioning or under-resourced compactor.
- GEM can now emit exemplars for its self-monitoring counter and histogram metrics, making it easier to jump from self-monitoring metrics to representative traces. For information on how to enable, see the exemplars section of the self-monitoring documentation.
The per-tenant limit compactor_blocks_retention_period can now be set via the admin-api. For more information, see Setting-per-tenant resource usage limits. That being said, we encourage those using the admin-api to migrate to managing per-tenant resource usage limits via runtime configuration because we intend to deprecate the ability to manage resource limits via admin-api in the future.

Upgrade considerations

GEM 2.0 is a major version upgrade and therefore introduces several breaking changes relative to GEM 1.7. The majority of the breaking changes are concentrated in GEM’s configuration, where we removed a significant number configuration flags (454 to be exact) that were no longer needed. We renamed and/or updated the defaults for numerous other flags whose legacy names or default values no longer made sense. This work allowed us to reduce and simplify GEM’s configuration surface and therefore make it more approachable to users.

To make the GEM 1.7 to GEM 2.0 upgrade as easy as possible, we’ve added functionality to mimirtool for automating the changes users need to make to their configuration.

We’ve paired this tooling with a detailed migration guide that walks through all required changes needed to go from GEM 1.7 to GEM 2.0 in a step-by-step manner. The guide also includes specific instructions on how to perform the upgrade using the Helm chart.

Once you upgrade your GEM backend to version 2.0, we recommend you upgrade your Grafana Enterprise GEM plugin to >= v3.4.0. This is needed to get the improved Compactor panel dashboards mentioned in the “Features and enhancements” section. It also ensures the plugin works with the newest version (v3) of the admin-api.

Bug fixes

v2.0.1

Fixed a bug where authentication using the wildcard (*) username will fail when there are tenants deleted via the v3 admin API endpoints.

v2.0.0

The following lists the most important bugs that are fixed in Grafana Enterprise Metrics 2.0.0:

Fixed a bug that caused GEM to report an incorrect license expiration timestamp in the metric grafana_labs_license_expiry_timestamp if multiple valid licenses existed in local storage and object storage.
Enterprise configuration extensions now appear in GEM’s /config endpoint. Before the fix, only configuration values present in the upstream OSS were returned.
Transient errors while attempting to authenticate user requests to GEM now return an HTTP 500 status code instead of an HTTP 401 code. The 500 code ensures that clients like Prometheus or the Grafana agent will now retry their remote write request rather than dropping it.
Transient errors are no longer cached by GEM, ensuring that GEM does not repeatedly return an HTTP 500 status code to the client due to a one-off failure.
When using a label-based access policy with multiple label selectors, GEM now ensures that label values are filtered properly. Before the bug fix, users with an LBAC policy with multiple label selectors could see label values they did not have access to. Metric names and metrics data were already filtered properly and unaffected by this bug.

For the full list of bug fixes, see the Changelog.

Grafana Enterprise Metrics v1.7 release notes

Thu, 06 Jan 2022 00:00:00 +0000

Version 1.7 release notes

The Grafana Enterprise Metrics (GEM) team is excited to announce the release of GEM 1.7.

Features and enhancements

GEM’s admin-api is now compatible with Azure Blob storage. Check out the admin_client_config section of the global configuration for more details on how to set this up.
Two enhancements have been made to the federation frontend, a service which allows users to query multiple GEM clusters at once and combine the results. See Cluster query federation for more information.
- The federation frontend service will now propagate a request’s bearer token, allowing that token to be used for authentication at its downstream targets.
- The federation frontend service now supports TLS configuration to its downstream targets.

Upgrade considerations

Users can upgrade from GEM 1.6.2 to GEM 1.7 with no additional considerations.

Bug fixes

v1.7.1

Fixed CVE-2022-44643

Grafana Enterprise Metrics v1.6 release notes

Mon, 15 Nov 2021 00:00:00 +0000

Version 1.6 release notes

The Grafana Enterprise Metrics (GEM) team is excited to announce the release of GEM 1.6.

Features and enhancements

Two new API endpoints help you to explore the metrics cardinality of tenants within a cluster. These endpoints make it possible to identify the highest cardinality metrics in a tenant, as well as the label names with the highest value count. The GEM plug-in (starting with version 3.3.0) includes a set of cardinality dashboards built on these endpoints, which allows users to explore the cardinality data visually. For more information, see Cardinality management.
Experimental support for query sharding was added. It allows you to split up a single query into groups of smaller series and parallelize it. This improves the performance of high cardinality queries that touch many series. This feature is under active development, so take care during its use.
Experimental support was added for a new compaction strategy known as split and merge compaction. This strategy enables users to parallelize the compaction of a large tenant horizontally across multiple machines. It also enables users to surpass the 64GiB block index limit inherent in Prometheus, and raise the ceiling on how many time series a single tenant can support. This feature is under development; use with caution.
The term instance, which had been used as a synonym for tenant, was eliminated. This allows GEM to more closely align with terminology used in Mimir. Previously, instance had multiple meanings besides tenant, such as an instance of an ingester.
- All references to “instance” in the GEM documentation and GEM Grafana plug-in have been replaced with the word “tenant”.
- We’ve added new v2 API endpoints which specify “tenant” in the request and response bodies. We have deprecated and will eventually delete the original endpoints that use the word ‘instance’. For more detail, see the admin-api page.
GEM 1.6 renames the default auth option to trust. This reflects the fact that the trust option does not actually do any authentication, but simply ‘trusts’ the value of the HTTP header X-Scope-Org-Id to determine what tenant to send a request to. GEM users are highly recommended to use auth type enterprise, not trust.
We’ve made some efficiency improvements, including reduced CPU and memory utilization on the ingesters and queries and when seeing large volumes of out-of-order samples.

Upgrade considerations

To get the dashboards associated with the Cardinality management feature, upgrade your Grafana GEM administrative plug-in to version 3.3.0.
- To ensure you receive the most up-to-date dashboards, you must disable and then re-enable the plug-in. Therefore, after you install the GEM plug-in version 3.3.0, navigate to its Plugin configuration page, and select Disable plugin. After Grafana reloads, select Enable plugin.
See the note under Changes about the auth option being renamed from default to trust. The default maximum number of inflight ingester push requests (-ingester.instance-limits.max-inflight-push-requests) changed to 30000. It is unlikely that this change affects the steady state operation of any GEM user. The maximum number acts as a safeguard against clusters getting overloaded by a spike in requests.
The option -querier.ingester-streaming has been removed. The querier and ruler now always use streaming method to query ingesters.
GEM 1.6 has removed support for block deletion marks migration (-compactor.block-deletion-marks-migration-enabled). If you’re upgrading to GEM 1.6 from GEM < 1.1.0, then you need to upgrade to GEM 1.5 first and run the migration at least once. Then you can upgrade to 1.6.

Bug fixes

1.6.2

GEM 1.6.2 fixes a bug that caused tenant limits set via the Admin API to be invalidated when upgrading to GEM 1.6.0 or GEM 1.6.1 from previous versions of GEM.

1.6.1

1.6.1 fixes one important bug in the 1.6.0 release:

GEM 1.6.1 fixes an issue with the way default tenant limits were applied when setting tenant limits using the Admin API.

1.6.0

GEM 1.6 changes the way per-tenant limits are stored to avoid breaking changes between version updates.
GEM 1.6 has a bugfix to ensure that the recording rules executed by the __system__ tenant (part of the self-monitoring feature) are properly split across rulers. This should eliminate out-of-order errors on the __system__ tenant.
GEM now enforces that a tenant must exist even when a wildcard access policy is used. Writes to non-existent tenants with a wildcard write access policy will fail. Similarly, if there are any read requests to a non-existent tenant using a wildcard write access policy, they will fail.
If a GEM tenant is deleted but the access policies to that GEM tenant are not, GEM will now fail read and write requests to that tenant.