Set up GEM on Grafana Labs

GEM hardware requirements

Mon, 18 Aug 2025 18:19:44 +0000

GEM hardware requirements

This page outlines the current hardware requirements for running Grafana Enterprise Metrics (GEM). Grafana Labs reserves the right to mark a support issue as ‘unresolvable’ if these requirements are not followed. See the Grafana Labs Enterprise Support SLA for more details.

Note
As of February 2025, you must run GEM on the managed Kubernetes services offered by AWS (Amazon Elastic Kubernetes Service), GCP (Google Kubernetes Engine), or Azure (Azure Kubernetes Service). Additionally, you must use the managed object storage services offered by AWS (S3), GCP (GCS), or Azure (Blob Storage).

No other deployment modes are supported.

Here are more details on the hardware and object storage requirements for CPU, memory, and disk for managed Kubernetes clusters.

CPU and memory

You should deploy GEM on machines with a 1:4 ratio of CPU to memory, meaning that for every CPU core, there should be 4GB of memory. Grafana Labs recommends deploying GEM onto machines with 16 CPU cores and 64GB of memory. All nodes in the Kubernetes cluster running GEM should be of the same type.

Disk

Various components of GEM, including the ingester and Alertmanager, require fast and persistent, local disk resources to be available to the host machine. For example, in the case of the ingester component, all incoming data is sent to a write-ahead log (WAL) to help withstand unexpected node termination. Additionally, a large amount of data is flushed to disk every 2 hours, by default, in preparation for uploading it to long-term object storage. The supported configurations for AWS, GCP, and Azure are as follows:

Amazon Web Services (AWS)

GEM is tested to run with io1 Provisioned IOPS SSD EBS volumes. The io1 storage must be provisioned at 50 IOPS per gigabyte, with a minimum of 150Gi allocated to ensure performant I/O.

Google Cloud Platform (GCP)

GEM is tested to run with pd-ssd SSD persistent disks.

Microsoft Azure

GEM is tested to run with Premium SSD SSD persistent disks.

Network

All components of GEM require fast network access. Nodes the software runs on should be connected by 10 gigabit/second or faster network connection speed.

Object storage

GEM requires object storage for configuration storage, as well as long-term data storage. AWS S3, GCP GCS, and Azure Blob storage are the only supported object stores for running GEM.

Amazon Web Services (AWS)

GEM is tested to run with AWS’s S3 object storage service using the Standard storage class.

Google Cloud Platform (GCP)

GEM is tested to run with GCP’s GCS object storage service using the STANDARD storage class in both regional and dual regional storage locations.

Microsoft Azure

GEM is tested to run Azure’s Blob Storage object storage service using the Standard storage class with replication type LRS.

Migrate to Grafana Mimir

Thu, 14 Aug 2025 18:42:07 +0000

Migrate to Grafana Mimir

Refer to the following migration guides when you migrate to Grafana Mimir.

Deploy Mimir with Helm

Thu, 14 Aug 2025 18:42:07 +0000

Deploy Mimir with Helm

The mimir-distributed Helm chart for Grafana Mimir and Grafana Enterprise Metrics allows you to configure, install, and upgrade Grafana Mimir or Grafana Enterprise Metrics within a Kubernetes cluster.

Example values files

The mimir-distributed Helm chart includes the following example values files:

File name	Description
`values.yaml`	Contains default values for testing GEM in non-production environments using a test MinIO deployment for object storage.
`small.yaml`	Contains values for a higher scale than defaults, for ingestion up to approximately one million series. Not suitable for high-availability production use, due to single replicas of key components.
`large.yaml`	Contains values for production use for ingestion up to approximately ten million series.

Set up the GEM plugin for Grafana

Wed, 10 Jun 2026 16:15:01 +0000

Set up the GEM plugin for Grafana

Warning
Grafana Enterprise Metrics (GEM), Grafana Enterprise Logs (GEL), and Grafana Enterprise Traces (GET) have entered Long-Term Support (LTS):

GEM LTS version is 2.17.

GEL LTS version is 3.6.

GET LTS version is 2.8.

Existing customers will continue to receive support, bug fixes, and security fixes, but no new features, through Feb. 1, 2029, when GEM, GEL, and GET will reach End-of-Life (EOL).

If you have questions, contact your account manager.

Requirements

Grafana Enterprise 9.5 or higher.

Refer to Deploy Grafana Enterprise on Kubernetes if you are using Kubernetes. Otherwise, refer to Install Grafana for more information.

Install the plugin in your GEM instance

There are multiple ways to install the plugin to your local Grafana Enterprise instance. For more information, refer to Grafana Enterprise Metrics app installation.

Enable and configure the plugin

Log in to your Grafana Enterprise Metrics.
Go to the Config/Plugins page and select the Grafana Enterprise Metrics plugin from list.
From the configuration page of the plugin, enable the plugin by clicking on the Enable plugin button.
Provide the necessary API settings so that the plugin can connect to your cluster:

Access Token: Enter the admin-scoped access token that you generated when setting up your GEM cluster.
Enterprise Metrics URL: Enter the URL of your GEM cluster. For single-process clusters, this is any node in the cluster. For microservice deployments, this URL is the GEM gateway.

Configuration GEM plugin page

Click Save API settings.
Verify that the plugin loads and can communicate with the GEM admin API endpoints.
Navigate to the Grafana Enterprise Metrics plugin through the main menu to see the default access policy under the Access policies tab.

Configure the plugin via provisioning

You can alternatively manage the GEM plugin via a plugin config.

Example GEM plugin configuration file

apiVersion: 1

apps:
  # <string> the type of app, plugin identifier. Required
  - type: grafana-metrics-enterprise-app
    # <int> Org ID. Default to 1, unless org_name is specified
    org_id: 1
    # <string> Org name. Overrides org_id unless org_id not specified
    org_name: Main Org.
    # <bool> disable the app. Default to false.
    disabled: false
    # <map> fields that will be converted to json and stored in jsonData.
    jsonData:
      # key/value pairs of string to object
      backendUrl: http://[...]
    # <map> fields that will be converted to json, encrypted and stored in secureJsonData.
    secureJsonData:
      # <string>
      accessToken: [...]

Optional `policiesOnly` field

For a simpler user experience that only shows the “Access policies” page, you can include the following in the jsonData configuration:

jsonData:
  policiesOnly: true

This simpler mode of operation only gives access to the ability to create access policies and tokens. It requires setting up the environment to include at least one tenant.

Legacy `base64EncodedAccessToken` field

Prior to version 6.0.0 of the plugin, a base64EncodedAccessToken field was required for authentication. This has been replaced with the simpler accessToken.

The following is provided for troubleshooting purposes, or for users of versions earlier than 6.0.0.

Moving forward, use accessToken in your configuration.

If both base64EncodedAccessToken and accessToken are present in the jsonData, the base64EncodedAccessToken is ignored in versions 6.0.0 and later.

The format of the base64EncodedAccessToken token is a base64-encoded representation of a user-token pair, separated by colon. For example, $USER:$TOKEN. If you’ve set your GEM token as the TOKEN environment variable, and you’ve also set a USER environment variable, then you can generate a compatible base64EncodedAccessToken token in the following format:

#USER INTENTIONALLY LEFT BLANK, INDICATING THE "ADMIN" USER
USER=
TOKEN=exampleTokenForBase64EncodedAccessToken
echo $USER:$TOKEN | tr -d '[:space:]' | base64 -w0

Though $USER is typically blank, you can opt to leave it in the expression to ensure that the : character is present. The following is a more direct way of obtaining the same result:

echo :$TOKEN | tr -d '[:space:]' | base64 -w0

These examples both result in the following base64EncodedAccessToken string:

OmV4YW1wbGVUb2tlbkZvckJhc2U2NEVuY29kZWRBY2Nlc3NUb2tlbgo=

You can apply the encoded token string to the configuration under secureJsonData:

secureJsonData:
  # <string> a base64 encoding of $USER:$TOKEN
  base64EncodedAccessToken: OmV4YW1wbGVUb2tlbkZvckJhc2U2NEVuY29kZWRBY2Nlc3NUb2tlbgo=

Next steps

Now that you have correctly configured the plugin, you can move on to setting up a GEM tenant and visualizing your data.

Install dashboards

Install dashboards to add the latest versions of the cardinality and self-monitoring dashboards to your GEM instance. These dashboards help you to gain additional insight into your data in GEM.

In your Grafana instance, go to Administration > Plugins and data > Plugins > Grafana Enterprise Metrics.
Select the Dashboards tab.
Select Install Cardinality dashboards, Install Self-monitoring dashboards, or both, depending on which dashboards you want to install.

The Installed dashboards list updates to show the dashboards you installed. To view a dashboard, select it from the list. To reinstall the dashboards following an update, repeat the same steps.

Configure a private certificate authority

If you are running GEM in a closed environment using a self-signed certificate, you need to configure a private certificate authority. For more information, refer to Configure a Private CA (Certificate Authority) in the Grafana documentation.

Limitations

The GEM plugin for Grafana allows to manage only 1 GEM cluster, and it doesn’t support managing multiple GEM clusters with a single GEM plugin installation. In case you need to manage multiple GEM clusters you should configure 1 GEM cluster per Grafana organization.

Set up a GEM tenant

Mon, 18 Aug 2025 18:19:44 +0000

Set up a GEM tenant

Create a Grafana Enterprise Metrics tenant:

Navigate to Grafana Enterprise Metrics > Tenants.
Click on Create new tenant.
Choose a name for this very tenant. For demonstration purposes, use the name dev-tenant.
Select the cluster.
Click Create tenant.

Create an access policy

Create a data source access policy and token, which are used by Grafana Enterprise itself to access the metrics in the tenant named dev-tenant:

Navigate to Grafana Enterprise Metrics > Access policies.
Click Create new access policy.
Choose a name for the policy. For demonstration purposes, use dev-read-write-policy.
Enable the scope metrics:read and metrics:write.
Select the tenant dev-tenant.
Click on Create.

From the newly created access policy, click + Add token.
Name the token dev-token, and click on Create.
In the next window, copy the token.

If you’re using Prometheus, you can add the following example of a remote-write configuration:

remote_write:
  - url: http://enterprise-metrics/api/v1/push
    basic_auth:
      username: dev-tenant
      password: ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==

If you’re using Grafana Alloy, you can add the following example of a remote-write configuration:

prometheus.remote_write "default" {
  endpoint {
    url = "http://enterprise-metrics/api/v1/push"
    basic_auth {
      username = "dev-tenant"
      password = "ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ=="
      }
  }
}

Setup remote-write to your tenant

To enable writes to your cluster, add the above remote-write configuration snippet to the configuration file of an existing Prometheus or Grafana Alloy collector.

If you do not have an existing metrics collector you can get started with Grafana Alloy.

An example agent configuration would be:

server:
  log_level: info
  http_listen_port: 8081

prometheus:
  wal_directory: /tmp/agent
  global:
    scrape_interval: 5s
  configs:
    - name: default
      scrape_configs:
        - job_name: enterprise-metrics
          static_configs:
            - targets: ["localhost:8080"]
        - job_name: agent
          static_configs:
            - targets: ["localhost:8081"]

integrations:
  agent:
    enabled: true
  node_exporter:
    enabled: true
  prometheus_remote_write:
    - url: http://<gme-hostname>/api/v1/push
      basic_auth:
        username: dev-tenant
        password: ZGV2LXJlYWQtd3JpdGVyLXBvbGljeS1kZW1vLXRva2VuOjY/ezduMTVhJDQvPGMvLzQ1SzgsJjFbMQ==

Setup your tenant as a Grafana datasource

Following that we can use this access policy and token to create a new datasource in Grafana:

Navigate to Configuration ≫ Data Sources.
Click on Add new data source.
Select Prometheus.
Set the URL to http://<gme-host>/prometheus.
Enable Basic Auth and use User dev-tenant and as Password the token from your clipboard.
Click Save & Test.

Visualize your data

Once you have created a datasource you should now be able to visualize your metrics in the Grafana Explore page.

Upgrade from Grafana Enterprise Metrics 1.7

Mon, 18 Aug 2025 18:19:44 +0000

Upgrade from Grafana Enterprise Metrics 1.7

As an operator, you can migrate a deployment of Grafana Enterprise Metrics 1.7 to Grafana Enterprise Metrics 2.3. The steps to do so apply to any deployment environment, and there are specific instructions for environments deployed with Helm.

Grafana Enterprise Metrics 2.0 extends Grafana Mimir. Grafana Enterprise Metrics 1.7 extends Cortex. Grafana Mimir includes significant changes that simplify the deployment and continuous operation of a horizontally scalable, multi-tenant time series database with long-term storage.

The following changes make Grafana Mimir easy to run:

Removed configuration parameters that don’t require tuning
Renamed some parameters so that they’re more easy to understand
Updated the default values of some existing parameters to work out of the box

The Grafana Mimirtool (mimirtool) automates configuration conversion for you. It provides a simple migration by generating a Grafana Mimir configuration from a Cortex configuration. It can also generate a Grafana Enterprise Metrics 2.3 configuration from a Grafana Enterprise Metrics 1.7 configuration.

Before you begin

Ensure that you are running Grafana Enterprise Metrics 1.7.X.

If you are running an older version of Grafana Enterprise Metrics, upgrade to 1.7.0 before proceeding with the migration.
Ensure that you have installed the Grafana Mimir monitoring mixin rules.

The monitoring mixin has both alerting and recording rules to install in either Prometheus or Grafana Enterprise Metrics. The rules replace the Cortex monitoring mixin rules and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. To upload rules to the ruler using Grafana Mimirtool, refer to Grafana Mimirtool.
Ensure that you have installed the Grafana Mimir monitoring mixin dashboards.

The dashboards replace the Cortex monitoring mixin dashboards and work for monitoring both Grafana Enterprise Metrics 1.7 and Grafana Enterprise Metrics 2.0. Alternatively, update the Grafana Enterprise Metrics plugin to automatically install the latest self monitoring dashboards.

Notable changes

Note
For a full list of changes, refer to the Grafana Mimir CHANGELOG.

By default, Grafana Enterprise Metrics 2.0 enables enterprise authentication. To continue using the Grafana Enterprise Metrics 1.7 default named trust, use -auth.type=trust.
The Grafana Enterprise Metrics 2.0 HTTP server defaults to listening on port 8080; previous versions defaulted to listening on port 80. The enterprise-metrics Helm chart already used port 8080 and is unaffected by this change.
Grafana Enterprise Metrics 2.0 uses anonymous as the default tenant ID where -auth.multitenancy=false. Previous versions used fake as the default tenant ID where -auth.enabled=false. Use -auth.no-auth-tenant=fake where -auth.multitenancy=false, to match the default tenant ID of previous versions.

Grafana Enterprise Metrics 2.0 removes the legacy HTTP prefixes.

Query endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/query`	`<prometheus-http-prefix>/api/v1/query`
`/<legacy-http-prefix>/api/v1/query_range`	`<prometheus-http-prefix>/api/v1/query_range`
`/<legacy-http-prefix>/api/v1/query_exemplars`	`<prometheus-http-prefix>/api/v1/query_exemplars`
`/<legacy-http-prefix>/api/v1/series`	`<prometheus-http-prefix>/api/v1/series`
`/<legacy-http-prefix>/api/v1/labels`	`<prometheus-http-prefix>/api/v1/labels`
`/<legacy-http-prefix>/api/v1/label/{name}/values`	`<prometheus-http-prefix>/api/v1/label/{name}/values`
`/<legacy-http-prefix>/api/v1/metadata`	`<prometheus-http-prefix>/api/v1/metadata`
`/<legacy-http-prefix>/api/v1/read`	`<prometheus-http-prefix>/api/v1/read`
`/<legacy-http-prefix>/api/v1/cardinality/label_names`	`<prometheus-http-prefix>/api/v1/cardinality/label_names`
`/<legacy-http-prefix>/api/v1/cardinality/label_values`	`<prometheus-http-prefix>/api/v1/cardinality/label_values`
`/api/prom/user_stats`	`/api/v1/user_stats`

Distributor endpoints

Legacy endpoint	Current
`/<legacy-http-prefix>/push`	`/api/v1/push`
`/all_user_stats`	`/distributor/all_user_stats`
`/ha-tracker`	`/distributor/ha_tracker`

Ingester endpoints

Legacy	Current
`/ring`	`/ingester/ring`
`/shutdown`	`/ingester/shutdown`
`/flush`	`/ingester/flush`
`/push`	`/ingester/push`

Ruler endpoints

Legacy	Current
`/<legacy-http-prefix>/api/v1/rules`	`<prometheus-http-prefix>/api/v1/rules`
`/<legacy-http-prefix>/api/v1/alerts`	`<prometheus-http-prefix>/api/v1/alerts`
`/<legacy-http-prefix>/rules`	`<prometheus-http-prefix>/config/v1/rules`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/<legacy-http-prefix>/rules/{namespace}/{groupName}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}/{groupName}`
`/<legacy-http-prefix>/rules/{namespace}`	`<prometheus-http-prefix>/config/v1/rules/{namespace}`
`/ruler_ring`	`/ruler/ring`

Alertmanager endpoints

Legacy	Current
`/<legacy-http-prefix>`	`/alertmanager`
`/status`	`/multitenant_alertmanager/status`

Generating configuration for Grafana Enterprise Metrics 2.3

Grafana Mimirtool provides the mimirtool config convert --gem command for converting a Grafana Enterprise Metrics 1.7 configuration into a Grafana Enterprise Metrics 2.0 configuration. It can convert both flags and configuration files.

Downloading Grafana Mimirtool

Download the appropriate release asset for your operating system and architecture, and make it executable. For Linux with the AMD64 architecture:

curl -fLo mimirtool https://github.com/grafana/mimir/releases/latest/download/mimirtool-linux-amd64
chmod +x mimirtool

Using Grafana Mimirtool

Note
To ensure that the Grafana Enterprise Metrics specific configuration isn’t removed from the output, you must use the --gem flag.

The tool removes any configuration parameters that are no longer available in Grafana Enterprise Metrics 2.3, and it renames any configuration parameter that has a new name. If you have explicitly set configuration parameters to a value that matches the Grafana Enterprise Metrics 1.7 default, by default, mimirtool config convert doesn’t update the value. To have mimirtool config convert --gem update explicitly set values from the Grafana Enterprise Metrics 1.7 defaults to the new Grafana Enterprise Metrics 2.3 defaults, provide the --update-defaults flag. For more information about using mimirtool for configuration conversion, refer to the Grafana Mimirtool configuration command named convert.

Migrating to Grafana Enterprise Metrics 2.3 with Helm

Grafana Enterprise Metrics 2.3 uses the Grafana Mimir mimir-distributed Helm chart version 3.1.0. You can update to the mimir-distributed Helm chart from the enterprise-metrics Helm chart.

Before you begin

Ensure that you are running the v1.9.0 release of the enterprise-metrics Helm chart.
Install yq v4.

To migrate to the mimir-distributed Helm chart:

Update your Helm repositories.
Bash
```
helm repo update
```
Convert your existing Grafana Enterprise Metrics configuration in your Helm values file.

a. Extract the Grafana Enterprise Metrics configuration from the deployed Grafana Enterprise Metrics Secret and write the output to the enterprise-metrics.yaml file.
Bash
```
kubectl get secret <RELEASE>-enterprise-metrics -n <namespace> -o json | jq -r '.data."enterprise-metrics.yaml"' | base64 -d > enterprise-metrics-1.7.yaml
```
If you are using an external configuration Secret, replace <RELEASE>-enterprise-metrics with the name of your Secret.

b. Use mimirtool to update the configuration.
Bash
```
mimirtool config convert --gem --yaml-file enterprise-metrics-1.7.yaml --yaml-out enterprise-metrics-2.3.yaml
```
c. Clean up the generated YAML configuration.

You have to remove some fields that either the command mimirtool config convert generates or that are coming from an old configuration, because the mimir-distributed Helm chart has already set the default value for them. Use the following script to clean up those fields:
Bash
```
yq -i 'del(.activity_tracker.filepath,.alertmanager.data_dir,.compactor.data_dir,.frontend_worker.frontend_address,.ingester.ring.tokens_file_path,.ruler.alertmanager_url,.ruler.rule_path,.runtime_config.file)' mimir.yaml
```
d. Make sure querier.query_ingester_within is greater than querier.query_store_after default value in mimir 2.3.0 which is set to 12 hours by default.

Note
The Grafana Enterprise Metrics Helm chart added some configuration validation that prevents the installation to start if the value of a configuration (or combination of configurations) is invalid. Fix the configuration based on the validation errors.

e. At the top level of your Helm values file, put the updated configuration under the mimir.structuredConfig key.

Note
The mimir.structuredConfig field, which is added in v3.0.0, allows you to override a specific configuration without needing to rewrite the whole block string literal, such as in mimir.config.

In your Helm values file:

mimir:
  structuredConfig: <CONFIGURATION FILE CONTENTS>

Example:

mimir:
  structuredConfig: 
    ingester:
      ring:
        num_tokens: 512

f. Enable enterprise configuration by setting enterprise.enabled to true and enterprise.legacyLabels to true. The value enterprise.legacyLabels installs resources using the same labels as the enterprise-metrics Helm chart.

In your Helm values file:

enterprise:
  enabled: true
  legacyLabels: true

g. Latest version MinIO changes its credential fields from combination of accessKey and secretKey to rootUser and rootPassword. If you’re using the enterprise-metrics chart MinIO for object storage, set the minio.rootUser to enterprise-metrics and set the secret key or password to rootPassword.

In your Helm values file:

minio:
  rootUser: "enterprise-metrics"
  rootPassword: <password>

Run the Helm upgrade with the mimir-distributed chart.

During the upgrade, requests to the gateway fail for a short period of time. Prometheus and Grafana Agent retry failed write requests to avoid data loss.

Note
The name of the release must match your enterprise-metrics Helm chart release.

helm upgrade <RELEASE> grafana/mimir-distributed --version 3.1.0 -f <VALUES FILE> --set-file 'license.contents=<LICENSE FILE>' [-n <NAMESPACE>]

To verify that the cluster is operating correctly, use the self monitoring or monitoring mixin dashboards installed in Grafana.

Set up GEM on Grafana Labs

GEM hardware requirements

GEM hardware requirements

CPU and memory

Disk

Amazon Web Services (AWS)

Google Cloud Platform (GCP)

Microsoft Azure

Network

Object storage

Amazon Web Services (AWS)

Google Cloud Platform (GCP)

Microsoft Azure

Migrate to Grafana Mimir

Migrate to Grafana Mimir

Deploy Mimir with Helm

Deploy Mimir with Helm

Example values files

See also

Set up the GEM plugin for Grafana

Set up the GEM plugin for Grafana

Requirements

Install the plugin in your GEM instance

Enable and configure the plugin

Configure the plugin via provisioning

Example GEM plugin configuration file

Optional policiesOnly field

Legacy base64EncodedAccessToken field

Next steps

Install dashboards

Configure a private certificate authority

Limitations

Set up a GEM tenant

Set up a GEM tenant

Create an access policy

Setup remote-write to your tenant

Setup your tenant as a Grafana datasource

Visualize your data

Upgrade from Grafana Enterprise Metrics 1.7

Upgrade from Grafana Enterprise Metrics 1.7

Before you begin

Notable changes

Generating configuration for Grafana Enterprise Metrics 2.3

Downloading Grafana Mimirtool

Using Grafana Mimirtool

Migrating to Grafana Enterprise Metrics 2.3 with Helm

Before you begin

Optional `policiesOnly` field

Legacy `base64EncodedAccessToken` field