Set up GEM on Grafana Labs

Deploy on Linux

Mon, 11 Apr 2022 17:01:18 +0100

Deploy GEM on a Linux host

This guide provides a step by step process for installing Grafana Enterprise Metrics on a Linux machine. It assumes you have access to a Linux machine and the permissions required to deploy a service with network and filesystem access. At the end of this guide you will have deployed a single GEM instance on a single node.

Prerequisites

In order to follow this guide you will the following:

A valid Grafana Labs license with an associated GEM cluster name.

Setup an object storage bucket

To begin, you need access to an object storage backend. GEM uses object storage as a backend to store time series data as well as data related to the state of the system. This topic assumes that you are using Amazon S3 on the AWS us-east-1 region. If you plan on using a different region or object storage service, update the storage fields in the configuration file to match below to configure your desired service. Currently, the supported services are object storage backends that support the S3 API or Google GCS.

After you have provisioned an object storage backend, be sure to pre-create two buckets: grafana-metrics-admin and grafana-metrics-tsdb. Those buckets will be referenced in the configuration file of this guide.

Prepare your system

Before running GEM, it is recommended you setup a system user and prepare the required directories on the filesystem. Run the following commands on every node as the root user. To login as the root user run:

sudo su -

Add dedicated user and group

groupadd --system ge-metrics
useradd --system --home-dir /var/lib/ge-metrics -g ge-metrics ge-metrics

Create directories and assign ownership

mkdir -p /etc/ge-metrics /var/lib/ge-metrics
chown ge-metrics:ge-metrics /etc/ge-metrics
chown ge-metrics:ge-metrics /var/lib/ge-metrics
chmod 0750 /etc/ge-metrics /var/lib/ge-metrics

Copy the license file

You will need to ensure you license token file is copied to the path /etc/ge-metrics/license.jwt.

Create a GEM configuration file

Copy the following YAML config to a file called /etc/ge-metrics/config.yaml.

auth:
  type: enterprise

target: all

license:
  path: /etc/ge-metrics/license.jwt

admin_client:
  storage:
    type: s3
    s3:
      endpoint: s3.amazonaws.com
      bucket_name: grafana-metrics-admin
      access_key_id: # TODO: insert access key id here
      secret_access_key: # TODO: insert secret access key here
distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946

ingester:
  lifecycler:
    num_tokens: 512
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

blocks_storage:
  tsdb:
    dir: /tmp/cortex/tsdb
  bucket_store:
    sync_dir: /tmp/cortex/tsdb-sync
  backend: s3
  s3:
    endpoint: s3.amazonaws.com
    bucket_name: grafana-metrics-tsdb
    access_key_id: # TODO: insert access key id here
    secret_access_key: # TODO: insert secret access key here

storage:
  engine: blocks

Next you will need to update the config file and set the blocks_storage.s3 and admin_client.storage.s3 sections to include access credentials for your object storage backend.

Download and configure the GEM binary

curl -Lo /usr/local/bin/ge-metrics https://dl.grafana.com/gem/releases/metrics-enterprise-v1.7.0-linux-amd64
echo "262ca08136dc28eee92a6494981ee8de7a7dd859c7a3ea5afa53ff99a56eb42d /usr/local/bin/ge-metrics" | sha256sum -c
chmod 0755 /usr/local/bin/ge-metrics
setcap 'cap_net_bind_service=+ep' /usr/local/bin/ge-metrics

Set up systemd unit

Copy the following file to the path /etc/systemd/system/ge-metrics.service:

[Unit]
After=network.target

[Service]
User=ge-metrics
Group=ge-metrics
WorkingDirectory=/var/lib/ge-metrics
ExecStart=/usr/local/bin/ge-metrics \
   -config.file=/etc/ge-metrics/config.yaml

[Install]
WantedBy=default.target

Enable startup at boot timeand start Grafana Enterprise Metrics

systemctl daemon-reload
systemctl enable ge-metrics.service
systemctl start ge-metrics.service

Generate an admin token

Generate an admin token by running the following on a single node in the cluster:

su ge-metrics -c "/usr/local/bin/ge-metrics \
--config.file=/etc/ge-metrics/config.yaml \
--target=tokengen"

The output of the above command should contain a token:

Token created:  YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw=

Verify your cluster is working

To verify your cluster is working you can run the following command using the token you generated in the previous step. For example:

curl -u :YWRtaW4tcG9saWN5LWJvb3RzdHJhcC10b2tlbjo8Ujc1IzQyfXBfMjd7fDIwMDRdYVxgeXw= localhost/ready

After running the above command you should see the following output:

ready

Next steps

Now that you have a working GEM deployment locally, refer to setup up the Grafana Enterprise Metrics plugin for instructions on how to integrate your metrics cluster with Grafana and give you a UI to interact with the Admin API.

Deploy on Kubernetes

Mon, 17 Mar 2025 14:18:51 +0100

Deploy GEM on Kubernetes

This guide will layout a step by step approach to deploying a Grafana Enterprise Metrics cluster on an existing Kubernetes namespace. This guide assumes you have a working Kubernetes cluster and the ability to deploy to that cluster using the kubectl tool. At the end of this guide you should have a functional GEM cluster running on Kubernetes using Minio as a storage backend. If you do not currently have access to a Kubernetes cluster consider following Linux deployment guide instead.

Deploy Minio

For this guide we will be using Minio as our object storage backend. Minio is a open source S3 compatible object storage service that is freely available and easy to run on Kubernetes. If you want to follow this guide but use a different object storage backend, refer to Grafana Enterprise Metrics configuration.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. Will be used in deployment below.
  name: minio-pv-claim
  labels:
    app: minio-storage-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 50Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  selector:
    matchLabels:
      app: minio
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the PVC created earlier
      volumes:
        - name: storage
          persistentVolumeClaim:
            # Name of the PVC created earlier
            claimName: minio-pv-claim
      initContainers:
        - name: create-buckets
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              "mkdir -p /storage/grafana-metrics-tsdb && mkdir -p /storage/grafana-metrics-admin",
            ]
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
      containers:
        - name: minio
          # Pulls the default Minio image from Docker Hub
          image: minio/minio:latest
          args:
            - server
            - /storage
          env:
            # Minio access key and secret key
            - name: MINIO_ACCESS_KEY
              value: "minio"
            - name: MINIO_SECRET_KEY
              value: "minio123"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: storage # must match the volume name, above
              mountPath: "/storage"
---
apiVersion: v1
kind: Service
metadata:
  name: minio
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Copy the above yaml into a file called minio.yaml and run the following command:

kubectl create -f minio.yaml

You can confirm you have setup Minio correctly by port-forwarding it and navigating to it in your browser:

kubectl port-forward service/minio 9000:9000

Then you can navigate to the minio admin console using your browser. The login credentials are the same as those set above, minio and minio123.

Create a license Secret

Next you will need to take the license.jwt file associated with you cluster and run the following command to load it as a Kubernetes Secret.

kubectl create secret generic ge-metrics-license --from-file license.jwt

Verify you have successfully created the secret by running the following command:

kubectl get secret ge-metrics-license -oyaml

The above command should print a kubernetes Secret object with a license.jwt field with a long base64 encoded value string.

Create a GEM config ConfigMap

Next you will create a configuration file for you cluster and deploy it as a Kubernetes ConfigMap. Copy the text and save it as config.yaml.

auth:
  type: enterprise

target: all

license:
  path: /etc/ge-metrics/license/license.jwt 

admin_client:
  storage:
    type: s3
    s3:
      endpoint: minio:9000
      bucket_name: grafana-metrics-admin 
      access_key_id: minio
      secret_access_key: minio123
      insecure: true

distributor:
  shard_by_all_labels: true
  pool:
    health_check_ingesters: true

memberlist:
  abort_if_cluster_join_fails: false
  bind_port: 7946
  join_members:
  - ge-metrics-discovery

ingester:
  lifecycler:
    num_tokens: 512
    ring:
      kvstore:
        store: memberlist
      replication_factor: 3

compactor:
  data_dir: "/data"
  sharding_enabled: true
  sharding_ring:
    kvstore:
      store: memberlist

blocks_storage:
  tsdb:
    dir: /data/cortex/tsdb
  bucket_store:
    sync_dir: /data/cortex/tsdb-sync
  backend: s3
  s3:
    endpoint: minio:9000
    bucket_name: grafana-metrics-tsdb
    access_key_id: minio
    secret_access_key: minio123 
    insecure: true 
storage:
  engine: blocks

Next run the following command to create the ConfigMap.

kubectl create configmap ge-metrics-config --from-file=config.yaml

Create the services for Grafana Enterprise Metrics

Two Kubernetes Services are required to run Grafana Enterprise Metrics as a StatefulSet. First is a service to support GRPC requests between replicas. Second is a gossip service port to allow the replicas to join together and form a hash ring to coordinate work.

Copy the following content into services.yaml:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics-discovery
  name: ge-metrics-discovery
spec:
  clusterIP: None
  ports:
    - name: ge-metrics-grpc
      port: 9095
      targetPort: 9095
    - name: ge-metrics-gossip
      port: 7946
      targetPort: 7946
  publishNotReadyAddresses: true
  selector:
    name: ge-metrics
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  ports:
    - name: ge-metrics-http-metrics
      port: 80
      targetPort: 80
  selector:
    name: ge-metrics

Create the services:

kubectl apply -f services.yaml

Deploy the Grafana Enterprise Metrics StatefulSet

Copy the following content into the statefulset.yaml file, and make sure to add the <version_number> of the GEM image that you would like to run:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    name: ge-metrics
  name: ge-metrics
spec:
  replicas: 3
  selector:
    matchLabels:
      name: ge-metrics
  serviceName: ge-metrics
  template:
    metadata:
      labels:
        name: ge-metrics
    spec:
      containers:
      - args:
        - -config.file=/etc/ge-metrics/config.yaml
        image: grafana/metrics-enterprise:<version_number>
        imagePullPolicy: IfNotPresent
        name: metrics-enterprise
        ports:
        - containerPort: 80
          name: http-metrics
        - containerPort: 9095
          name: grpc
        - containerPort: 7946
          name: gossip
        readinessProbe:
          httpGet:
            path: /ready
            port: 80
          initialDelaySeconds: 15
          timeoutSeconds: 1
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
      securityContext:
        fsGroup: 10001
        runAsUser: 10001
        runAsGroup: 10001
        runAsNonRoot: true
      terminationGracePeriodSeconds: 300
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50Gi

Create the StatefulSet:

kubectl apply -f statefulset.yaml

Start the GEM compactor as a Kubernetes deployment

The single binary of GEM does not enable the compactor component by default. Therefore, you need to run the compactor separately as a Kubernetes StatefulSet. For more information about what the compactor does, refer to the Compactor section of the Cortex documentation.

Copy the following content into the compactor.yaml file, and make sure to update the <version_number> of the GEM image that you would like to run:

  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    labels:
      name: ge-metrics-compactor
    name: ge-metrics-compactor
  spec:
    replicas: 1
    selector:
      matchLabels:
        name: ge-metrics-compactor
    serviceName: ge-metrics-compactor
    template:
      metadata:
        labels:
          name: ge-metrics-compactor
      spec:
        containers:
        - args:
          - -target=compactor
          - -config.file=/etc/ge-metrics/config.yaml
          image: grafana/metrics-enterprise:<version_number>
          imagePullPolicy: IfNotPresent
          name: ge-metrics-compactor
          ports:
          - containerPort: 80
            name: http-metrics
          - containerPort: 9095
            name: grpc
          - containerPort: 7946
            name: gossip
          readinessProbe:
            httpGet:
              path: /ready
              port: 80
            initialDelaySeconds: 15
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 1200m
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          volumeMounts:
          - mountPath: /data
            name: data
          - mountPath: /etc/ge-metrics
            name: ge-metrics-config
          - mountPath: /etc/ge-metrics/license
            name: ge-metrics-license
        securityContext:
          fsGroup: 10001
          runAsUser: 10001
          runAsGroup: 10001
          runAsNonRoot: true
        terminationGracePeriodSeconds: 300
        volumes:
        - name: ge-metrics-config
          configMap:
            name: ge-metrics-config
        - name: ge-metrics-license
          secret:
            secretName: ge-metrics-license
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi

Create the compactor StatefulSet:

kubectl apply -f compactor.yaml

Generate an admin token

To generate a token, use a Kubernetes Job.
Copy the following content it into the tokengen-job.yaml file, and make sure to update the <version_number> of the GEM image that you would like to run:

apiVersion: batch/v1
kind: Job
metadata:
  name: ge-metrics-tokengen
spec:
  template:
    spec:
      containers:
      - name: ge-metrics-tokengen
        image: grafana/metrics-enterprise:<version_number>
        args:
        - --config.file=/etc/ge-metrics/config.yaml
        - --target=tokengen
        volumeMounts:
        - mountPath: /etc/ge-metrics
          name: ge-metrics-config
        - mountPath: /etc/ge-metrics/license
          name: ge-metrics-license
        securityContext:
          runAsUser: 10001
          runAsGroup: 10001
          runAsNonRoot: true
      securityContext:
        fsGroup: 10001
      volumes:
      - name: ge-metrics-config
        configMap:
          name: ge-metrics-config
      - name: ge-metrics-license
        secret:
          secretName: ge-metrics-license
      restartPolicy: Never

Create the tokengen kubernetes Job:

$ kubectl apply -f tokengen-job.yaml

Check the status of the Pod, and after it displays ‘Completed’, check the logs for the new admin token:

kubectl logs job.batch/ge-metrics-tokengen

The output of the above command should contain a token in string in the logs:

Token created:  Ym9vdHN0cmFwLXRva2VuOmA3PzkxOF0zfVx0MTlzMVteTTcjczNAPQ==

Be sure to note down this token since it will be required later when setting up your cluster.

Verify your cluster is working

To verify your cluster is working you can run the following command using the token you generated in the previous step.

First, port-forward the ge-metrics Service to port 9000 on your machine:

kubectl port-forward service/ge-metrics 9000:80

Then run the following command:

curl -u :<your_token> localhost:9000/ready

After running the above command you should see the following output:

ready

Next steps

After you have a GEM deployment working locally, refer to Set up the GEM plugin for Grafana to integrate your metrics cluster into Grafana. Doing so gives you a UI via which you can interact with the Admin API.

Deploy on Kubernetes with Tanka

Mon, 17 Mar 2025 14:18:51 +0100

Deploy on Kubernetes with Tanka

To deploy Grafana Enterprise Metrics to Kubernetes using a Jsonnet library and Grafana Tanka, you can use MinIO to provide object storage. It is best to leverage your cloud provider’s object storage service to avoid the operational overhead of running object storage in production.

Prerequisites

Before you begin, verify that you have the following:

Kubernetes cluster
GEM license

Deployment instructions

Create a namespace:

$ kubectl create namespace enterprise-metrics

Create a Kubernetes Secret for your GEM license:

$ kubectl --namespace=enterprise-metrics create secret generic gem-license --from-file=license.jwt

Install Grafana Tanka; refer to Installing Tanka.
Install jsonnet-bundler; refer to the jsonnet-bundler README.

Set up Tanka environment:

a. Initialize Tanka:

$ tk init --k8s=false
$ tk env add environments/enterprise-metrics
$ tk env set environments/enterprise-metrics \
  --namespace=enterprise-metrics \
  --server-from-context=<KUBECFG CONTEXT NAME>

b. Install k.libsonnet for your version of Kubernetes:

$ mkdir -p lib
$ export K8S_VERSION=1.18
$ jb install github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}@main
$ cat <<EOF > lib/k.libsonnet
import 'github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}/main.libsonnet'
EOF

Install the Grafana Enterprise Metrics Jsonnet library.
console
```
$ jb install github.com/grafana/jsonnet-libs/enterprise-metrics
```
Deploy MinIO object storage; refer to Deploy Minio for the YAML manifests:
console
```
$ kubectl apply --namespace enterprise-metrics -f minio.yaml
```

Deploy a GEM cluster using the MinIO object storage by replacing the contents of the environments/enterprise-metrics/main.jsonnet file with the following configuration:

$ cat <<EOF > environments/enterprise-metrics/main.jsonnet
local gem = import 'github.com/grafana/jsonnet-libs/enterprise-metrics/main.libsonnet';

gem {
  _config+:: {
    commonArgs+:: {
      'admin.client.backend-type': 's3',
      'admin.client.s3.access-key-id': 'minio',
      'admin.client.s3.bucket-name': 'grafana-metrics-admin',
      'admin.client.s3.endpoint': 'minio:9000',
      'admin.client.s3.insecure': true,
      'admin.client.s3.secret-access-key': 'minio123',
      'blocks-storage.backend': 's3',
      'blocks-storage.s3.access-key-id': 'minio',
      'blocks-storage.s3.bucket-name': 'grafana-metrics-tsdb',
      'blocks-storage.s3.endpoint': 'minio:9000',
      'blocks-storage.s3.insecure': true,
      'blocks-storage.s3.secret-access-key': 'minio123',
    },
  },
  // Run without an alertmanager or ruler.
  alertmanager:: null,
  ruler:: null,
  // Make tokengen manifests available on a first run.
  tokengen+::: {}
}
EOF

Deploy Grafana Enterprise Metrics.

$ tk apply environments/enterprise-metrics/main.jsonnet

GEM limits recommendations

Mon, 17 Mar 2025 14:18:51 +0100

GEM limits recommendations

You can configure GEM limits to protect a system from any data sender (be it a machine or a person) who inadvertently sends or queries too much data, which can cause the system to run out of resources and crash. The limits also ensure a quality of service and fairness of a service across your user base.

GEM has the concept of tenants and an API where you are able to administer the limits set for your tenants. This is the preferred way to manage your limits and keep your system stable. The GEM Plugin for Grafana allows you to easily set these limits from inside your Grafana installation.

Since GEM 1.5.0, a self-monitoring __system__ tenant is available to help determine some of these limit values. For instructions on how to enable self-monitoring, please see the self-monitoring page.

If you do not have self-monitoring enabled or are using an earlier version of GEM, you need to query an external monitoring Prometheus to determine these values.

> Note: The terms tenant, and user are synonymous within the context of limit names; these terms all reference the same concept.

Global limits and configuration

You can use the following configuration options in your configuration file for all relevant GEM components or you can set them via CLI flags. Before making limits changes to individual tenants, restart the system. This stabilizes the system before you configure it using the API or plugin.

Max inflight ingester push requests

This option is a last-resort limit. If this limit is reached, remote_write samples that were sent to the write path return 5xx errors, which are retried by the sender (in Prometheus and Grafana Agent). A value of 5000 is conservative but effective, based on internal testing. If you change this setting on an already running system, you need to roll restart the ingesters, in the same way you would with any configuration change to the ingesters.

ingester:
  instance_limits:
    # CLI flag: -ingester.instance-limits.max-inflight-push-requests=5000
    max_inflight_push_requests: 5000 # default is 0 or unlimited in GEM 1.5.1 and earlier

Shard by all labels

This option helps to evenly distribute metrics across the pool of ingesters, because the incoming data is hashed by the metric name and the labels rather than only by the metric name. If you do not enable this option, some configured limits will not work correctly. If you change this setting on an already running system, you need to roll restart the distributors, in the same way you would with any configuration change to the distributors.

distributor:
  # CLI flag: -distributor.shard-by-all-labels=true
  shard_by_all_labels: true

Querier max concurrent

This option changes the number of queries a querier process is able to perform at once. A value of 8 is recommended. If you change this setting on an already running system, you need to roll restart the queriers, in the same way you would with any configuration change to the queriers.

querier:
  # CLI flag: -querier.max-concurrent
  max_concurrent: 8

Default minimal global settings

You can set limits for tenants above these minimal defaults. However, these limits help to keep the system stable if a new tenant is created and its limits have not been explicitly set. To keep things simple, apply these limits on every component because there are some components which embed other components and are as a result sometimes missed.

To override them, based on the needs of individual tenants, see Tenant limits

limits:
  max_global_series_per_user: 1000000
  ingestion_rate: 142857
  ingestion_burst_size: 142857
  ingestion_rate_strategy: global
  ruler_max_rules_per_rule_group: 20
  ruler_max_rule_groups_per_tenant: 70

Tenant limits

To set the following limits on each of your tenants, use the GEM plugin. Alternatively, you can manually use the Tenant API within the Admin API:

Max global series per user

max_global_series_per_user

Set the maximum number of series per tenant, which are enforced at the ingester level. This is generally described as the “number of metrics” that a tenant has. Note: The term user in the metric name does not mean a person using the system; in this context, the term is synonymous with tenant.

If you are onboarding a new tenant, set this value to a low number such as 10000 to ensure that the system has enough memory and CPU headroom before it starts accepting metrics. To determine if CPU and memory metrics for each component are below the limits that you have set, refer to the GEM system monitoring / GEM / Reads resources and GEM system monitoring / GEM / Writes resources dashboards. If a customer’s metrics are being discarded, slowly raise this value until the discarded samples go to zero, while checking to ensure that the system still has enough memory and CPU headroom to handle the additional load.
If you are retroactively applying limits for a tenant that is sending data, and the system is stable, use the GEM system monitoring / GEM / Per-tenant usage dashboard for each tenant to determine the value to set this to. For each tenant, choose Tenant from the top of the dashboard, set the dashboard date in the upper right-hand corner to a preferred time frame, and choose a value higher than the maximum value of the Number of active time series graph, which will ensure the senders’ data is not discarded. You might want to add a 10% buffer on top of the maximum number to allow for some ingesters to contain slightly more metrics than others due to imperfect load balancing across ingesters.

To determine if you are discarding samples for a tenant, use the Explore functionality in Grafana to perform the following query against the GEM System Monitoring datasource:

sum by (reason) (rate(cortex_discarded_samples_total{user="TENANT-ID"}[1m]))

where TENANT-ID is the ID of the tenant.

The reason label per_user_series_limit should be 0, otherwise GEM is rejecting samples as there is more data than allowed being sent.

Max fetched chunk bytes per query

max_fetched_chunk_bytes_per_query

This limit helps guard against querier processes utilizing too much memory and causing crashes.

For microservices deployments, Grafana Labs recommends leaving this limit unset. If you observe querier processes crashing frequently because they are out of memory you can consider setting this limit, but infrequent querier process crashes are tolerated by the distributed system. Alternatively, setting the querier limit -querier.max-concurrent to a lower value than the default is an option to lower per querier memory usage.
For single binary deployments, Grafana Labs recommends determining the value for this limit by taking the amount of memory you are willing to allocate to queries, and dividing that number of bytes by the value of the querier limit -querier.max-concurrent (see Querier max concurrent). Set the final value of max_fetched_chunk_bytes_per_query to 50% of the calculated value to account for various factors which increase memory overhead in the system. For example, if you are allocating 20GB of memory to queries, and the max-concurrent is set to 8, set the value to 1342177280.

Ingestion rate

ingestion_rate

This limit helps guard against an increase in high frequency incoming samples from using too much memory on the system. The number of series and the number of series per metric limits set above are also important, but if a sender was updating a series every second, it would take more memory than one that was updated every minute.

If you are onboarding a new tenant, Grafana Labs recommends setting this value to the value determined for max_global_series_per_user, divided by the scraping interval in seconds, then adding a 10% buffer. For example, if you have a value of 1000000 set for your max_global_series_per_user, the scraping interval of a sender is 15s, you should set the ingestion_rate to 73333. If you have higher or lower frequency data, you may want to increase or decrease this value, being sure to watch memory and CPU utilization on the system.
If you are retroactively applying limits for a tenant that is sending data, and the system is stable, use the GEM system monitoring / GEM / Per-tenant usage dashboard for each tenant to determine the value to set this to. For each tenant, choose its name in the Tenant dropdown at the top of the dashboard, set the dashboard date in the upper right-hand corner to a preferred time frame, and choose a value higher than the maximum value of the Ingestion rate graph, which will ensure the senders’ data is not discarded. You might want to add a 10% buffer on top of the maximum number to allow for some variability in the rate of data being sent.

To determine if senders are hitting the ingestion rate limit, use the Explore functionality in Grafana to perform the following query against the GEM System Monitoring datasource:

sum by (reason) (rate(cortex_discarded_samples_total{user="TENANT-ID"}[1m]))

where TENANT-ID is the ID of the tenant.

The reason label rate_limited should be 0, otherwise GEM is rejecting samples as there is more data than allowed being sent.

Ingestion burst size

ingestion_burst_size

This limit allows for some temporary influx of data to be sent above the normal ingestion rate, but for short periods of time. To be extra careful and cautious, set this value to the same value you have set for ingestion_rate, which allows for no bursting. However if you have enough headroom in your system and want to hit limits less frequently if a sender temporarily bursts above, you are able to set this value. Depending on your headroom you may want to set this to a higher or lower number, but a relatively safe value would be to double it.

For customers deploying using single binary or in a scenario where the queriers are part of the write path, Grafana Labs recommends setting this value equal to ingestion_rate, which disables bursting.

Ruler max rules per rule group

ruler_max_rules_per_rule_group

If you are using the ruler component, you should set limits around how many rules can exist in a rule group. A smaller number of rules per rule group allows for more even distribution across the rulers. Grafana Labs recommends setting this value to 20.

Ruler max rule groups per tenant

ruler_max_rule_groups_per_tenant

If you are using the ruler component, you should set limits around how many rule groups a tenant is allowed. This number can be higher or lower depending on how many ruler instances you have. The effective number of rules a tenant can have is the ruler_max_rule_groups_per_tenant multiplied by ruler_max_rules_per_rule_group.

Grafana Labs recommends setting this to a lower number, such as 10, and increasing it if tenants need more ruler capacity.

To determine if the rulers have capacity to allow an increase of rule groups, reference the GEM System Monitoring / GEM / Reads Resources dashboard in the Ruler section, to determine if CPU and memory metrics are beneath the hard limits you have set.