Documentation Plugins Splunk data source Configure

Configure the Splunk data source

This document explains how to configure the Splunk data source in Grafana.

Before you begin

Before configuring the data source, ensure you have:

• Grafana permissions: Organization administrator role.
• Splunk account: A Splunk account with credentials (username and password or an authentication token).
• Network access: Port 8089 enabled on your Splunk instance.

Key concepts

If you’re new to Splunk, these terms are used throughout the configuration:

Add the data source

To install the plugin, refer to Install the Splunk data source. For general information on adding a data source, refer to Add a data source.

To add the Splunk data source, complete the following steps:

1. Click Connections in the left-side menu.
2. Under Connections, click Add new connection.
3. Enter Splunk in the search bar.
4. Select Splunk data source.
5. Click Add new data source in the upper right.

You are taken to the Settings tab where you set up your Splunk configuration.

Configuration options

The following sections describe the available configuration options.

General settings

Connection settings

Authentication

The data source supports multiple authentication methods. Choose the method that best fits your deployment.

Basic authentication

The most common authentication method. Use your Splunk username and password to connect. Avoid using the default administration account; create a dedicated user for Grafana with appropriate permissions.

Authentication token

Use an authentication token generated in Splunk instead of a username and password. Tokens let you provide access to environments without sharing standard credentials.

To configure token authentication:

1. Generate a token in Splunk. Refer to the Splunk Create authentication tokens documentation.
2. In the data source configuration, enter the token in the Authentication token field under Alternative authentication.

Forward OAuth Identity

Forward the logged-in user’s OAuth token to Splunk for authentication. This enables user-level authentication without requiring separate Splunk credentials.

TLS configuration

The following TLS options are available:

Custom HTTP headers

Add custom HTTP headers to all requests sent to the data source. This is useful for custom authentication or routing requirements.

Advanced options

The advanced options are available under the Additional settings section in the data source configuration.

Advanced HTTP settings

Query and performance options

Search mode

There are two search mode settings:

• Fields search mode: Sets the search mode for field queries. Options are quick and full.
• Variables search mode: Sets the search mode for variable queries. Options:
  • fast: Turns field discovery off for event searches. No event or field data for stats searches.
  • smart: Turns field discovery on for event searches. No event or field data for stats searches.
  • verbose: Returns all event and field data.

Secure Socks Proxy

Toggle on to route data source traffic through a secure SOCKS proxy.

Search Head Cluster configuration

If your Splunk deployment uses a Search Head Cluster (SHC) behind a load balancer, additional configuration is required to ensure the data source works reliably.

The Splunk plugin creates a search job and then issues follow-up requests to poll its status and retrieve results. All requests for a given search job must reach the same search head, because search job IDs (sid) are local to the search head that created them.

Requirements

1. Use token authentication or basic authentication. Authentication tokens replicate across Search Head Cluster members, so any search head can validate a token. Basic authentication also works across members: Grafana sends the username and password on every HTTP request, rather than relying on a Splunk session cookie. Either method still requires sticky sessions, because search job IDs (sid) are local to the search head that created the job.

2. Configure load balancer sticky sessions. Set up session affinity (source IP or cookie-based) on the load balancer in front of your SHC. This ensures all requests from a single Grafana instance are routed to the same search head for the duration of a search job. Sticky sessions are especially important when Async queries or Preview mode is enabled, because these modes issue multiple polling requests per query.

3. Point the data source URL at the load balancer. Enter the load balancer address (for example, https://splunk-lb.example.com:8089) in the URL field, not the address of an individual search head.

Provisioning example

YAML Copy

apiVersion: 1
datasources:
  - name: Splunk (SHC)
    type: grafana-splunk-datasource
    access: proxy
    editable: true
    enabled: true
    jsonData:
      authType: custom-splunk
      fieldSearchType: quick
      variableSearchLevel: fast
    secureJsonData:
      authToken: <SPLUNK_AUTH_TOKEN>
    url: https://splunk-lb.example.com:8089

Grafana high availability

If you run multiple Grafana instances (for example, in a Kubernetes deployment), each instance maintains its own connection to Splunk. Ensure the load balancer sticky session configuration accounts for traffic from multiple source IP addresses, one per Grafana replica.

Data links

Data links allow you to associate data with other Grafana data sources or external URLs. They are commonly used in Explore mode.

To set up a data link, click + Add under the Data links header in data source settings.

Click + Add to add multiple data links. Click the red X to remove a link.

Verify the connection

Click Save & test to verify the connection. On success, you see a message like:

Connected to Splunk version: “9.1.3” build: “d95b3bc7f6d0”

If the connection fails, refer to Troubleshoot Splunk data source issues.

Provision the data source

You can define and configure the Splunk data source in YAML files as part of the Grafana provisioning system. For more information about provisioning, refer to Provision Grafana.

Basic authentication example:

YAML Copy

apiVersion: 1
datasources:
  - name: Splunk
    type: grafana-splunk-datasource
    access: proxy
    basicAuth: true
    basicAuthUser: <USERNAME>
    editable: true
    enabled: true
    jsonData:
      fieldSearchType: quick
      internalFieldsFiltration: true
      variableSearchLevel: fast
      previewMode: false
      maxResultCount: 999
    secureJsonData:
      basicAuthPassword: <PASSWORD>
    url: <SPLUNK_URL>

Token authentication example:

YAML Copy

apiVersion: 1
datasources:
  - name: Splunk
    type: grafana-splunk-datasource
    access: proxy
    editable: true
    enabled: true
    jsonData:
      authType: custom-splunk
      fieldSearchType: quick
      variableSearchLevel: fast
    secureJsonData:
      authToken: <SPLUNK_AUTH_TOKEN>
    url: <SPLUNK_URL>

Set query results limit

To improve performance, you can limit the number of returned results. When more than one positive limit is set, the lowest value is used.

1. The Results limit value in the data source configuration. A positive value is used as the limit. 0 applies the backend safety cap of 10000.
2. The GF_PLUGIN_GRAFANA_SPLUNK_DATASOURCE_MAX_RESULT_LIMIT environment variable (self-managed only). When this variable and Results limit are both positive, the lower value is used. When Results limit is 0, this variable replaces the 10000 safety cap.
3. The per-query Limit in the query editor. The plugin uses this value only when it is lower than the resolved data source limit and lower than 10000.