---
title: "Needle-in-a-haystack searches | Grafana Cloud documentation"
description: "Use Logline to accelerate needle-in-a-haystack (NIAH) searches in Grafana Loki."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Needle-in-a-haystack searches

Logline is a secondary indexing technology for Grafana Loki that dramatically accelerates “needle-in-a-haystack” (NIAH) searches, which are queries that search large volumes of log data for rare, highly specific values.

A needle-in-a-haystack query searches a large volume of logs for a rare, specific value. These searches usually return only a small number of matching log lines, but can be expensive because Loki may need to scan many chunks to find them.

Logline works by building a compact probabilistic index from ingested log data. During query execution, the index identifies which chunks contain matching log lines. Loki skips chunks that the index rules out, which reduces bytes scanned and query latency.

Acceleration does not change query results. If the index lookup fails or takes too long, Loki runs the query without acceleration and returns the same results.

> Note
> 
> Queries that scan less than 10 GiB of log data are not accelerated. These queries already complete quickly without index assistance.

Needle-in-a-haystack search is generally available (GA) and is rolling out to Grafana Cloud Logs customers during the fourth quarter of 2026.

For background on the technology, see the [Logline acquisition blog post](/blog/grafana-labs-acquires-logline/).

## Use cases

Logline is useful when you need to quickly locate a specific, rare value across large log volumes, for example:

- A transaction ID or trace ID across weeks of payment logs
- A specific user ID or device ID in high-volume application logs
- An intermittent error string across a large multi-service environment

## Supported queries

Logline accelerates the following LogQL filter expressions:

- Exact string matches: `|= "value"`
- Regular expression matches: `|~ "value"` (case-sensitive and case-insensitive using `(?i)`). A regular expression is accelerated when it contains a literal run of 6 or more characters that every match must contain. For example, in `|~ "request_id=[0-9a-f]+ status=timeout"`, the literals `request_id=` and `status=timeout` are used to accelerate the search.
- Filters on stream labels or structured metadata, checked before any parser stage, for example `| trace_id="40ca1eb8-3f45-4d28-9033-62469a064a46"`. The value must have 6 or more characters.
- Filters on fields parsed with `json`, `logfmt`, `regexp`, or `pattern`, for example `| json | transaction_id="txn-123456"`. The value must appear exactly as written in the original log line. Values that contain quotation marks, backslashes, or control characters are not accelerated.

If a query has more than one filter, only one filter needs to match a supported pattern for the query to be accelerated. The query engine applies the remaining filters normally.

**Examples:**

Search for an exact transaction ID in payment logs:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="payments"} |= "txn-123456"
```

Case-insensitive search for an error phrase:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"} |~ "(?i)timeout exception"
```

Case-sensitive literal regular expression search for a trace ID:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{app="frontend"} |~ "trace-id: 40ca1eb8-3f45-4d28-9033-62469a064a46"
```

Regular expression with a long literal value surrounded by wildcards:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{env="production"} |~ ".*90429581536108547623.*"
```

Search for a structured metadata or stream label value, checked before the log line is parsed:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"} | trace_id="40ca1eb8-3f45-4d28-9033-62469a064a46"
```

Search for a value in a field parsed from JSON:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"} | json | transaction_id="txn-123456"
```

Pipeline with an accelerated filter followed by an additional filter (the index accelerates `|=`; the query engine handles `!=`):

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{cluster="prod"} |= "user-id-789" != "healthcheck"
```

Search across multiple services for a specific device ID:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{env="production"} |= "device-abc-9876"
```

## Limitations

The following queries still run, but without acceleration:

- Filter values with fewer than 6 characters. Short values appear too frequently in the index to provide meaningful narrowing:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} |= "12345"
  ```
- Negative filters (`!=` and `!~`). The index only accelerates positive filters that confirm a value is present:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} != "timeout exception"
  ```
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} !~ "(?i)connection refused"
  ```
- Numeric, duration, and byte-size comparisons. The index works with text values, not numeric ranges:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} | status >= 500
  ```
- IP address filters. These use a specialized matching function that the index does not support:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} |= ip("192.0.2.1")
  ```
- Filters joined with `or`. Logline does not yet support accelerating joined filters.
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} |= "timeout" or "unavailable"
  ```
- Filters that come after a stage that rewrites the log line, such as `line_format`, `decolorize`, or `unpack`. Filters before these stages are still accelerated:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} | line_format "{{.msg}}" |= "timeout exception"
  ```
- Filters on fields after a stage that changes or discards labels, such as `label_format`, `keep`, `drop`, or `unpack`:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} | json | keep msg | transaction_id="txn-123456"
  ```
- Regular expressions with no literal run of 6 or more characters. The index needs a fixed substring to look up:
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} |~ "[0-9]+"
  ```
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  {service="api"} |~ "(error|warning)"
  ```
- Metric queries. Logline does not yet support accelerating Metrics queries.
  
  logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
  
  ```logql
  count_over_time({env="production"} |= "device-def-5432")
  ```