> ## Documentation Index
> Fetch the complete documentation index at: https://docs.startree.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Accessing logs

> Search your cluster's component logs in Grafana with LogQL, pull raw log files from a specific Pinot instance over the API, and temporarily raise a log level to reproduce a problem.

Metrics tell you that something happened; logs tell you what it said. StarTree Cloud gives you three ways in, and they answer different questions:

| Surface                        | Best for                                                                                     |
| ------------------------------ | -------------------------------------------------------------------------------------------- |
| **Grafana Logging dashboards** | Searching across components and time — "which pod logged this error, and when did it start?" |
| **Pinot `/loggers` API**       | Pulling the raw log file from one named instance, unparsed and complete                      |
| **Data Portal task logs**      | Logs for one minion task or subtask, next to its progress                                    |

Start with Grafana. It is indexed, searchable across every component at once, and does not require you to know which pod to look at.

## Searching logs in Grafana

Open [Grafana](/corecapabilities/observability/grafana) and go to the **Logging** dashboards. They search the log lines your cluster's components have written, indexed so you can filter and count rather than scroll.

### The dashboards

| Dashboard                                    | What it gives you                                                                                                                                                                         |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Logging / Common (historical log search)** | The general-purpose one. Log volume over time, level distribution, per-pod breakdown, an errors-only panel, and a free-text search box. Filter by log group, application, pod, and level. |
| **Logging / Application Logs**               | Scoped to application logs, with a timeline and an error/warning summary.                                                                                                                 |
| **Logging / Query Stats**                    | Query log lines with latency, rows returned, and the query text. Includes slowest-query and most-frequent-query panels.                                                                   |
| **Logging / Slow Queries Drilldown**         | Queries above a latency threshold you set, with per-table maximum latency and drill-down by request ID.                                                                                   |
| **Logging / Table Queries**                  | The same query view, scoped to one table.                                                                                                                                                 |
| **Logging / Kubernetes Events**              | Container OOM kills, failed liveness checks, evictions, and warnings. This is where you confirm a component was killed rather than crashed.                                               |
| **Logging / Tool & K8s Cluster Logs**        | Platform and tooling component logs.                                                                                                                                                      |

### Filtering

Each Logging dashboard has selectors at the top, and they map onto the labels every log line carries:

| Selector / label | What it narrows to                                                                                                                                                            |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `log_group`      | The broad category of log — `applications` for your Pinot components, `query_stats` for query log lines, `kubernetes_events` for pod-level events such as out-of-memory kills |
| `app_name`       | The component that wrote the line                                                                                                                                             |
| `pod_name`       | A single instance of that component                                                                                                                                           |
| `log_level`      | `INFO`, `WARN`, `ERROR`, and so on                                                                                                                                            |

Start broad and narrow down. `app_name` plus `log_level="ERROR"` over a wide time range is usually enough to find the window worth reading in detail, and `pod_name` then tells you whether the problem is one replica or all of them.

### Writing your own queries

The dashboards are built from LogQL you can adapt. In Grafana, choose **Explore** with the Loki data source.

Find every error from one application in the current time range:

```logql theme={null}
{log_group="applications", app_name="pinot-server", log_level="ERROR"}
```

Search for a string across a component's logs, regardless of level:

```logql theme={null}
{log_group="applications", app_name="pinot-server"} |= "OutOfMemoryError"
```

Count errors per pod, to see whether a problem is one replica or all of them:

```logql theme={null}
sum by (pod_name) (
  count_over_time({log_group="applications", app_name="pinot-server", log_level="ERROR"}[$__range])
)
```

Query-stats lines expose their fields individually, so you can filter on latency numerically:

```logql theme={null}
{log_group="query_stats"} | json | qs_timeMs > 5000
```

Query-stats lines expose `qs_requestId`, `qs_table`, `qs_timeMs`, `qs_docs`, `qs_entries`, and `qs_query` — the same values shown as columns on the query dashboards. `qs_requestId` is the one to carry between dashboards when drilling into a single slow query.

<Info>
  For anything beyond ad-hoc searching of query behaviour, prefer [`system_query_log`](/corecapabilities/query_data/advanced_operations/query-logger) over the `query_stats` log group. It is a queryable Pinot table with a much richer set of columns, and you can join and aggregate it with SQL. The log view is the fallback when the query logger is not enabled on your cluster.
</Info>

### Retention

Logs are retained for a bounded window, set for your environment — days rather than months. Confirm the exact retention with your StarTree team if you need to rely on it. Beyond that window the lines are gone, so for a problem you may need to explain later, export what you need while it is still in range: use **Inspect** → **Data** on a log panel, or copy the lines out of the log view.

This matters most for intermittent problems. If something fires monthly, do not plan to go back and look at the previous occurrence — capture each one as it happens.

## Pulling raw log files from an instance

When you need the unparsed file from one specific component — a full stack trace, a startup sequence, everything around a single second — use the Pinot controller's logger API. This is also the route when you need to hand a complete log file to support.

All calls go to your controller host and use the same authentication as the rest of the Pinot API; see [Manage API tokens](/corecapabilities/security/manage-api-tokens).

<Steps>
  <Step title="List the instances that have logs">
    ```
    GET /loggers/instances
    ```

    ```bash theme={null}
    curl -X GET "https://<your-controller-host>/loggers/instances" \
      -H "Authorization: Bearer <token>"
    ```

    See [Collect log files from all the instances](/api-reference/logger/collect-log-files-from-all-the-instances).
  </Step>

  <Step title="List the log files on one instance">
    ```
    GET /loggers/instances/{instanceName}
    ```

    Use the instance name exactly as it appears in the previous response — `Server_pinot-server-0.…_8098`, for example. See [Collect log files from a given instance](/api-reference/logger/collect-log-files-from-a-given-instance).

    For the controller you are already talking to, `GET /loggers/files` lists its local files directly ([reference](/api-reference/logger/get-all-local-log-files)).
  </Step>

  <Step title="Download the file">
    ```
    GET /loggers/instances/{instanceName}/download?filePath=<path>
    ```

    ```bash theme={null}
    curl -X GET "https://<your-controller-host>/loggers/instances/<instance>/download?filePath=<path>" \
      -H "Authorization: Bearer <token>" -o component.log
    ```

    See [Download a log file from a given instance](/api-reference/logger/download-a-log-file-from-a-given-instance), or [Download a log file](/api-reference/logger/download-a-log-file) for the local-controller equivalent.
  </Step>
</Steps>

<Warning>
  Log files are rotated. A file that existed when you listed it may be gone minutes later, and the file you want may already have rotated out — which is the main reason to search in Grafana first and reach for raw files only when you know what you are after.
</Warning>

## Raising a log level to reproduce something

If a problem is reproducible but the logs are not saying enough, you can raise a logger's level at runtime without a restart.

<Steps>
  <Step title="Find the logger">
    ```
    GET /loggers
    ```

    Returns every logger name currently configured. See [Get all the loggers](/api-reference/logger/get-all-the-loggers). `GET /loggers/{loggerName}` returns the level of one ([reference](/api-reference/logger/get-logger-configs)).
  </Step>

  <Step title="Raise the level">
    ```
    PUT /loggers/{loggerName}?level=DEBUG
    ```

    See [Set logger level](/api-reference/logger/set-logger-level).
  </Step>

  <Step title="Reproduce, collect, and put it back">
    Reproduce the problem, collect what you need from Grafana or the file API, then set the level back to `INFO`.
  </Step>
</Steps>

<Warning>
  **Always restore the level.** `DEBUG` on a busy component produces a very large volume of log lines, which consumes log retention for every other component sharing the environment and can shorten how far back you are able to search. Change the narrowest logger that covers the code you care about rather than the root logger, and change it back as soon as you have what you need.
</Warning>

## Task and subtask logs

For minion work, the Data Portal shows logs in context rather than making you search for them. Open a task plan and use its **Logs** tab, or open an individual subtask for its own log view alongside its progress. See [Task observability](/corecapabilities/observability/task-observability).

## What is not available to you

Some logs exist but are not exposed:

* **Logs from the managed platform itself.** The services StarTree runs to operate and manage your environment are not exposed here.
* **Cloud provider infrastructure logs.** Network and cloud-API audit logs for StarTree-managed infrastructure are not exposed through the Data Portal. In a BYOC or BYOK deployment the data plane runs in your own cloud account, so those logs are in your account and under your control.
* **Anything older than retention.** Once the window has passed, the data is deleted rather than archived.

If you have worked through the Logging dashboards and the file API and the answer is not there, that is itself a useful result to report — say so in the ticket, with the queries you ran and the window you searched, so support does not repeat the same search.

## Related

* [Grafana dashboards](/corecapabilities/observability/grafana) — where the Logging dashboards live
* [Query Logger](/corecapabilities/query_data/advanced_operations/query-logger) — `system_query_log`, the structured alternative for query analysis
* [Pinot audit logs](/corecapabilities/security/audit/pinot-audit-logs) — who did what to the cluster, as opposed to what it logged
* [Task observability](/corecapabilities/observability/task-observability) — logs for minion tasks
* [Troubleshooting: start here](/corecapabilities/observability/troubleshooting)