Debug with Logs
Logs are enabled for every Haystack Platform pipeline and index by default.
Viewing Pipeline Logs
- In Haystack Enterprise Platform, go to Pipelines and click the name of the pipeline you want to debug.The pipeline's Analytics page opens.
- Switch to Analytics and open the Logs tab.
Viewing Index Logs
-
In Haystack Enterprise Platform, go to Indexes and click the name of the index you want to debug.
-
Switch to Logs.
- To view pipeline logs, choose Pipelines.
- To view index logs, choose Indexes.
-
Click the name of your pipeline or index to open its details.
-
Open the Logs tab.
You can filter the logs by message and pipeline type or date added. You can also search for keywords in log messages.
To download the logs, click Download CSV.
After you run a pipeline in Builder, you can see its real-time logs right there.
You can also open the trace for a query directly from Playground: click the View trace (bug) icon below the answer. For details, see Test Your Pipeline in Playground. To debug while building, use the logs in Builder. To learn how to do it, see Debug with Pipeline Builder.
To check logs for a specific trace, go to Analytics > Traces and open a trace. Then, open the span for the component you want to investigate and click View Logs. For details, see Trace Your Pipelines in the UI.
View Logs for a Specific Trace
Traces are collected for queries run on or after 25 June 2026. Before that date, traces are not available. To collect traces for pipelines deployed before that date, undeploy the pipeline and deploy it again.
You can check log entries generated during a specific pipeline run:
- In Haystack Enterprise Platform, go to Pipelines and click Analytics next to the pipeline you want to debug.
- Go to the Logs tab to view all logs generated during the trace.
Check Index Hardware Metrics
If an index runs slowly or fails, check its hardware metrics to see whether it's running out of CPU or memory.
- In Haystack Enterprise Platform, go to Indexes and click the name of the index you want to check.
- Click Metrics.
The Metrics tab shows CPU usage, memory usage, and replica count charts, grouped by pod. Use them together with the index logs to find the cause of a problem.
To confirm that indexing is running and to see how many batches the index processed over time, check the Executions chart on the same tab. Each batch of files that an index processes counts as one execution, including batches on indexes that don't pin a Haystack version. Failed batches count as errors, not as executions.
To tell whether an index is close to its memory limit, compare its memory usage with the limit for its pod size:
- Small: 4 GiB
- Medium: 8 GiB
- Large: 16 GiB
- XL: 32 GiB
An index without a pod size uses the default of 8 GiB. If the index has a memory limit set explicitly, that limit applies instead of the size. The CPU limit is only reported when the index has an explicit CPU limit. If memory usage stays close to the limit, set a larger pod size. For steps, see Set the Indexing Pod Size with the API.
Read the Replicas Chart
To find out when an index was running, for example to check whether it scaled to zero between runs, read the Replicas chart. It shows one band per replica, so you can see when each replica was running. An index that's scaled to zero between runs shows empty space between its bands, and the y axis shows only 0 and 1 for a single replica. For example, an index that ran for one minute at 4:21 p.m. and one minute at 6:36 p.m. shows two short bands with nothing between them. Hover over the chart to see which replicas were running at that moment and how many there were. The legend lists the same replicas. Each replica keeps the same color in the CPU usage, memory usage, and Replicas charts, so you can match a replica's CPU and memory usage to the time it was running.
Related Information
Was this page helpful?