Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Enable Streaming

Streaming refers to a large language model generating text as it's produced rather than waiting for the entire response to be ready before showing it. It's similar to watching someone type real-time. Enable streaming for the LLMs in your pipelines.

About This Task​

Streaming is a technique often used in chat interfaces. It makes the responses seem faster as users can immediately see the output and can start reading while the rest of the text generates. It also makes it possible to interrupt the LLM if needed. This is particularly useful for longer responses where waiting for the generation to complete may take a couple of seconds.

In Haystack Enterprise Platform, streaming is enabled by default for all LLMs.

Streaming in API endpoints​

When using Haystack Platform through the API, use the stream endpoints to get streaming responses. For example, use the Chat Stream endpoint to get a streaming response from a chat pipeline or the Search Stream endpoint to get a streaming response from a search pipeline. For request parameters, event types, and examples, see Stream Pipeline Responses.

Configure Streaming Components​

Use the top-level streaming_components field in your pipeline YAML to control which components stream their output. This is the recommended way to configure streaming. It works consistently across both the Playground and the API endpoints.

Enable Streaming for Specific Components​

To specify which components should stream, list their names in the streaming_components field:

max_runs_per_component: 100
streaming_components:
  - llm_1
  - llm_2
  - agent
components:
  llm_1:
    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
    # ... component configuration
  llm_2:
    type: haystack.components.generators.chat.openai.OpenAIChatGenerator
    # ... component configuration
  agent:
    type: haystack.components.agents.agent.Agent
    # ... component configuration

Enable Streaming for All Components​

To enable streaming for all compatible components in your pipeline, use the wildcard value:

max_runs_per_component: 100
streaming_components: all
components:
  # ... your pipeline components

Default Behavior​

If you don't include the streaming_components field, only the last streaming-capable component in your pipeline streams its output.

Legacy: Streaming with streaming_callback​

Legacy Approach

This approach is supported for backward compatibility with existing pipelines. For new pipelines, use the streaming_components field instead.

Older pipelines may have streaming enabled by setting the streaming_callback init parameter on individual components instead of using the streaming_components YAML field.

This approach only works predictably in the Playground. By default, the API renders agent tool calls the way the Playground does: include_tool_calls defaults to "rendered", which embeds tool calls as inline markdown in delta events rather than emitting separate tool_call_delta events. To get structured tool_call_delta events instead, set include_tool_calls to true in your stream request. See Stream request parameters for details.

For legacy pipelines that use streaming_callback, Hayhooks hybrid mode is enabled by default (allow_sync_streaming_callbacks: true). You can disable it by passing allow_sync_streaming_callbacks: false in your stream request.

For legacy pipelines, set streaming_callback: deepset_cloud_custom_nodes.callbacks.streaming.streaming_callback on each component you want to stream. All configured components receive streaming output through the API. This differs from the streaming_components field: when you omit that field, only the last streaming-capable component streams by default.

Generators and ChatGenerators​

CohereGenerator:
    type: haystack_integrations.components.generators.cohere.generator.CohereGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - COHERE_API_KEY
        - CO_API_KEY
        strict: false
      model: command-r
      streaming_callback: deepset_cloud_custom_nodes.callbacks.streaming.streaming_callback

Agents​

For agents, streaming_callback is set at the agent level, not on the inner chat_generator:

agent:
  type: haystack.components.agents.agent.Agent
  init_parameters:
    chat_generator:
      type: haystack.components.generators.chat.openai.OpenAIChatGenerator
      init_parameters:
        api_key:
          type: env_var
          env_vars:
          - OPENAI_API_KEY
          strict: false
        model: gpt-5.5
        streaming_callback: # leave empty on the inner generator
    system_prompt: You are a deep research assistant.
    streaming_callback: deepset_cloud_custom_nodes.callbacks.streaming.streaming_callback
    tools:
    # ...