Streams

Streaming is a powerful feature when using LLMs that allows you to process chunks of an LLM response in real-time as they are generated. This can be particularly useful for long-running tasks, providing immediate feedback to users, or implementing more responsive applications.

This approach offers several benefits:

1. Immediate feedback: Users can see responses as they're being generated, creating a more interactive experience.
2. Reduced latency: For long responses, users don't have to wait for the entire generation to complete before seeing results.
3. Incremental processing: Applications can process and act on partial results as they arrive.
4. Efficient resource use: Memory usage can be optimized by processing chunks instead of storing the entire response.
5. Early termination: If the desired information is found early in the response, processing can be stopped without waiting for the full generation.

Basic Usage and Syntax

To use streaming, simply set the stream parameter to True in your call decorator:

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True) 
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy") 
for chunk, _ in stream: 
    print(chunk.content, end="", flush=True) 

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True) 
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy") 
for chunk, _ in stream: 
    print(chunk.content, end="", flush=True) 

In this example:

1. We use the call decorator with stream=True to enable streaming.
2. The recommend_book function now returns a generator that yields (chunk, tool) tuples of the response.
3. We iterate over the chunks, printing each one as it's received.
4. We use end="" and flush=True parameters in the print function to ensure that the output is displayed in real-time without line breaks.

Handling Streamed Responses

When streaming, the initial response will be a provider-specific BaseStream instance (e.g. OpenAIStream), which is a generator that yields tuples (chunk, tool) where chunk is a provider-specific BaseCallResponseChunk (e.g. OpenAICallResponseChunk) that wraps the original chunk in the provider's response. These objects provide a consistent interface across providers while still allowing access to provider-specific details.

Common Chunk Properties and Methods

All BaseCallResponseChunk objects share these common properties:

• content: The main text content of the response. If no content is present, this will be the empty string.
• finish_reasons: A list of reasons why the generation finished (e.g., "stop", "length"). These will be typed specifically for the provider used. If no finish reasons are present, this will be None.
• model: The name of the model used for generation.
• id: A unique identifier for the response if available. Otherwise this will be None.
• usage: Information about token usage for the call if available. Otherwise this will be None.
• input_tokens: The number of input tokens used if available. Otherwise this will be None.
• output_tokens: The number of output tokens generated if available. Otherwise this will be None.

Common Stream Properties and Methods

Once exhausted, all BaseStream objects share the same common properties and methods as BaseCallResponse, except for usage, tools, tool, and __str__.

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(chunk.content, end="", flush=True)

print(f"Content: {stream.content}") 

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(chunk.content, end="", flush=True)

print(f"Content: {stream.content}") 

You can access the additional missing properties by using the method construct_call_response to reconstruct a provider-specific BaseCallResponse instance:

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(chunk.content, end="", flush=True)

print(f"Content: {stream.content}")

call_response = stream.construct_call_response() 
print(f"Usage: {call_response.usage}") 

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(chunk.content, end="", flush=True)

print(f"Content: {stream.content}")

call_response = stream.construct_call_response() 
print(f"Usage: {call_response.usage}") 

Provider-Specific Response Details

While Mirascope provides a consistent interface, you can always access the full, provider-specific response object if needed. This is available through the chunk property of the BaseCallResponseChunk object:

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(f"Original chunk: {chunk.chunk}") 
    print(chunk.content, end="", flush=True)

from mirascope import llm


@llm.call(provider="openai", model="gpt-4o-mini", stream=True)
def recommend_book(genre: str) -> str:
    return f"Recommend a {genre} book"


stream = recommend_book("fantasy")
for chunk, _ in stream:
    print(f"Original chunk: {chunk.chunk}") 
    print(chunk.content, end="", flush=True)

Multi-Modal Outputs

While most LLM providers focus on text streaming, some providers support streaming additional output modalities like audio. The availability of multi-modal streaming varies among providers:

Audio Streaming

For providers that support audio outputs, you can stream both text and audio responses simultaneously:

Each stream chunk provides access to:

• chunk.audio: Raw audio data in bytes format
• chunk.audio_transcript: The transcript of the audio

This allows you to process both text and audio streams concurrently. Since audio data is received in chunks, you could technically begin playback before receiving the complete response.

Error Handling

Error handling in streams is similar to standard non-streaming calls. However, it's important to note that errors may occur during iteration rather than at the initial function call:

In these examples we show provider-specific error handling, though you can also catch generic exceptions.

Note how we wrap the iteration loop in a try/except block to catch any errors that might occur during streaming.

Next Steps

By leveraging streaming effectively, you can create more responsive and efficient LLM-powered applications with Mirascope's streaming capabilities.

Next, we recommend taking a look at the Chaining documentation, which shows you how to break tasks down into smaller, more directed calls and chain them together.