> ## Documentation Index
> Fetch the complete documentation index at: https://daily-docs-deploy-no-op-warm-pods-t3002.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Grok Realtime

> GrokRealtimeLLMService provides real-time multimodal speech-to-speech conversation using xAI's Grok Realtime API.

## Overview

`GrokRealtimeLLMService` provides real-time, multimodal conversation capabilities using xAI's Grok Voice Agent API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management with low-latency response times.

<CardGroup cols={2}>
  <Card title="Grok Realtime API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.xai.realtime.llm.html">
    Pipecat's API methods for Grok Realtime integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/realtime/realtime-grok.py">
    Complete Grok Realtime conversation example
  </Card>

  <Card title="Async Tool Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/realtime/realtime-grok-async-tool.py">
    Grok with async function calling (cancel\_on\_interruption=False)
  </Card>

  <Card title="Grok Voice Documentation" icon="book" href="https://docs.x.ai/docs/guides/voice/agent">
    Official xAI Grok Voice Agent API documentation
  </Card>

  <Card title="xAI Console" icon="external-link" href="https://console.x.ai/">
    Access Grok models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Grok Realtime services, install the required dependencies:

```bash theme={null}
uv add "pipecat-ai[grok]"
```

## Prerequisites

### xAI Account Setup

Before using Grok Realtime services, you need:

1. **xAI Account**: Sign up at [xAI Console](https://console.x.ai/)
2. **API Key**: Generate a Grok API key from your account dashboard
3. **Model Access**: Ensure access to Grok Voice Agent models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

* `XAI_API_KEY`: Your xAI API key for authentication

### Key Features

* **Real-time Speech-to-Speech**: Direct audio processing with low latency
* **Multilingual Support**: Support for multiple languages
* **Voice Activity Detection**: Server-side VAD for automatic speech detection
* **Function Calling**: Seamless support for external functions and tool integration
* **Multiple Voice Options**: Various voice personalities available
* **WebSocket Support**: Real-time bidirectional audio streaming

## Configuration

### GrokRealtimeLLMService

<ParamField path="api_key" type="str" required>
  xAI API key for authentication.
</ParamField>

<ParamField path="base_url" type="str" default="wss://api.x.ai/v1/realtime">
  WebSocket base URL for the Grok Realtime API. Override for custom deployments.
</ParamField>

<ParamField path="session_properties" type="SessionProperties" default="None" deprecated>
  Configuration properties for the realtime session. If `None`, uses default
  `SessionProperties` with voice `"eve"` and server-side VAD enabled. See
  [SessionProperties](#sessionproperties) below.

  *Deprecated in v0.0.105. Use `settings=GrokRealtimeLLMService.Settings(session_properties=...)` instead.*
</ParamField>

<ParamField path="settings" type="GrokRealtimeLLMService.Settings" default="None">
  Runtime-configurable settings. See [Settings](#settings) below.
</ParamField>

<ParamField path="start_audio_paused" type="bool" default="False">
  Whether to start with audio input paused.
</ParamField>

### Settings

Runtime-configurable settings passed via the `settings` constructor argument using `GrokRealtimeLLMService.Settings(...)`. These can be updated mid-conversation with `LLMUpdateSettingsFrame`. See [Service Settings](/pipecat/fundamentals/service-settings) for details.

| Parameter            | Type                | Default               | Description                                                                                          |
| -------------------- | ------------------- | --------------------- | ---------------------------------------------------------------------------------------------------- |
| `model`              | `str`               | `"grok-voice-latest"` | Model identifier. Defaults to xAI's recommended Voice Agent alias. *(Inherited from base settings.)* |
| `system_instruction` | `str`               | `NOT_GIVEN`           | System instruction/prompt. *(Inherited from base settings.)*                                         |
| `session_properties` | `SessionProperties` | `NOT_GIVEN`           | Session-level configuration (voice, audio config, tools, etc.).                                      |

<Note>
  `NOT_GIVEN` values are omitted, letting the service use its own defaults. Only
  parameters that are explicitly set are included.
</Note>

### SessionProperties

| Parameter        | Type                 | Default                            | Description                                                                                                                                                                                                |
| ---------------- | -------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `instructions`   | `str`                | `None`                             | System instructions for the assistant.                                                                                                                                                                     |
| `voice`          | `str`                | `"eve"`                            | Voice the model uses to respond. Accepts any built-in voice ID (see [xAI's voice catalogue](https://docs.x.ai/docs/guides/voice/agent)) or a custom voice ID from the Custom Voices API. Case-insensitive. |
| `turn_detection` | `TurnDetection`      | `TurnDetection(type="server_vad")` | Turn detection configuration. Set to `None` for manual turn detection.                                                                                                                                     |
| `audio`          | `AudioConfiguration` | `None`                             | Configuration for input and output audio formats.                                                                                                                                                          |
| `tools`          | `List[GrokTool]`     | `None`                             | Available tools: `web_search`, `x_search`, `file_search`, `mcp`, or custom `function` tools.                                                                                                               |
| `reasoning`      | `Reasoning`          | `None`                             | Reasoning effort controls. Set `effort="high"` to enable reasoning or `effort="none"` to disable.                                                                                                          |
| `resumption`     | `SessionResumption`  | `None`                             | Session resumption opt-in. Set `enabled=True` to cache conversation history for reconnection.                                                                                                              |
| `replace`        | `dict[str, str]`     | `None`                             | Pronunciation replacement map applied before TTS (e.g., `{"Acme": "Ack-mee"}`).                                                                                                                            |

### TurnDetection

The `turn_detection` field in `SessionProperties` configures voice activity detection and turn management:

| Parameter             | Type    | Default | Description                                                                        |
| --------------------- | ------- | ------- | ---------------------------------------------------------------------------------- |
| `type`                | `str`   | `None`  | Detection type: `"server_vad"` for automatic detection, `None` for manual control. |
| `threshold`           | `float` | `None`  | VAD activation threshold (0.1–0.9). Higher values require louder audio.            |
| `silence_duration_ms` | `int`   | `None`  | Milliseconds of silence before the server ends the user turn.                      |
| `prefix_padding_ms`   | `int`   | `None`  | Audio (ms) included before detected speech start.                                  |
| `idle_timeout_ms`     | `int`   | `None`  | Milliseconds of silence after assistant response before proactive check-in.        |

### AudioConfiguration

The `audio` field in `SessionProperties` accepts an `AudioConfiguration` with `input` and `output` sub-configurations:

**AudioInput** (`audio.input`):

| Parameter       | Type                        | Default | Description                                                                                                               |
| --------------- | --------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------- |
| `format`        | `AudioFormat`               | `None`  | Input audio format. Supports `PCMAudioFormat` (configurable rate), `PCMUAudioFormat` (8kHz), or `PCMAAudioFormat` (8kHz). |
| `transcription` | `InputAudioTranscription`   | `None`  | Transcription settings. Set `model="grok-transcribe"` for streaming user captions.                                        |
| `transport`     | `Literal["json", "binary"]` | `None`  | Wire path for input audio.                                                                                                |

**AudioOutput** (`audio.output`):

| Parameter   | Type                        | Default | Description                                        |
| ----------- | --------------------------- | ------- | -------------------------------------------------- |
| `format`    | `AudioFormat`               | `None`  | Output audio format. Same format options as input. |
| `speed`     | `float`                     | `None`  | Playback speed multiplier (0.7–1.5).               |
| `transport` | `Literal["json", "binary"]` | `None`  | Wire path for assistant audio.                     |

Grok PCM audio supports sample rates: 8000, 16000, 21050, 24000, 32000, 44100, and 48000 Hz.

**InputAudioTranscription** (`audio.input.transcription`):

| Parameter       | Type        | Default | Description                                                               |
| --------------- | ----------- | ------- | ------------------------------------------------------------------------- |
| `model`         | `str`       | `None`  | Transcription model. Use `"grok-transcribe"` for streaming user captions. |
| `language_hint` | `str`       | `None`  | BCP-47 language code to bias ASR (e.g., `"en-US"`).                       |
| `keyterms`      | `list[str]` | `None`  | Domain terms to bias transcription (max 100 terms, ≤50 chars each).       |

### Built-in Tools

Grok provides several built-in tools in addition to custom function tools:

| Tool             | Type          | Description                                                        |
| ---------------- | ------------- | ------------------------------------------------------------------ |
| `WebSearchTool`  | `web_search`  | Search the web for current information                             |
| `XSearchTool`    | `x_search`    | Search X (Twitter) for posts. Supports `allowed_x_handles` filter. |
| `FileSearchTool` | `file_search` | Search uploaded document collections by `vector_store_ids`         |
| `McpTool`        | `mcp`         | Remote MCP tool configuration managed by xAI                       |

## Usage

<Tip>
  Pair this service with `LLMContextAggregatorPair(context,
      realtime_service_mode=True)`. Realtime mode keeps context-writing correct for
  speech-to-speech services and adapts turn handling to the service. See
  [Realtime (Speech-to-Speech)
  Services](/api-reference/server/utilities/turn-management/external-turn-management#realtime-speech-to-speech-services).
</Tip>

### Basic Setup

```python theme={null}
import os
from pipecat.services.xai.realtime.llm import GrokRealtimeLLMService

llm = GrokRealtimeLLMService(
    api_key=os.getenv("XAI_API_KEY"),
)
```

### With Session Configuration

```python theme={null}
from pipecat.services.xai.realtime.llm import GrokRealtimeLLMService
from pipecat.services.xai.realtime.events import (
    SessionProperties,
    TurnDetection,
    AudioConfiguration,
    AudioInput,
    AudioOutput,
    PCMAudioFormat,
)

session_properties = SessionProperties(
    instructions="You are a helpful assistant.",
    voice="rex",
    turn_detection=TurnDetection(type="server_vad"),
    audio=AudioConfiguration(
        input=AudioInput(format=PCMAudioFormat(rate=16000)),
        output=AudioOutput(format=PCMAudioFormat(rate=16000)),
    ),
)

llm = GrokRealtimeLLMService(
    api_key=os.getenv("XAI_API_KEY"),
    settings=GrokRealtimeLLMService.Settings(
        session_properties=session_properties,
    ),
)
```

### With Built-in Tools

```python theme={null}
from pipecat.services.xai.realtime.llm import GrokRealtimeLLMService
from pipecat.services.xai.realtime.events import (
    SessionProperties,
    WebSearchTool,
    XSearchTool,
)

llm = GrokRealtimeLLMService(
    api_key=os.getenv("XAI_API_KEY"),
    settings=GrokRealtimeLLMService.Settings(
        session_properties=SessionProperties(
            instructions="You are a helpful assistant with access to web search.",
            voice="eve",
            tools=[
                WebSearchTool(),
                XSearchTool(allowed_x_handles=["@elonmusk"]),
            ],
        ),
    ),
)
```

### Updating Settings at Runtime

```python theme={null}
from pipecat.frames.frames import LLMUpdateSettingsFrame
from pipecat.services.xai.realtime.llm import GrokRealtimeLLMSettings
from pipecat.services.xai.realtime.events import SessionProperties

await worker.queue_frame(
    LLMUpdateSettingsFrame(
        delta=GrokRealtimeLLMSettings(
            session_properties=SessionProperties(
                instructions="Now speak in Spanish.",
                voice="eve",
            ),
        )
    )
)
```

<Tip>
  The deprecated `session_properties` constructor parameter is replaced by
  `Settings` as of v0.0.105. Use `Settings` / `settings=` instead. See the
  [Service Settings guide](/pipecat/fundamentals/service-settings) for migration
  details.
</Tip>

## Methods

### delete\_conversation\_item

```python theme={null}
await service.delete_conversation_item(item_id: str)
```

Delete a conversation item by ID. Sends a `conversation.item.delete` event to the server.

**Parameters:**

* `item_id` (`str`): ID of the conversation item to delete

### force\_message

```python theme={null}
await service.force_message(text: str)
```

Speak a hard-coded TTS line without involving the LLM. Sends a `force_message` conversation item. The server automatically injects the response lifecycle, so no `response.create` call is needed.

**Parameters:**

* `text` (`str`): Verbatim text to synthesize and play

## Notes

* **Model versioning**: The default model `"grok-voice-latest"` tracks xAI's recommended Voice Agent. For stability, pin a specific version in settings (e.g., `model="grok-voice-think-fast-1.0"`).
* **Audio format auto-configuration**: If audio format is not specified in `session_properties`, the service automatically configures PCM input/output using the pipeline's sample rates.
* **Server-side VAD**: Enabled by default. When VAD is enabled, the server handles speech detection and turn management automatically. Set `turn_detection` to `None` for manual turn detection.
* **Audio flow timing**: User audio flows after `session.updated` is received. Audio-only pipelines work without explicitly calling `_create_response`.
* **Interruption handling**: Interruptions send `response.cancel` to stop in-flight assistant audio on the wire and `conversation.item.truncate` to align server-side history with what the user heard. With server VAD enabled, the input buffer is preserved (not cleared) so interrupting user speech remains intact. Manual turn mode clears the input buffer on interruption.
* **Voice IDs**: Accepts any built-in voice ID from [xAI's catalogue](https://docs.x.ai/docs/guides/voice/agent) or a custom voice ID from the Custom Voices API. Voice IDs are case-insensitive. Default is `"eve"`.
* **G.711 support**: PCMU and PCMA formats are supported at a fixed 8000 Hz rate, useful for telephony integrations.
* **System instruction precedence**: The `system_instruction` from service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.
* **Async tool support**: Functions registered with `cancel_on_interruption=False` are supported. The final tool result is delivered via the formal tool-result channel once the async function completes. Streamed intermediate results (`is_final=False`) are not supported.

## Event Handlers

| Event                            | Parameters                                 | Description                                                   |
| -------------------------------- | ------------------------------------------ | ------------------------------------------------------------- |
| `on_conversation_item_created`   | `item_id`, `item`                          | Called when a new conversation item is created in the session |
| `on_conversation_item_updated`   | `item_id`, `item`                          | Called when a conversation item is updated or completed       |
| `on_conversation_item_deleted`   | `item_id`                                  | Called when a conversation item is deleted                    |
| `on_conversation_item_truncated` | `item_id`, `content_index`, `audio_end_ms` | Called when a conversation item is truncated                  |
| `on_idle_timeout`                | `item_id`                                  | Called when the idle timeout fires (proactive check-in)       |
| `on_dtmf_received`               | `button`                                   | Called when a DTMF digit is received (SIP sessions)           |
| `on_mcp_event`                   | `event_type`, `event`                      | Called for MCP discovery and call lifecycle events            |

```python theme={null}
@llm.event_handler("on_conversation_item_created")
async def on_item_created(service, item_id, item):
    print(f"New conversation item: {item_id}")

@llm.event_handler("on_conversation_item_updated")
async def on_item_updated(service, item_id, item):
    print(f"Conversation item updated: {item_id}")

@llm.event_handler("on_conversation_item_deleted")
async def on_item_deleted(service, item_id):
    print(f"Conversation item deleted: {item_id}")

@llm.event_handler("on_idle_timeout")
async def on_idle_timeout(service, item_id):
    print(f"Idle timeout triggered")

@llm.event_handler("on_dtmf_received")
async def on_dtmf(service, button):
    print(f"DTMF digit received: {button}")

@llm.event_handler("on_mcp_event")
async def on_mcp(service, event_type, event):
    print(f"MCP event: {event_type}")
```