Plugin Overview

The LLMAI plugin provides the core AI integration capabilities for Unreal Engine 5 projects. This plugin handles all the low-level communication with OpenAI's, LocalAI's, and Grok's Realtime API, for text and audio processing, and provides Blueprint-friendly components for easy integration.

What's Included

Core Components

• ULLMAIClientComponent - Main AI communication component
• ULLMAIAudioStreamComponent - (Optional component) Advanced audio capture and streaming
• ULLMAIBlueprintLibrary - (Optional component) Utility functions and helpers
• ULLMAISettings - Plugin configuration system

Data Assets

• ULLMAIFunctionDefinitionAsset - Define AI-callable functions through the editor
• ULLMAIFunctionProfile - Group multiple function definitions together

Editor Integration

• Function Asset Factory - Create function definitions in Content Browser
• Project Settings Panel - Configure plugin settings through UI
• Blueprint Integration - Full Blueprint support for all features

Quick Integration

Add to Any Actor

// Add component to any Actor
UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = "AI")
class ULLMAIClientComponent* AIClient;

// In constructor
AIClient = CreateDefaultSubobject<ULLMAIClientComponent>(TEXT("AIClient"));

Basic Connection

// Connect to OpenAI (cloud)
AIClient->ConnectToAI(
    "OpenAI",
    "gpt-realtime", 
    "You are a helpful assistant", 
    {"text", "audio"}
);

// Or connect to LocalAI (local/offline)
AIClient->ConnectToAI(
    "LocalAI",
    "your-local-model", 
    "You are a helpful assistant", 
    {"text", "audio"}
);

// Or connect to Grok (xAI)
AIClient->ConnectToAI(
    "Grok",
    "grok-1118", 
    "You are a helpful assistant", 
    {"text", "audio"},
    0.8f,
    "Rex"  // Voice parameter - REQUIRED for Grok!
);

Handle AI Responses

// Bind to events
AIClient->OnAITextResponseDelta.AddDynamic(this, &AMyActor::HandleAIResponse);
AIClient->OnAIFunctionCallRequested.AddDynamic(this, &AMyActor::HandleFunctionCall);

Plugin Structure

If you have downloaded the full project, then there are example demos and further documentation in the parent folders.

Plugin Dependencies

Required Plugin Dependencies

• Built-in AudioCapture Plugin - For microphone input and voice communication
  • Automatically enabled when LLMAI plugin is activated
  • Required for voice mode functionality
• Built-in LiveLink Plugin - For lipsync to MetaHuman characters
  • Automatically enabled when LLMAI plugin is activated
  • Required for voice mode functionality
• Built-in MetaHuman Plugin - For creating and controlling MetaHuman characters
  • Automatically enabled when LLMAI plugin is activated
  • Required for voice mode functionality

Supported Platforms

• Windows 64-bit - Full support with all features
• Other Platforms - Not currently supported due to platform-specific audio libraries

Runtime Module Dependencies (LLMAI)

Core Unreal Modules

• Core - Basic Unreal Engine functionality
• CoreUObject - Object system integration
• Engine - Game engine features
• DeveloperSettings - Plugin configuration system
• Projects - Project file access

Networking & Communication

• WebSockets - Real-time AI communication
• HTTP - API requests and responses
• Json - JSON parsing and generation
• JsonUtilities - Advanced JSON operations

Audio Processing

• AudioCapture - Microphone input capture
• AudioMixer - Audio output and real-time processing

Platform-Specific Libraries

Windows (Win64):

• winmm.lib - Windows Multimedia API for low-level audio
• dsound.lib - DirectSound for audio device management

Editor Module Dependencies (LLMAIEditor)

Core Editor Modules

• UnrealEd - Core editor functionality
• AssetTools - Asset creation and management
• Projects - Project settings integration

User Interface

• Slate - Core UI framework
• SlateCore - UI rendering system
• EditorStyle - Consistent editor styling
• EditorWidgets - Specialized editor controls
• ToolMenus - Context menu and toolbar integration

Settings Configuration

Access through Edit > Project Settings > Plugins > LLMAI:

Note: These are global project settings. For per-component instance settings (visible in the Details panel), see Component Instance Properties below.

General Settings

• Default Provider - Select between OpenAI (cloud) or LocalAI (local/offline)
• Default Instructions - Base prompt defining AI behavior

OpenAI Provider Settings (Project Settings > LLMAI > OpenAI Provider)

• OpenAI API Key - Your OpenAI API key for cloud service access
• Default Model - See DefaultGame.ini for available models (default: gpt-realtime)
• Default Voice - See DefaultGame.ini for available voices (default: alloy)

LocalAI Provider Settings (Project Settings > LLMAI > LocalAI Provider)

• Endpoint URL - LocalAI server endpoint address
• Default Model - Local model to use for AI processing
• TTS Model - Default Chatterbox, Text-to-Speech model for voice synthesis
• STT Model - Default Whisper, Speech-to-Text model for voice recognition
• Default Voice - Voice for TTS output

Note: LocalAI is available as a separate distribution. See LocalAI Quick Start Guide for download and setup.

Grok Provider Settings (Project Settings > LLMAI > Grok Provider)

• Grok API Key - Your xAI API key (from console.x.ai)
• Default Model - Model to use (default: grok-1118)
• Default Voice - Voice for audio responses (Ara, Rex, Sal, Eve, Leo)
• Enable Search - Enable Grok's live web/X search capability

API Key Discovery: The plugin automatically checks (in order):

1. Plugin Settings
2. Environment variables: XAI_API_KEY, LLMAI_GROK_API_KEY, GROK_API_KEY
3. Command line: -GrokKey=YOUR_KEY or -XAIKey=YOUR_KEY

Audio Settings

• Sample Rate - Audio quality (24000 Hz recommended)
• Voice Activity Detection - Automatic speech detection sensitivity
• Audio Buffer Size - Balance between latency and stability
• Audio Format - PCM16 format settings

Advanced Options

• Connection Timeout - WebSocket connection timeout duration
• Auto-Reconnect - Automatic reconnection on connection loss
• Debug Logging - Enable detailed logging for troubleshooting
• Audio File Saving - Save audio streams for debugging

Component Instance Properties

When you add a ULLMAIClientComponent to an Actor, these properties are configurable in the Details panel. These are per-instance settings that can vary between different components.

Voice Settings

Audio Settings

Usage Examples

// In Blueprint: Set via Details panel or use nodes
// In C++: Modify properties before connecting

AIClient->VoiceThreshold = 0.02f;          // Less sensitive speech detection
AIClient->MaxSilenceDuration = 2.0f;       // Wait longer before committing
AIClient->bEnableAutoMicGating = true;     // Prevent feedback
AIClient->AudioGainMultiplier = 1.5f;      // Boost quiet microphones

ULLMAIAudioStreamComponent Properties

If using a custom ULLMAIAudioStreamComponent, these properties can be configured:

Function Calling System

Define AI Functions

Create function definition assets in the Content Browser:

1. Right-click in Content Browser
2. AI Functions > LLMAI Function Definition
3. Configure function name, description, and parameters
4. Register with AI client component

Define AI Function Profiles

A function profile is a set of AI Functions. This makes it easy to associate a set of functions with a particular purpose or connection.

1. Simply select the AI functions which relate to the profile
2. Register the profile with the AI client component

Handle Function Calls

UFUNCTION()
void HandleAIFunctionCall(const FLLMFunctionCall& FunctionCall)
{
    if (FunctionCall.Name == "my_function")
    {
        // Extract parameters
        FString Param = ULLMAIBlueprintLibrary::GetAIFunctionStringParameter(
            FunctionCall.ArgumentsJson, "parameter_name", "default_value"
        );
        
        // Execute your logic
        FString Result = ExecuteMyFunction(Param);
        
        // Return result to AI
        AIClient->SendAIFunctionCallResult(FunctionCall.CallId, Result);
    }
}

Audio Integration

The client component will automatically create needed audo components and begin and end streaming where necessary. You only need to set up the audio components manaually if you have a specific scenerio or settings you need to account for. For example if you wanted to connect the audio to a seperate actor or dynamically choose particular outputs and inputs.

Basic Audio Setup

// Setup voice capture (automatic microphone detection)
AIClient->SetupVoiceCapture();

// Setup audio playback
UAudioComponent* AudioComp = CreateDefaultSubobject<UAudioComponent>(TEXT("AIAudio"));
AIClient->SetupAudioPlayback(AudioComp);

Advanced Audio (Custom Capture)

// Create audio stream component for custom capture
ULLMAIAudioStreamComponent* StreamComp = CreateDefaultSubobject<ULLMAIAudioStreamComponent>(TEXT("AudioStream"));
AIClient->SetAudioStreamComponent(StreamComp);

// Handle custom audio data
StreamComp->OnAudioDataReceived.AddDynamic(this, &AMyActor::HandleAudioData);

Event System

Connection Events

AIClient->OnConnected.AddDynamic(this, &AMyActor::OnAIConnected);
AIClient->OnDisconnected.AddDynamic(this, &AMyActor::OnAIDisconnected);
AIClient->OnError.AddDynamic(this, &AMyActor::OnAIError);

Communication Events

AIClient->OnAISessionReady.AddDynamic(this, &AMyActor::OnAIReady);
AIClient->OnAITextResponseDelta.AddDynamic(this, &AMyActor::OnAITextReceived);
AIClient->OnAIResponseComplete.AddDynamic(this, &AMyActor::OnAIResponseFinished);

Voice Events

AIClient->OnVoiceModeActivated.AddDynamic(this, &AMyActor::OnVoiceModeStarted);
AIClient->OnInputAudioTranscriptionDelta.AddDynamic(this, &AMyActor::OnTranscription);

Thread Safety

The plugin is designed for thread-safe operation:

• WebSocket callbacks are marshaled to the game thread
• Audio processing occurs on background threads
• UI updates are synchronized to prevent conflicts
• Critical sections protect shared data structures

Always check IsInGameThread() before UI updates in custom handlers.

Debugging and Troubleshooting

The plugin includes a comprehensive debug & logging system for troubleshooting issues. See Debug-Logging.htm for complete details.

Quick Debug Commands

# In Unreal Console (` key):
llmai.debug.EnableAll              # Turn on all debugging
llmai.debug.OpenAILevel 2          # Debug API communication
llmai.debug.AudioLevel 2           # Debug voice issues
llmai.debug.FunctionLevel 2        # Debug function calling

Common Issues

• "WebSockets module not available" - Check module dependencies
• "Audio capture failed" - Use llmai.debug.AudioLevel 2 to diagnose
• "API key invalid" - Use llmai.debug.OpenAILevel 2 to see API errors (OpenAI), or verify LocalAI is running
• "Connection timeout" - Use llmai.debug.LogConnections 1 to monitor

Log Categories

Monitor these log categories in Output Log:

• LogLLMAI - General plugin information
• LogLLMAIWebSocket - WebSocket communication details
• LogLLMAIAudio - Audio processing information
• LogLLMAIFunctions - Function call execution

Full Debug & Logging Guide: See Debug-Logging.htm for complete troubleshooting commands and techniques.

Installation & Setup

For plugin installation, compilation, and initial configuration, see Installation.htm.

Changelog

Support

• Installation - See Installation.htm for setup and troubleshooting
• API Reference - See API.htm for comprehensive function documentation
• Debug Guide - See Debug-Logging.htm for troubleshooting

---

Engineered by