Plugin architecture

Plugin Overview

The LLMAI plugin provides the core AI integration capabilities for Unreal Engine 5 projects. This plugin handles all the low-level communication with OpenAI's, LocalAI's, and Grok's Realtime API, for text and audio processing, and provides Blueprint-friendly components for easy integration.

What's Included

Core Components

• ULLMAIClientComponent - Main AI communication component
• ULLMAIAudioStreamComponent - (Optional component) Advanced audio capture and streaming
• ULLMAIBlueprintLibrary - (Optional component) Utility functions and helpers
• ULLMAISettings - Plugin configuration system

Data Assets

• ULLMAIFunctionDefinitionAsset - Define AI-callable functions through the editor
• ULLMAIFunctionProfile - Group multiple function definitions together

Editor Integration

• Function Asset Factory - Create function definitions in Content Browser
• Project Settings Panel - Configure plugin settings through UI
• Blueprint Integration - Full Blueprint support for all features

Quick Integration

Add to Any Actor

// Add component to any Actor
UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = "AI")
class ULLMAIClientComponent* AIClient;

// In constructor
AIClient = CreateDefaultSubobject<ULLMAIClientComponent>(TEXT("AIClient"));

Basic Connection

// Connect to OpenAI (cloud)
AIClient->ConnectToAI(
    "OpenAI",
    "gpt-realtime", 
    "You are a helpful assistant", 
    {"text", "audio"}
);

// Or connect to LocalAI (local/offline)
AIClient->ConnectToAI(
    "LocalAI",
    "your-local-model", 
    "You are a helpful assistant", 
    {"text", "audio"}
);

// Or connect to Grok (xAI)
AIClient->ConnectToAI(
    "Grok",
    "grok-1118", 
    "You are a helpful assistant", 
    {"text", "audio"},
    0.8f,
    "Rex"  // Voice parameter - REQUIRED for Grok!
);

Handle AI Responses

// Bind to events
AIClient->OnAITextResponseDelta.AddDynamic(this, &AMyActor::HandleAIResponse);
AIClient->OnAIFunctionCallRequested.AddDynamic(this, &AMyActor::HandleFunctionCall);

Plugin Structure

The plugin can be used as-is without modifying C++ code (see Installation on your projects). Full source is included if you want to change or extend functionality optionally.

Demos are available as separate downloads from the Fab store LLMAI page.

YourProject/
└── Plugins/
    └── LLMAI/                           # Plugin root folder
        ├── Config/
        │   └── DefaultGame.ini          # Plugin configuration
        ├── Content/                     # Plugin content (currently empty)
        └── Source/
            ├── LLMAI/                   # Runtime module
            │   ├── Public/              # Public headers
            │   └── Private/             # Implementation files
            └── LLMAIEditor/             # Editor module
                ├── Public/
                └── Private/

Plugin Dependencies

Required Plugin Dependencies

• Built-in AudioCapture Plugin - For microphone input and voice communication Automatically enabled when LLMAI plugin is activated Required for voice mode functionality
• Built-in LiveLink Plugin - For lipsync to MetaHuman characters Automatically enabled when LLMAI plugin is activated Required for voice mode functionality
• Built-in MetaHuman Plugin - For creating and controlling MetaHuman characters Automatically enabled when LLMAI plugin is activated Required for voice mode functionality

Supported Platforms

• Windows 64-bit - Full support with all features
• Other Platforms - Not currently supported due to platform-specific audio libraries

Runtime Module Dependencies (LLMAI)

Core Unreal Modules

• Core - Basic Unreal Engine functionality
• CoreUObject - Object system integration
• Engine - Game engine features
• DeveloperSettings - Plugin configuration system
• Projects - Project file access

Networking & Communication

• WebSockets - Real-time AI communication
• HTTP - API requests and responses
• Json - JSON parsing and generation
• JsonUtilities - Advanced JSON operations

Audio Processing

• AudioCapture - Microphone input capture
• AudioMixer - Audio output and real-time processing

Platform-Specific Libraries

Windows (Win64) :

• winmm.lib - Windows Multimedia API for low-level audio
• dsound.lib - DirectSound for audio device management

Editor Module Dependencies (LLMAIEditor)

Core Editor Modules

• UnrealEd - Core editor functionality
• AssetTools - Asset creation and management
• Projects - Project settings integration

User Interface

• Slate - Core UI framework
• SlateCore - UI rendering system
• EditorStyle - Consistent editor styling
• EditorWidgets - Specialized editor controls
• ToolMenus - Context menu and toolbar integration

Settings Configuration

Project-wide Realtime configuration lives under Edit → Project Settings → Plugins → LLMAI. LLMAI exposes one Realtime API session model across three provider profiles (OpenAI, Grok, LocalAI). Most settings—general defaults, audio, and advanced connection options—are shared; only authentication and a few profile-specific fields differ.

For the full setup guide, settings priority stack (Project Settings vs plugin defaults vs API key sources), provider profiles, verification, and troubleshooting links, see Configuration reference:

• Project Settings catalog
• Provider profiles
• Settings priority

Note: Settings above are global project defaults. Per-component instance properties (Details panel) are documented below.

Component Instance Properties

When you add a ULLMAIClientComponent to an Actor, these properties are configurable in the Details panel . These are per-instance settings that can vary between different components.

Voice Settings

Property	Default	Range	Description
VoiceThreshold	0.01	0.001 - 1.0	Minimum audio level to detect speech. Lower = more sensitive.
MaxSilenceDuration	1.5	0.5 - 5.0 sec	How long to wait after speech stops before committing audio.
bEnableClientsideVAD	true	,	Detect microphone input and interrupt server responses. Essential for interrupting local audio playback.
InterruptionThreshold	0.01	0.001 - 0.1	Audio level needed to trigger voice interruption.
bEnableAutoMicGating	false	,	Automatically mute microphone when AI is speaking to prevent feedback.
OutputAudioThreshold	0.01	0.001 - 0.1	Output audio level threshold for mic gating when interruption is disabled.
OutputAudioDecayRate	0.95	0.8 - 0.999	How quickly output audio envelope decays. Higher = slower decay.
VoiceOutputHoldTimeSeconds	2.0	0.0 - 3.0 sec	Hold time after AI speech ends before triggering OnVoiceOutputEnd. Covers natural speech pauses.

Audio Settings

Property	Default	Range	Description
AudioGainMultiplier	1.0	0.1 - 10.0	Amplify or reduce microphone input volume before sending to AI.
bAutoCreateAudioStream	true	,	Automatically create audio stream component for microphone capture.
bAutoCreateAudioPlayback	true	,	Automatically create audio component for AI voice playback.

Usage Examples

// In Blueprint: Set via Details panel or use nodes
// In C++: Modify properties before connecting

AIClient->VoiceThreshold = 0.02f;          // Less sensitive speech detection
AIClient->MaxSilenceDuration = 2.0f;       // Wait longer before committing
AIClient->bEnableAutoMicGating = true;     // Prevent feedback
AIClient->AudioGainMultiplier = 1.5f;      // Boost quiet microphones

ULLMAIAudioStreamComponent Properties

If using a custom ULLMAIAudioStreamComponent , these properties can be configured:

Property	Default	Description
AudioSourceMode	Microphone	Microphone - Capture mic input to send to AI Loopback - Capture AI voice output for MetaHuman lip-sync
SampleRate	24000	Audio sample rate in Hz. 24000 is recommended for AI voice.
NumChannels	1	Number of audio channels (mono = 1, stereo = 2).
BitsPerSample	16	Audio bit depth. 16-bit PCM is standard for AI services.
StreamingChunkSize	4096	Size of audio chunks for streaming. Smaller = lower latency.
OutputFilePath	(empty)	Optional path to save captured audio to WAV file for debugging.

Function Calling System

Define AI Functions

Create function definition assets in the Content Browser:

1. Right-click in Content Browser
2. AI Functions > LLMAI Function Definition
3. Configure function name, description, and parameters
4. Register with AI client component

Define AI Function Profiles

A function profile is a set of AI Functions. This makes it easy to associate a set of functions with a particular purpose or connection.

1. Simply select the AI functions which relate to the profile
2. Register the profile with the AI client component

Handle Function Calls

UFUNCTION()
void HandleAIFunctionCall(const FLLMFunctionCall& FunctionCall)
{
    if (FunctionCall.Name == "my_function")
    {
        // Extract parameters
        FString Param = ULLMAIBlueprintLibrary::GetAIFunctionStringParameter(
            FunctionCall.ArgumentsJson, "parameter_name", "default_value"
        );
        
        // Execute your logic
        FString Result = ExecuteMyFunction(Param);
        
        // Return result to AI
        AIClient->SendAIFunctionCallResult(FunctionCall.CallId, Result);
    }
}

Audio Integration

The client component will automatically create needed audo components and begin and end streaming where necessary. You only need to set up the audio components manaually if you have a specific scenerio or settings you need to account for. For example if you wanted to connect the audio to a seperate actor or dynamically choose particular outputs and inputs.

Basic Audio Setup

// Setup voice capture (automatic microphone detection)
AIClient->SetupVoiceCapture();

// Setup audio playback
UAudioComponent* AudioComp = CreateDefaultSubobject<UAudioComponent>(TEXT("AIAudio"));
AIClient->SetupAudioPlayback(AudioComp);

Advanced Audio (Custom Capture)

// Create audio stream component for custom capture
ULLMAIAudioStreamComponent* StreamComp = CreateDefaultSubobject<ULLMAIAudioStreamComponent>(TEXT("AudioStream"));
AIClient->SetAudioStreamComponent(StreamComp);

// Handle custom audio data
StreamComp->OnAudioDataReceived.AddDynamic(this, &AMyActor::HandleAudioData);

Event System

Connection Events

AIClient->OnConnected.AddDynamic(this, &AMyActor::OnAIConnected);
AIClient->OnDisconnected.AddDynamic(this, &AMyActor::OnAIDisconnected);
AIClient->OnError.AddDynamic(this, &AMyActor::OnAIError);

Communication Events

AIClient->OnAISessionReady.AddDynamic(this, &AMyActor::OnAIReady);
AIClient->OnAITextResponseDelta.AddDynamic(this, &AMyActor::OnAITextReceived);
AIClient->OnAIResponseComplete.AddDynamic(this, &AMyActor::OnAIResponseFinished);

Voice Events

AIClient->OnVoiceModeActivated.AddDynamic(this, &AMyActor::OnVoiceModeStarted);
AIClient->OnInputAudioTranscriptionDelta.AddDynamic(this, &AMyActor::OnTranscription);

Thread Safety

The plugin is designed for thread-safe operation:

• WebSocket callbacks are marshaled to the game thread
• Audio processing occurs on background threads
• UI updates are synchronized to prevent conflicts
• Critical sections protect shared data structures

Always check IsInGameThread() before UI updates in custom handlers.

Debugging and Troubleshooting

The plugin includes a comprehensive debug & logging system for troubleshooting issues. See Debug-Logging.htm for complete details.

Quick Debug Commands

# In Unreal Console (` key):
llmai.debug.EnableAll              # Turn on all debugging
llmai.debug.OpenAILevel 2          # Debug API communication
llmai.debug.AudioLevel 2           # Debug voice issues
llmai.debug.FunctionLevel 2        # Debug function calling

Common Issues

• "WebSockets module not available" - Check module dependencies
• "Audio capture failed" - Use llmai.debug.AudioLevel 2 to diagnose
• "API key invalid" - Use llmai.debug.OpenAILevel 2 to see API errors (OpenAI), or verify LocalAI is running
• "Connection timeout" - Use llmai.debug.LogConnections 1 to monitor

Log Categories

Monitor these log categories in Output Log:

• LogLLMAI - General plugin information
• LogLLMAIWebSocket - WebSocket communication details
• LogLLMAIAudio - Audio processing information
• LogLLMAIFunctions - Function call execution

Full Debug & Logging Guide : See Debug-Logging.htm for complete troubleshooting commands and techniques.

Installation & Setup

For plugin installation, compilation, and initial configuration, see Installation.htm .