📚 API Reference

The LLMAI plugin provides AI integration for Unreal Engine 5, featuring real-time voice communication, text processing, and function calling capabilities.

Supports OpenAI (cloud) and LocalAI (local/offline). Future releases will add additional providers such as Claude and Gemini.
📋 Version 2.0 Changes: Functions and events marked with v2.0 are new in this release. LocalAI provider support, advanced voice control, reasoning model events, and LiveLink integration have been added.

📋 API Contents

🔌 Provider Functions

🎯 Core Components

📡 Connection & Communication

🔧 Function Calling

📢 Events

🎵 Audio

⚙️ Settings & Utilities

Provider-Based Functions (Recommended)

Multi-Provider Support: The plugin provides provider-agnostic functions that work with multiple AI services. Currently supported providers are OpenAI (cloud) and LocalAI (local/offline). These functions are located in ULLMAIBlueprintLibrary and are the recommended approach for flexibility.

Provider Discovery

Get Available AI Providers

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static TArray<FString> GetAvailableAIProviders();
// Returns: ["OpenAI", "LocalAI"]
// Future: Additional providers (Claude, Gemini, etc.)

Default Provider Management

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static FString GetDefaultAIProvider();

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
static void SetDefaultAIProvider(const FString& Provider);

Model Management

Model Discovery & Validation

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static TArray<FString> GetKnownAIModels(const FString& Provider);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static bool IsValidAIModel(const FString& Provider, const FString& Model);

Default Model Management

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static FString GetDefaultAIModel(const FString& Provider);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
static void SetDefaultAIModel(const FString& Provider, const FString& Model);

Voice Management

Voice Discovery & Validation

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static TArray<FString> GetKnownAIVoices(const FString& Provider);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static bool IsValidAIVoice(const FString& Provider, const FString& Voice);

Default Voice Management

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
static FString GetDefaultAIVoice(const FString& Provider);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
static void SetDefaultAIVoice(const FString& Provider, const FString& Voice);

Core Components

1. ULLMAIClientComponent

The main component for AI communication and session management.

Connection Management

Provider-Agnostic Connection (Recommended)

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void ConnectToAI(const FString& Provider,
                    const FString& Model, 
                    const FString& Instructions, 
                    const TArray<FString>& Modalities,
                    float Temperature = 0.8f);
// Provider: "OpenAI" or "LocalAI"
// Modalities: ["text"], ["audio"], or ["text", "audio"]

Session State Functions

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
bool IsAISession() const;
// True if connected to any AI provider

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI", BlueprintPure)
bool IsAISessionReady() const;
// True if session is ready for communication

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
ELLMAISessionState GetCurrentSessionState() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsAIActivelyResponding() const;

Basic WebSocket Connection

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void Connect(const FString& URL, const FString& Protocol = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void Disconnect();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsConnected() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
bool IsConnecting() const;

Provider-Specific (Internal)

// These are called internally by ConnectToAI
void ConnectToOpenAI(const FString& Model, 
                         const FString& Instructions, 
                         const TArray<FString>& Modalities,
                         float Temperature = 0.8f);

void ConnectToLocalAI(const FString& Model, 
                         const FString& Instructions, 
                         const TArray<FString>& Modalities,
                         float Temperature = 0.8f);

Session State Enum

ELLMAISessionState

UENUM(BlueprintType)
enum class ELLMAISessionState : uint8
{
    Disconnected,
    Connecting,
    Connected,
    SessionCreating,
    SessionReady,
    VoiceStarting,
    VoiceActive,
    Error
};

Voice Communication

Voice Mode Control

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
bool StartVoiceMode(const FString& Voice = TEXT(""),
                       const FString& TTSModel = TEXT("")); // v2.0: TTSModel param

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
bool EndVoiceMode(bool bClearConversation = true);

Audio Data Transmission

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void SendAudioToAI(const TArray<float>& AudioSamples);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void CommitAudioBuffer();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void TruncateAudioBuffer(int32 AudioEndMs = 0);

Microphone Control v2.0

Manual Mute Control

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetMicrophoneMuted(bool bMuted);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void ToggleMicrophoneMute();

UFUNCTION(BlueprintPure, Category = "LLMAI|Voice")
bool IsMicrophoneMuted() const;

Auto-Gating Status

UFUNCTION(BlueprintPure, Category = "LLMAI|Voice")
bool IsMicrophoneAutoGated() const;
// Returns true when mic is auto-muted because AI is speaking

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void StopBufferedAudioPlayback();
// Stop playback when AI finished but audio still buffered

Text Communication

Provider-Agnostic Text (Recommended)

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void SendTextToAI(const FString& Text, 
                      bool bUseTextModality = true, 
                      bool bUseAudioModality = true);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI")
void AddContextToAI(const FString& ContextText);

AI Response Control

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerAIResponse(const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void InterruptAIResponse();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void CancelResponse();

Modality-Specific Response v2.0

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerAIResponseWithModalities(
    const TArray<FString>& Modalities,
    const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
void TriggerTextOnlyResponse(const FString& Instructions = TEXT(""));

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void TriggerAudioOnlyResponse(const FString& Instructions = TEXT(""));

Legacy Provider-Specific Functions

// These still work but use generic versions above instead
void SendTextToOpenAI(const FString& Text, 
                           bool bUseTextModality = true, 
                           bool bUseAudioModality = true);

void AddContextToOpenAI(const FString& ContextText);
void SendTextToLocalAI(...);  // Same signature
void AddContextToLocalAI(...); // Same signature

Function Calling

Function Registration

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void RegisterAIFunction(const FLLMFunctionDefinition& FunctionDefinition);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void UnregisterAIFunction(const FString& FunctionName);

Function Call Handling

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions")
void SendAIFunctionCallResult(const FString& CallId, 
                                    const FString& Result);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Functions", BlueprintPure)
TArray<FString> GetRegisteredAIFunctionNames() const;

Function Profiles

Profile Management

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles")
void RegisterFunctionProfile(ULLMAIFunctionProfile* Profile);

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles")
void RegisterFunctionDefinitionAsset(ULLMAIFunctionDefinitionAsset* FunctionAsset);

Profile Queries

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles", BlueprintPure)
bool IsFunctionProfileLoaded(const FString& ProfileName) const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|AI Function Profiles", BlueprintPure)
TArray<FString> GetLoadedProfileNames() const;

Audio Setup

Voice Capture & Playback

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetupVoiceCapture(UAudioCaptureComponent* MicComponent = nullptr);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice")
void SetupAudioPlayback(UAudioComponent* AudioComponent, 
                            bool bIgnoreFlushing = true);

Audio Streaming Integration

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void SetAudioStreamComponent(ULLMAIAudioStreamComponent* AudioComponent);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void StartAudioStreaming();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream")
void StopAudioStreaming();

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioStreaming() const;

Auto-Creation Status

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioStreamAutoCreated() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Stream", BlueprintPure)
bool IsAudioPlaybackAutoCreated() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Voice", BlueprintPure)
bool IsAudioPlaying() const;

Response & Session Identifiers

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
FString GetCurrentSessionIdentifier() const;

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
FString GetCurrentResponseId() const;

Event Delegates

Connection Events

Socket Connection Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Socket")
FOnSocketConnected OnConnected;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Socket")
FOnSocketDisconnected OnDisconnected;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Socket")
FOnSocketError OnError;

AI Communication Events

Session Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnAISessionReady OnAISessionReady;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceModeActivated OnVoiceModeActivated;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceSessionComplete OnVoiceSessionComplete;

Text Response Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Text Response")
FOnAITextResponseBegin OnAITextResponseBegin;
// (ResponseId)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Text Response")
FOnAITextResponseDelta OnAITextResponseDelta;
// (DeltaText) - streaming text chunks

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Text Response")
FOnAITextComplete OnAITextComplete;
// (FullText) - complete text when done

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Text Response")
FOnAIResponseComplete OnAIResponseComplete;
// (ResponseId) - fires after text/audio complete

Transcription Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Transcription")
FOnInputAudioTranscriptionDelta OnInputAudioTranscriptionDelta;
// (ItemId, DeltaText)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Transcription")
FOnInputAudioTranscriptionCompleted OnInputAudioTranscriptionCompleted;
// (ItemId, FullTranscript)

Conversation & Voice Session Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Conversation")
FOnConversationItemCreated OnConversationItemCreated;
// (ItemId, Role, ItemType)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceSessionComplete OnVoiceSessionComplete;
// (GeneratedAudioFile) - USoundWave* with recorded audio

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceSessionProgress OnVoiceSessionProgress;
// (StatusMessage)

Voice State Events v2.0

Microphone Control Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnMicrophoneMuted OnMicrophoneMuted;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnMicrophoneUnmuted OnMicrophoneUnmuted;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnMicrophoneAutoGated OnMicrophoneAutoGated;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnMicrophoneAutoUngated OnMicrophoneAutoUngated;

Voice Output Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceOutputBegin OnVoiceOutputBegin;
// Fires when AI starts speaking

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnVoiceOutputEnd OnVoiceOutputEnd;
// Fires when AI finishes speaking

Server VAD Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnServerVADSpeechStarted OnServerVADSpeechStarted;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnServerVADSpeechStopped OnServerVADSpeechStopped;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Voice")
FOnServerVADAudioCommitted OnServerVADAudioCommitted;

Reasoning Model Events v2.0

Multi-Step Reasoning: These events support thinking/reasoning models that produce intermediate content before final output. Some LocalAI thinking and reasoning models produce thinking output.

Output Item Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Reasoning")
FOnAIOutputItemAdded OnAIOutputItemAdded;
// (ItemId)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Reasoning")
FOnAIContentPartAdded OnAIContentPartAdded;
// (ItemId, ContentIndex, ContentType)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Reasoning")
FOnAIOutputItemComplete OnAIOutputItemComplete;
// (ItemId, ContentJson)

Reasoning Content Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Reasoning")
FOnAIReasoningDelta OnAIReasoningDelta;
// (ItemId, ReasoningDelta)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Reasoning")
FOnAIReasoningComplete OnAIReasoningComplete;
// (ItemId, FullReasoning)

Function Calling Events

Function Call Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|AI Functions")
FOnAIFunctionCallRequested OnAIFunctionCallRequested;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|AI Functions")
FOnAIFunctionCallResult OnAIFunctionCallResult;

// v2.0: Function Profile Events
UPROPERTY(BlueprintAssignable, Category = "LLMAI|Function Profiles")
FOnAIFunctionProfileLoaded OnFunctionProfileLoaded;

UPROPERTY(BlueprintAssignable, Category = "LLMAI|Function Profiles")
FOnAIFunctionProfileLoadFailed OnFunctionProfileLoadFailed;

Error Events

AI Service Error Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|OpenAI")
FOnAIServerError OnAIServerError;
// (ErrorType, ErrorCode, ErrorMessage, EventId)

UPROPERTY(BlueprintAssignable, Category = "LLMAI|OpenAI")
FOnAIRateLimitExceeded OnAIRateLimitExceeded;
// (RateLimitType, ResetSeconds, Remaining)

LocalAI Events v2.0

UPROPERTY(BlueprintAssignable, Category = "LLMAI|LocalAI")
FOnContextCompacted OnContextCompacted;
// (TokensAfter, ContextUsagePercent, Message)
// Fires when LocalAI compacts context to free memory

Data Structures

Function Definition

FLLMFunctionDefinition

USTRUCT(BlueprintType)
struct FLLMFunctionDefinition
{
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Definition")
    FString Name;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Definition")
    FString Description;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Definition")
    TArray<FLLMFunctionParameter> Parameters;
};

Function Parameter

FLLMFunctionParameter

USTRUCT(BlueprintType)
struct FLLMFunctionParameter
{
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    FString Name;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    ELLMFunctionParameterType Type;  // String, Number, Integer, Boolean, Array, Object

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    FString Description;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    bool bIsRequired;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    FString DefaultValue;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Parameter")
    TArray<FString> AllowedValues;  // Optional enum values
};

Parameter Type Enum

ELLMFunctionParameterType

UENUM(BlueprintType)
enum class ELLMFunctionParameterType : uint8
{
    String,
    Number,
    Integer,
    Boolean,
    Array,
    Object
};

AI Provider Enum v2.0

EAIProvider

UENUM(BlueprintType)
enum class EAIProvider : uint8
{
    None,       // Not connected to any provider
    OpenAI,     // OpenAI cloud service
    LocalAI     // LocalAI local/offline service
};

Function Call

FLLMFunctionCall

USTRUCT(BlueprintType)
struct FLLMFunctionCall
{
    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Call")
    FString CallId;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Call")
    FString Name;

    UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Function Call")
    FString ArgumentsJson;
};

Audio Utilities

Audio Processing Functions

Format Conversion

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Utils")
TArray<float> ConvertPCM16ToFloat(const TArray<uint8>& PCM16Data);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Utils")
TArray<uint8> ConvertFloatToPCM16(const TArray<float>& FloatData);

Audio Analysis

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Utils")
float CalculateAudioLevel(const TArray<float>& AudioSamples);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Utils")
TArray<float> ApplyBasicNoiseReduction(const TArray<float>& AudioSamples, 
                                                float NoiseFloor = 0.02f);

Audio Generation

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Generation")
USoundWave* CreateSoundWaveFromPCM16(
    const TArray<uint8>& PCM16Data, 
    const FString& SoundName = TEXT("GeneratedAudio"));

Audio Filters

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Filters")
void SetupAudioFilters(float LowPassFreq = 0.0f, 
                          float HighPassFreq = 0.0f, 
                          bool bEnableFilters = true);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Filters")
void SetFilterFrequencies(float LowPassFreq, float HighPassFreq);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Filters")
void EnableAudioFilters(bool bEnabled);
MetaHuman Lip-Sync: The LiveLink audio integration automatically routes AI voice output to MetaHuman characters for real-time lip-sync animation. Audio is sent to a LiveLink subject that drives facial animation.

LiveLink Events

UPROPERTY(BlueprintAssignable, Category = "LLMAI|LiveLink")
FOnLiveLinkAudioInitialized OnLiveLinkAudioInitialized;
// (SubjectName) - Fires when LiveLink audio source is ready

External Audio Input

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio Input")
void ReceiveExternalAudioData(const TArray<float>& AudioSamples);
// For UE 5.5.4 compatibility - feed audio from external sources

Settings Access Functions v2.0

Plugin Settings

UFUNCTION(BlueprintCallable, BlueprintPure, Category = "LLMAI|Settings")
FString GetDefaultInstructions(const FString& Provider) const;

UFUNCTION(BlueprintCallable, BlueprintPure, Category = "LLMAI|Settings")
FString GetDefaultOpenAIAPIKey() const;

UFUNCTION(BlueprintCallable, BlueprintPure, Category = "LLMAI|Settings")
FString GetDefaultInputAudioFormat() const;

UFUNCTION(BlueprintCallable, BlueprintPure, Category = "LLMAI|Settings")
FString GetDefaultOutputAudioFormat() const;

UI Safety

UFUNCTION(BlueprintCallable, BlueprintPure, Category = "LLMAI|Utilities")
bool CanSafelyUpdateUI() const;
// Returns true if safe to update UI (world exists, not tearing down)
// Use before updating widgets in event handlers

ULLMAIBlueprintLibrary

Static utility functions for common operations.

Component Creation

Create Components

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio")
static ULLMAIAudioStreamComponent* CreateAudioStreamComponent(
    AActor* Actor, const FString& ComponentName);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket")
static ULLMAIClientComponent* CreateSocketClientComponent(
    AActor* Actor, const FString& ComponentName);

Get Existing Components

UFUNCTION(BlueprintCallable, Category = "LLMAI|Audio", BlueprintPure)
static ULLMAIAudioStreamComponent* GetAudioStreamComponent(AActor* Actor);

UFUNCTION(BlueprintCallable, Category = "LLMAI|Socket", BlueprintPure)
static ULLMAIClientComponent* GetLLMAIClientComponent(AActor* Actor);
📖 Plugin Documentation 🐛 Debug Guide