Skip to main content

Getting started

Upgrading the plugin?

If your project uses an older version of LLMAI, installing a newer plugin version does not automatically refresh your saved Project Settings. Values in {Project}/Config/DefaultGame.ini stay as-is unless you reset them.

Before opening the project with the new plugin version: back up DefaultGame.ini, then delete the [/Script/LLMAI.LLMAISettings] section. On first launch, LLMAI reloads defaults from the shipped plugin configuration. Re-enter API keys and any custom provider settings afterward.

See Changelog — Upgrading from a previous version for why this is needed. Step-by-step details: Configuration reference — Upgrading after a plugin update.

Step 1: Plugin install options​

Epic Games Launcher​

After purchase on Fab.com, go to your Fab library in the Launcher, click Install to Engine, pick your UE5 version, and it lands in the engine's Plugins folder (for example C:\Program Files\EpicGames\UE_5.x\Engine\Plugins\Marketplace).

Close your project and reopen it. Go to Edit → Plugins, find or search for LLMAI, and click to enable it. You will now have a new LLMAI section under Edit → Project Settings → Plugins → LLMAI. See Configuration quick-start for setting up OpenAI, Grok, and LocalAI for your project.

Manual install​

Optional — advanced users only

This path copies plugin C++ source into your project so you can modify it. It is not required for normal use and is not recommended for beginners. Use Epic Games Launcher above unless you specifically need to change plugin source code.

If you want to modify or extend the included C++ code, use Install to Engine from the Fab library as above, then copy the LLMAI folder from the engine Plugins directory into your project's Plugins/ folder (for example from Engine\Plugins\Marketplace\LLMAI to YourProject\Plugins\LLMAI). Right-click your .uproject file and choose Generate Visual Studio project files, then open the solution in Visual Studio and rebuild your project.

For more detailed information, see the Installation on your projects page.

Step 2: Try the Demos​

LLMAI comes in two demo projects. Both use the same Terminal interface but showcase different AI capabilities:

The demos are available as separate downloads on the Fab store LLMAI page. Download and extract each zip to your preferred Unreal projects folder. Inside you will find the .uproject file you can open to run the demo. After opening a demo for the first time, follow Configuration quick-start to set up your preferred provider (OpenAI, Grok, or LocalAI).

LLMAI_Arcade (Rocketeer Game)​

  1. Open Level : Content/LLMAITerminal/Levels/LVL_LLMAITerminal
  2. Click Play (or press Alt+P) and connect to AI
  3. Enable Voice Mode and say "Launch Rocketeer"
  4. Control your rocket : Arrows to move, spacebar to fire
  5. Try AI commands : "Make it faster", "Give me an extra life", "Spawn some enemies"
  6. Advance stages : Ask AI to set difficulty and progress

Learn More: Rocketeer Documentation

  1. Open Level : Content/LLMAITerminal/Levels/LVL_LLMAITerminal
  2. Click Play (or press Alt+P) and connect to AI
  3. Enable Voice Mode to see lip-sync in action
  4. Try gesture commands : "Wave hello", "Give me a thumbs up", "Nod yes"
  5. Try movement : "Move to the left", "Go to the right", "Jump!"
  6. Have a conversation and watch natural lip-sync

Learn More: Live Link demo

Step 3: Enable in your project​

If you already enabled LLMAI in Step 1, skip to Step 4.

  1. Open your Unreal Engine project
  2. Go to Edit → Plugins
  3. Search for "LLMAI"
  4. Check the Enabled checkbox
  5. Click Yes when prompted to compile the plugin
  6. Restart the editor when prompted
note

The plugin and its dependencies will be enabled in Unreal Engine 5 on reboot.

Step 4: Use AI in Your Project​

Blueprint Method (Easy)​

  1. Open any Blueprint Actor
  2. Add Component : Search for "LLMAI Client Component"
  3. In Begin Play : Use Connect to AI node Set Instructions: You are a helpful assistant Set Modalities: text, audio
  4. Send Text to AI : Use Send Text to AI node to trigger a conversation Connect a string input (e.g., "Hello, how are you?") This automatically triggers an AI response (no additional trigger needed) This can be triggered by user input, button press, or any game event
  5. Handle Responses : Easiest Method: Bind to On AI Text Response Complete event This provides the full response when complete (recommended for most use cases) Streaming Method (Optional): For real-time streaming text: Bind to On AI Text Response Start event (response begins) Bind to On AI Text Response Delta event (each text chunk) Bind to On AI Text Response End event (response complete) Print the response or update your UI

C++ Method (Advanced)​

// In your Actor's header file
UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = "AI")
class ULLMAIClientComponent* AIClient;

// In constructor
AIClient = CreateDefaultSubobject<ULLMAIClientComponent>(TEXT("AIClient"));

// In BeginPlay
AIClient->OnAITextResponseComplete.AddDynamic(this, &AMyActor::HandleAIResponse);
TArray<FString> Modalities = {"text", "audio"};

// Connect to AI (OpenAI example)
AIClient->ConnectToAI(
    "OpenAI",           // Provider: "OpenAI" or "LocalAI"
    "gpt-realtime", 
    "You are a helpful assistant", 
    {"text", "audio"}
);

// To send text to AI (trigger this from user input, button press, etc.)
AIClient->SendTextToAI("Hello, how are you?");
// This automatically triggers an AI response - no additional call needed

Configure your AI provider under Edit → Project Settings → Plugins → LLMAI. See Configuration for OpenAI, Grok, and LocalAI setup.

Step 5: Enable Voice (Optional)​

Start Voice Mode​

// In Blueprint: use "Start Voice Mode" node with voice name "alloy"  
// In C++:
AIClient->StartVoiceMode("alloy");
// Audio capture and streaming are handled automatically!

Note :

  • StartVoiceMode() automatically handles audio setup - no manual setup required
  • Audio streaming starts automatically when voice mode activates

Step 6: Add AI Functions (Optional)​

Let the AI call functions in your game!

Create Function Definition​

  1. Right-click in Content Browser
  2. Miscellaneous > LLMAI Function Definition
  3. Name : "spawn_enemy"
  4. Description : "Spawn an enemy in the game"
  5. Add Parameters as needed

Register and Handle​

// Register the function
AIClient->RegisterAIFunction(MyFunctionDefinition);

// Handle AI function calls
AIClient->OnAIFunctionCallRequested.AddDynamic(this, &AMyActor::HandleFunctionCall);

// In your handler
void AMyActor::HandleFunctionCall(const FLLMFunctionCall& FunctionCall)
{
    if (FunctionCall.Name == "spawn_enemy")
    {
        // Your game logic here
        SpawnEnemy();
        
        // Tell AI it worked
        AIClient->SendAIFunctionCallResult(FunctionCall.CallId, "Enemy spawned successfully!");
    }
}

What to Try Next​

Basic Conversations​

  • "Hello, how are you?" - Test basic chat
  • "What can you help me with?" - Explore AI capabilities
  • "Tell me about this game" - Context-aware responses

Voice Features​

  • Enable voice mode and have natural conversations
  • Try different voices : Known available voices are listed in DefaultGame.ini
  • Test interruption : Talk while AI is responding

Function Calling​

  • Ask AI to control your game : "Spawn 3 enemies"
  • Complex requests : "Make the game harder by adding obstacles"
  • Context-aware functions : "If I'm doing well, increase difficulty"

Troubleshooting​

"Plugin not found"​

  • Check LLMAI folder is in Plugins/LLMAI/
  • Verify LLMAI.uplugin file exists
  • Restart Unreal Engine editor

"API key invalid" or connection issues​

See Configuration — Troubleshooting and the FAQ.

"Connection failed"​

  • Check internet connection
  • Verify firewall isn't blocking WebSocket connections
  • Try different network if on corporate/school WiFi
  • Debug command : llmai.debug.WebSocketLevel 2 (in console)

"No voice input"​

  • Check microphone permissions in Windows
  • Test microphone in other applications
  • Try plugging in microphone if using external device
  • Debug command : llmai.debug.AudioLevel 2 (in console)

"Audio not playing"​

  • Check audio output device
  • Adjust system volume
  • Test with different audio device
  • Debug command : llmai.debug.SaveAudioFiles 1 (to analyze audio data)

Advanced Troubleshooting​

For complex issues, LLMAI includes a comprehensive debug system:

# Open console (` key) and use these commands:
llmai.debug.EnableAll           # Turn on all debugging
llmai.debug.OpenAILevel 2       # See API communication details
llmai.debug.FunctionLevel 2     # Debug function calling

Full Debug Guide : Debug & Logging Documentation

Next Steps​

Once you've got the basics working:

  1. Read the Plugin Installation Guide for advanced setup options
  2. Explore the Rocketeer Game Demo Documentation for detailed examples of function calling in a game
  3. Explore the Terminal Demo Documentation for detailed examples of text and voice communication
  4. Study the API Reference for complete functionality
  5. Review the Plugin Documentation for technical details

Congratulations!​

You now have realtime AI available in Unreal Engine!

The AI can:

  • Chat naturally about any topic
  • Understand context about your game
  • Control game systems through function calls
  • Communicate by voice for hands-free interaction