Skip to main content

FAQ

General Questions​

What is LLMAI?

LLMAI is a comprehensive AI integration plugin for Unreal Engine 5 that enables real-time text and voice conversations with AI assistants, plus the ability for AI to call functions and control game systems.

What versions of Unreal Engine are supported?
  • Minimum : Unreal Engine 5.6
  • Recommended : Unreal Engine 5.6+
  • Tested : Up to Unreal Engine 5.7
What platforms are supported?

Why Windows only? The plugin uses platform-specific audio libraries for real-time voice processing:

  • Windows : winmm.lib and dsound.lib for low-latency audio capture
  • Mac : Framework support for AudioUnit , AudioToolbox , and CoreAudio is prepared but not tested
  • Linux : asound (ALSA) support is prepared but not tested

Technical Details : See Plugin Dependencies for complete dependency information.

Do I need an OpenAI account?

You have three options:

  • OpenAI (Cloud): Requires an OpenAI account with API access. Uses OpenAI's Realtime API for conversation and function calling features.
  • Grok (xAI Cloud): Requires a Grok API key from console.x.ai . Includes web search capability.
  • LocalAI (Offline): Use the LLMAI LocalAI distribution (available separately) for local AI processing without any cloud account.
What's the difference between OpenAI, LocalAI, and Grok providers?

All three providers support the Realtime API protocol with minor differences:

  • OpenAI - Cloud-based, most features, can change voice mid-session, Semantic VAD support
  • LocalAI - Self-hosted, offline capable, full customization, no usage limits
  • Grok (xAI) - Cloud-based, web search capability, voice locked at connection, Server VAD only

See Provider Comparison Table for detailed feature comparison.

Why can't I change the voice on Grok after connecting?

Grok has a technical limitation where the voice is locked after the first session configuration. You must specify the Voice parameter when calling ConnectToAI or ConnectToGrok , or set it in Project Settings > Plugins > LLMAI > Grok Provider .

To change voices, you need to disconnect and reconnect with a different voice.

This limitation does not apply to OpenAI or LocalAI - those providers allow changing voices mid-session via StartVoiceMode() .

What does "Rate limit exceeded" mean?

This means you've made too many API requests in a short time. Each provider has different limits:

Wait a few minutes before retrying. The error message will include a link to check your API usage for cloud providers.

Setup and Installation​

How do I get an OpenAI API key?

Note: An OpenAI API key is only required if using OpenAI's cloud service. For local/offline operation, use LocalAI instead.

  1. Go to platform.openai.com
  2. Sign up or log in
  3. Navigate to API keys section
  4. Create a new API key
  5. Copy the key to LLMAI plugin settings
Where do I put my API key?

See Configuration — Settings priority and Provider profiles for API key order, Project Settings field names, and LocalAI endpoint setup.

For LocalAI: No API key is required. See LocalAI Realtime Endpoint for server download and setup.

Can I use other AI services besides OpenAI?

Yes! LLMAI currently supports three Realtime provider profiles:

  • OpenAI — cloud Realtime API (requires API key and internet)
  • Grok (xAI) — cloud Realtime API with optional web search (requires API key)
  • LocalAI — local/offline Realtime endpoint (no API key; separate distribution)

Compare capabilities on the Providers page. Quick setup: Configuration quick-start. Full reference: Configuration.

Technical Details: Provider-agnostic functions (ConnectToAI, SendTextToAI) work across profiles. See API Documentation.

The plugin doesn't appear in my plugins list
  • Ensure the LLMAI folder is in your project's Plugins/ directory
  • Verify the LLMAI.uplugin file exists in the plugin folder
  • Restart Unreal Engine editor completely
  • Check that you're using UE 5.6 or later

Usage and Features​

What AI models can I use?

The plugin uses the OpenAI Realtime API (GA as of LLMAI v2.2). For the most current model list, see KnownModels in Plugins/LLMAI/Config/DefaultGame.ini.

As of v2.2, OpenAI defaults include gpt-realtime-2 (recommended), plus gpt-realtime, gpt-realtime-mini, and gpt-realtime-1.5. Preview-era model names from older plugin versions are no longer supported by OpenAI.

When upgrading from an older plugin version, reset [/Script/LLMAI.LLMAISettings] in your project's Config/DefaultGame.ini so model defaults refresh—see Changelog — Upgrading from a previous version.

What voices are available?

For the most current list of available voices , see Plugins/LLMAI/Config/DefaultGame.ini under the OpenAI provider block (KnownVoices).

Current voices include:

Note : OpenAI periodically updates available voices. The configuration file always contains the most current list.

Can the AI control my game?

Yes! The plugin provides function calling capabilities that let AI interact with your game systems.

See the Rocketeer Game Demo (LLMAI_Arcade) for a complete example where AI can:

  • Launch and close the game
  • Set stage and difficulty
  • Trigger game events
  • Add or remove lives
  • Change rocket firing mode
  • Add and modify flight paths
  • Spawn enemy series
  • and lots more

Technical Details : See API Documentation for function definition syntax and implementation patterns.

What is the LLMAI_LiveLink demo?

The LLMAI_LiveLink demo showcases AI-driven character animation with a MetaHuman (Cooper). It demonstrates:

  • Lip-Sync : AI voice output automatically drives MetaHuman lip animations via LiveLink
  • 16 Gestures : Wave, thumbs up/down, nod yes/no, dance, and more
  • Character Movement : AI can move the character between waypoints (left, middle, right)
  • Jump Action : Full character jump with animation state machine

Learn More : See Live Link demo .

How does lip-sync work in the LiveLink demo?

The lip-sync feature works automatically when voice mode is enabled:

  1. AI generates voice response via TTS (Text-to-Speech)
  2. Audio is streamed through LiveLink to the MetaHuman
  3. MetaHuman's facial animation system drives lip movements in real-time

No additional configuration is needed - simply enable voice mode and the lip-sync activates automatically.

What gestures are available in the LiveLink demo?

The AI can trigger 16 different character gestures through natural language:

Try natural commands like "Wave hello", "Give me a thumbs up", or "Show me you're not sure".

Is voice communication real-time?

Yes, the plugin uses OpenAI's Realtime API for low-latency voice conversations. You can speak naturally and the AI responds with both voice and text.

Demo : Try the LLMAITerminal Demo to experience real-time voice conversations in action.

Technical Questions​

Can I use this in C++ projects?

Yes, LLMAI works in both Blueprint and C++ projects. All functionality is available through both interfaces.

How do I handle API costs?
  • Monitor your OpenAI API usage in their dashboard
  • Implement rate limiting in your application
  • Consider caching responses for common queries
  • Use text mode when voice isn't necessary
Does this work offline?

OpenAI: Requires an internet connection to communicate with OpenAI's servers.

LocalAI: Yes! The LLMAI LocalAI distribution (available separately) supports offline operation with local AI processing and TTS voice synthesis.

Performance and Optimization​

How much bandwidth does voice mode use?

Voice mode uses approximately:

  • Upload : 24kbps for voice input
  • Download : Variable based on AI response length
  • Text mode : Much lower bandwidth usage

Troubleshooting​

Voice input isn't working​

  1. Check microphone permissions in Windows
  2. Test microphone in other applications
  3. Verify audio drivers are up to date
  4. Try different microphone if using external device
  5. Check Windows privacy settings for microphone access

Connection keeps dropping​

  1. Check firewall settings - ensure WebSocket connections allowed
  2. Try different network if on corporate/restricted WiFi
  3. Check proxy settings if behind corporate proxy
  4. Verify stable internet connection
  5. Enable auto-reconnect in plugin settings

Function calls aren't working​

  1. Check function definitions for correct syntax
  2. Verify function registration in logs
  3. Test with simple functions first
  4. Check AI instructions mention function availability
  5. Ensure proper parameter types are defined

Development Questions​

Can I modify the plugin source code?

Yes, if you need to modify the plugin:

  1. Ensure your project is a C++ project
  2. Place the plugin in your project's Plugins folder
  3. Regenerate project files
  4. Build the project in Visual Studio

Technical Details : See Plugin Documentation for development setup and architecture overview.

How do I add custom audio processing?

Use the ULLMAIAudioStreamComponent for custom audio capture and processing.

Implementation Examples : See API Documentation for audio processing patterns and component usage details.

Can I create custom AI personalities?

Yes, by modifying the instructions parameter when connecting to AI. You can create different personalities for different game characters or contexts.

Examples and Syntax : See API Documentation for instruction parameter details and personality examples.

How do I debug and troubleshoot issues?

LLMAI includes a comprehensive debug system:

  1. Enable debug logging in the console: llmai.debug.EnableAll # Turn on all debugging llmai.debug.OpenAILevel 2 # Debug API issues llmai.debug.AudioLevel 2 # Debug voice problems
  2. Check specific issues : Connection problems: llmai.debug.WebSocketLevel 2 Voice not working: llmai.debug.SaveAudioFiles 1 Function calls failing: llmai.debug.FunctionLevel 2
  3. Find debug output : Output Log: Window > Developer Tools > Output Log Log Files: YourProject/Saved/Logs/ Debug Files: YourProject/Saved/Logs/LLMAI/

Complete Guide : See Debug & Logging Documentation for detailed troubleshooting.

How do I handle errors gracefully?

Bind to error events:

  • OnError for connection errors
  • OnAIServerError for API errors
  • OnAIRateLimitExceeded for rate limiting
  • Use debug logging to understand error causes
  • Implement fallback behaviors for each error type

Event Details : See API Documentation for complete event signatures and error handling patterns.

License and Distribution​

What license is LLMAI under?

LLMAI is distributed under the Fab Standard License when available on fab.com. This license allows for commercial and non-commercial use in your projects. See the Fab EULA for complete terms.

Can I distribute games with LLMAI?

Yes, you can distribute games that include the LLMAI plugin. Ensure you handle API key security appropriately.

Do I need to credit LLMAI?

While not required by the Fab Standard License, attribution is appreciated. Consider mentioning LLMAI plugin in your game's credits or documentation.

Future Development​

What features are coming next?

Planned features include:

  • Support for additional AI providers (Claude, Gemini, Grok, llama)
  • Mac and Linux platform support
  • Mobile platform support
  • Enhanced LocalAI capabilities