❓ Frequently Asked Questions

LLMAI is a comprehensive AI integration plugin for Unreal Engine 5 that enables real-time text and voice conversations with AI assistants, plus the ability for AI to call functions and control game systems.

• Minimum: Unreal Engine 5.6
• Recommended: Unreal Engine 5.6+
• Tested: Up to Unreal Engine 5.6

Why Windows only? The plugin uses platform-specific audio libraries for real-time voice processing:

• Windows: winmm.lib and dsound.lib for low-latency audio capture
• Mac: Framework support for AudioUnit, AudioToolbox, and CoreAudio is prepared but not tested
• Linux: asound (ALSA) support is prepared but not tested

Technical Details: See Plugin Dependencies for complete dependency information.

You have two options:

• OpenAI (Cloud): Requires an OpenAI account with API access. Uses OpenAI's Realtime API for conversation and function calling features.
• LocalAI (Offline): Use the LLMAI LocalAI distribution (available separately) for local AI processing without an OpenAI account.

Note: An OpenAI API key is only required if using OpenAI's cloud service. For local/offline operation, use LocalAI instead.

1. Go to platform.openai.com
2. Sign up or log in
3. Navigate to API keys section
4. Create a new API key
5. Copy the key to LLMAI plugin settings

For OpenAI: The plugin checks for your OpenAI API key in multiple locations (in this order):

1. Project Settings (Recommended):
   • Go to Edit > Project Settings > Plugins > LLMAI
   • Enter your key in "Default OpenAI API Key" field
2. Environment Variables:
   • Set LLMAI_OPENAI_API_KEY=your_key_here in your system environment
   • Or use the standard OPENAI_API_KEY=your_key_here
3. Command Line Parameter:
   • Launch with -OpenAIKey=your_key_here parameter
   • Useful for automated builds or CI/CD pipelines

Note: The plugin tries these methods in order and uses the first valid key it finds. Project Settings is recommended for most users.

For LocalAI: No API key is required. Configure the LocalAI endpoint URL in Project Settings instead. See LocalAI Quick Start Guide for download and setup.

Yes! LLMAI currently supports two providers:

• OpenAI — Cloud-based AI using OpenAI's Realtime API (requires API key and internet)
• LocalAI — Local/offline AI processing with local STT and TTS (available separately, no API key required)

Support for additional providers (Anthropic, Gemini, Grok) is planned for future releases.

Technical Details: The plugin includes provider-agnostic functions (ConnectToAI, SendTextToAI) designed for multi-provider support. See API Documentation for the provider architecture.

• Ensure the LLMAI folder is in your project's Plugins/ directory
• Verify the LLMAI.uplugin file exists in the plugin folder
• Restart Unreal Engine editor completely
• Check that you're using UE 5.6 or later

The plugin is optimized for OpenAI's Realtime API. For the most current list of supported models, see Plugins/LLMAI/Config/DefaultGame.ini which contains the latest KnownRealtimeModels list.

As of this version, supported models include:

• gpt-realtime (recommended default)
• gpt-4o-realtime-preview with various date variants
• gpt-4o-mini-realtime-preview with date variants

For the most current list of available voices, see Plugins/LLMAI/Config/DefaultGame.ini which contains the latest KnownOpenAIRealtimeVoices list.

Current voices include:

Note: OpenAI periodically updates available voices. The configuration file always contains the most current list.

Yes! The plugin provides function calling capabilities that let AI interact with your game systems.

See the Rocketeer Game Demo (LLMAI_Arcade) for a complete example where AI can:

• Launch and close the game
• Set stage and difficulty
• Trigger game events
• Add or remove lives
• Change rocket firing mode
• Add and modify flight paths
• Spawn enemy series
• and lots more

Technical Details: See API Documentation for function definition syntax and implementation patterns.

The LLMAI_LiveLink demo showcases AI-driven character animation with a MetaHuman (Cooper). It demonstrates:

• Lip-Sync: AI voice output automatically drives MetaHuman lip animations via LiveLink
• 16 Gestures: Wave, thumbs up/down, nod yes/no, dance, and more
• Character Movement: AI can move the character between waypoints (left, middle, right)
• Jump Action: Full character jump with animation state machine

Learn More: See LiveLink Demo Details.

The lip-sync feature works automatically when voice mode is enabled:

1. AI generates voice response via TTS (Text-to-Speech)
2. Audio is streamed through LiveLink to the MetaHuman
3. MetaHuman's facial animation system drives lip movements in real-time

No additional configuration is needed - simply enable voice mode and the lip-sync activates automatically.

The AI can trigger 16 different character gestures through natural language:

Try natural commands like "Wave hello", "Give me a thumbs up", or "Show me you're not sure".

Yes, the plugin uses OpenAI's Realtime API for low-latency voice conversations. You can speak naturally and the AI responds with both voice and text.

Demo: Try the LLMAITerminal Demo to experience real-time voice conversations in action.

Yes, LLMAI works in both Blueprint and C++ projects. All functionality is available through both interfaces.

• Monitor your OpenAI API usage in their dashboard
• Implement rate limiting in your application
• Consider caching responses for common queries
• Use text mode when voice isn't necessary

OpenAI: Requires an internet connection to communicate with OpenAI's servers.

LocalAI: Yes! The LLMAI LocalAI distribution (available separately) supports offline operation with local AI processing and TTS voice synthesis.

Voice mode uses approximately:

• Upload: 24kbps for voice input
• Download: Variable based on AI response length
• Text mode: Much lower bandwidth usage

Yes, if you need to modify the plugin:

1. Ensure your project is a C++ project
2. Place the plugin in your project's Plugins folder
3. Regenerate project files
4. Build the project in Visual Studio

Technical Details: See Plugin Documentation for development setup and architecture overview.

Use the ULLMAIAudioStreamComponent for custom audio capture and processing.

Implementation Examples: See API Documentation for audio processing patterns and component usage details.

Yes, by modifying the instructions parameter when connecting to AI. You can create different personalities for different game characters or contexts.

Examples and Syntax: See API Documentation for instruction parameter details and personality examples.

LLMAI includes a comprehensive debug system:

1. Enable debug logging in the console:

   llmai.debug.EnableAll              # Turn on all debugging
   llmai.debug.OpenAILevel 2          # Debug API issues  
   llmai.debug.AudioLevel 2           # Debug voice problems

2. Check specific issues:
   • Connection problems: llmai.debug.WebSocketLevel 2
   • Voice not working: llmai.debug.SaveAudioFiles 1
   • Function calls failing: llmai.debug.FunctionLevel 2
3. Find debug output:
   • Output Log: Window > Developer Tools > Output Log
   • Log Files: YourProject/Saved/Logs/
   • Debug Files: YourProject/Saved/Logs/LLMAI/

Complete Guide: See Debug & Logging Documentation for detailed troubleshooting.

Bind to error events:

• OnError for connection errors
• OnAIServerError for API errors
• OnAIRateLimitExceeded for rate limiting
• Use debug logging to understand error causes
• Implement fallback behaviors for each error type

Event Details: See API Documentation for complete event signatures and error handling patterns.

LLMAI is distributed under the Fab Standard License when available on fab.com. This license allows for commercial and non-commercial use in your projects. See the Fab EULA for complete terms.

Yes, you can distribute games that include the LLMAI plugin. Ensure you handle API key security appropriately.

While not required by the Fab Standard License, attribution is appreciated. Consider mentioning LLMAI plugin in your game's credits or documentation.

Planned features include:

• Support for additional AI providers (Claude, Gemini, Grok, llama)
• Mac and Linux platform support
• Mobile platform support
• Enhanced LocalAI capabilities