LogoLogo
WebsiteBlogForumsSign up
  • Welcome
  • How Ready Player Me works
  • FAQ
  • 👩‍💻Integration Guides
    • Overview
    • Unity
      • Quickstart
      • Avatar Creator Integration
        • WebView Avatar Creator
          • Avatar Creator for Android & iOS
          • Avatar Creator for WebGL
        • Build your own Avatar Creator
          • Start with the sample
          • Elements
            • Asset Selection Element
            • Color Selection Element
            • Template Selection Element
            • Body Shapes Selection Element
            • Gender Selection Element
            • Avatar List Element
            • Photo Capture Element
            • Signup Element
            • Login Element
            • Logout Element
          • User Management
      • Load 2D Renders
      • Avatar Loader Window
      • Setup Multiplayer
      • Setup for XR (Beta)
        • Setup the Player Avatar
        • Setup Meta Movement SDK
        • Setup Final IK
        • Setup XR Hands
          • RpmHandDriver script
        • Facial Animations
        • VR Avatar Creator
      • Animations
        • Ready Player Me Animation Library
        • Mixamo Animations
      • Optimize
        • Avatar Configuration
        • Avatar Caching
        • Defer Agents
      • Code Samples
        • Loading 3D Avatars
        • Loading 2D Avatars
        • Avatar Creator (WebGL)
        • Avatar Creator (Desktop)
        • Avatar Creator (Mobile)
        • Distance-based LODs
      • Troubleshooting
        • Editor UI Window Issues
        • Avatar loading issues
        • iOS issues
        • WebGL issues
        • Firewall issues
        • Updating the SDK
          • Unity SDK 3.0.0 Migration guide
          • Unity SDK 4.0.0 Migration guide
          • Unity SDK 5.0.0
          • Unity SDK 7.0.0
          • WebView 2.0 Migration
        • Package installation issues
      • Help us improve the Unity SDK
      • FAQ for Unity
      • Early Access Features
    • Unreal Engine
      • Quickstart
        • Import SDK Manually
      • Avatar Creator Integration
        • Build your own Avatar Creator
          • Sample Structure
          • Customization Guide
        • WebView Avatar Creator
      • Load Avatars
      • Animations
        • Ready Player Me Animation Library
        • Loading Mixamo animations
        • Mannequin animation retargeting
        • Blender FBX export to UE
        • Oculus Lipsync integration
      • Optimize
        • Avatar configuration
        • Avatar caching
        • Avatar preloading
      • Code Samples
        • Unreal Engine 5 samples
        • Lyra project integration example
        • VR Avatars
        • Unreal Engine 4
      • Troubleshooting
        • Project compilation issues
        • Missing materials on built applications
        • Avatar loading issues
        • Animation issues
        • Unreal SDK breaking changes
        • Avatar lighting issues
        • Updating from earlier versions
          • Unreal SDK 3.0.0 Migration guide
        • Auto LOD issues
      • Help us improve the Unreal Engine SDK
      • FAQ for Unreal Engine
      • Early Access Features
    • React
      • Quickstart
    • React Native
    • iOS Native
    • Android Native
    • Web Integration
      • Quickstart
      • User Management
        • Guest Accounts
        • Account Linking
      • Avatar Creator integration
      • Optimize
    • API Integration
      • Quickstart
      • Custom Avatar Creator
      • User management
        • Ready Player Me Account
        • Anonymous Accounts
    • UX/UI Guidelines
      • Summary & quick tips
      • UX and UI guidelines and essentials
        • Foundation for good user experience
          • Layout
          • Colors
          • Typography
          • Icons
          • Dark mode and light mode
        • Mobile best practices
          • Layout
        • Making Ready Player Me feel native
          • Seamless Avatar Integration
          • Consistent Branding
          • Responsive Layout
          • Performance Optimization
        • Discoverability of the avatar editor
          • Onboarding tutorial
          • Clear navigation
          • Contextual Placement
  • 🖌️Customizing Guides
    • Studio (Developer Dashboard)
    • Avatar Creator Appearance
      • Avatar URLs
      • Avatar Shortcodes
    • Upload and Manage Custom Assets
      • Asset Manager in Studio
      • Configure in Studio
      • Manage custom assets using the API
      • Unlocking assets through API (Beta)
    • Create Custom Assets
      • Fullbody Outfits
        • Checking Skin Weights
      • Tops, Bottoms, Shoes
        • Editing templates
      • Hairstyle
      • Headwear
      • Facewear
      • Glasses
      • Hero Characters
      • Modeling Guidelines
  • 🔃API Reference
    • REST API
      • Authentication
      • Avatars
        • GET - 3D avatar
        • GET - 2D Render of an Avatar
        • GET - Metadata
        • PUT - Equip an asset
        • PUT - Unequip an asset
      • Assets
        • Asset Entity Properties
        • POST - Create Asset
        • GET - List Assets
        • PATCH - Update Asset
        • POST - Upload Asset Files
        • POST - Add Asset to Application
        • DELETE - Remove Asset from Application
        • PUT - Unlock asset for a user
        • PUT - Lock asset for a user
      • Users
        • POST - Create User
      • Auth
        • GET - Token
      • Changelog
    • Avatars
      • Full-body avatars
      • Full-body XR avatars
      • Half-body avatars
      • Morph targets
        • Apple ARKit
        • Oculus OVR LipSync
    • Avatar Creator
  • 🔉Support
    • Forums
    • Licensing & Privacy
Powered by GitBook
On this page
  • Plugin setup
  • Frame sequence for the sound
  • Avatar config
  • Component setup
  • Blueprint logic

Was this helpful?

  1. Integration Guides
  2. Unreal Engine
  3. Animations

Oculus Lipsync integration

Apply Oculus Lip Sync facial animations to the Ready Player Me avatars

PreviousBlender FBX export to UENextOptimize

Last updated 2 years ago

Was this helpful?

Oculus Lipsync is a plugin provided by Meta that can be used to sync avatar lip movements to speech sounds and laughter. Meta provides public and an example project for the OVRLipSync integration.

Important: To integrate OVRLipSync you first need to read and agree to Meta's terms and conditions of using the plugin.

Ready Player Me avatars come with Oculus Viseme morph targets needed for the OVRLipSync plugin.

Plugin setup

You need to follow the steps described in the . When you finish with the described steps, you would have the OVRLipSync plugin successfully added to your project. You can also check the public example provided by Meta to better understand how it works.

Note: While in UE4 everything will work out of the box, for UE5 you would need to do manual fixes inside of the ovrLipSync plugin, the project wasn't updated for a while. Open the OVRLipSyncEditorModule.cpp file and add the following line in line 90. SoundWave->LoadingBehavior = ESoundWaveLoadingBehavior::ForceInline; . Open the OVRLipSync.Build.cs file and add AndroidPermission dependency to the PublicDependencyModuleNames dependencies.

Follow the steps for setting Up Ready Player Me Unreal SDK for the created project.

Now that all of the plugins are added to the project and the project is compilable we can start with adding Blueprint files.

There are two ways you can integrate the OVRLipSync:

  1. By live capturing the sound from an external audio source and applying it to OVRLipSync.

  2. By playing an already existing soundtrack track and applying it to OVRLipSync.

The plugin provides two actor components OVRLipSync used for live capturing and OVRLipSyncPayback used for playing the existing sound. In this example, for simplicity, we will use OVRLipSyncPayback with an already-existing soundtrack.

Frame sequence for the sound

In the first step, we import the speech audio file to the project. Right-click on the imported audio file and from the dropdown select Generate LipSyncSequence. A new FrameSequence blueprint file will be created next to the audio file. This file will be used by the OVRLipSyncPayback component.

Avatar config

To load avatars that contain Oculus Viseme morph targets, we need to create An avatar config and a standard morph target group. We simply duplicate DA_RPM_AvatarConfig and DA_RPM_StandardMorphTargetGroup from the ReadyPlayerMe plugin into our project and change them. We add the Oculus Viseme morph target group to the duplicated data asset.

The next step is to create a BP_RPM_Actor actor blueprint that will represent the talking model. We will add 4 components to it. SkeletalMesh, Audio, ReadyPlayerMe, and OVRLipSyncPlayback.

Component setup

In the SkeletalMesh component, we set the skeletal mesh to RPM_Mixamo_SkeletalMesh.

In the Audio component, we set the sound to the speech sound we want to play and disable the Auto Activate flag to play the sound after the avatar is loaded. We can also position it close to the mouth of the avatar in the viewport and make it a child of the SkeletalMesh component so that the sound will be attached to the mesh.

In the ReadyPlayerMe component, we set the avatar URL to the avatar that we want to load. Target Skeleton to RPM_Mixamo_Skeleton. Avatar config to the newly created DA_VisemeAvatarConfig.

In the OVRLipSyncPlayback component, we set the sequence property to the generated FrameSequence blueprint file.

Blueprint logic

In the Begin Play event, we hide the skeletal mesh and load the avatar. When the new avatar is loaded we show the skeletal mesh and call the Start function of the OVRLipSyncPayback component, to start paying the animation.

The morph target names of the Ready Player Me avatars are slightly different from the hardcoded names that are added in the OVRLipSync plugin, they have a `viseme_` prefix. For applying the correct names for the morph targets, select the OVRLipSyncPayback component, open the properties tab, and click on the + button next to the On Visemes Ready. This will create an event in the event graph. In this event, we call the Assign Visemes to Morph Targets function of the OVRLipSyncPayback component and set the list of morph target names. Below is the list of morph target names in the order that is required.

viseme_sil, viseme_PP, viseme_FF, viseme_TH, viseme_DD, viseme_kk, viseme_CH, viseme_SS, viseme_nn, viseme_RR, viseme_aa, viseme_E, viseme_I, viseme_O, viseme_U

We are done with the integration, we place the actor on the map and press the play button, and we will get the following result.

👩‍💻
documentation
OVRLipSync setup documentation
7MB
OculusLipSyncDemo.mp4