FaceUnity AR Filter

The FaceUnity AR Filter extension uses 3D vision, 3D graphics, and deep learning technologies to provide industry’s leading AR portrait video effects for Video SDK. It enables you to add diverse special effects that provide full coverage of the human face, including:

• Basic beauty: Skin beautification, whitening, rosy.
• Advanced beauty: Video beauty effects for shape, face and skin.
• Number detection: Detect the number of faces, humans, or gestures.

This page shows you how to integrate and use the FaceUnity AR Filter extension in your app.

To quickly integrate FaceUnity's AR filter capabilities in your app, you set extension properties using the key and value parameters in Video SDK. When you call setExtensionProperty or setExtensionPropertyWithVendor method of Video SDK and pass in a pair of key and value parameters, it is equivalent to calling the corresponding FaceUnity API. The key is named after the FaceUnity API method, and value wraps the parameters required for that method in JSON. Currently, the extension encapsulates part of the APIs of the FaceUnity Nama SDK. For details, see the FaceUnity key-value overview.

The development environment requirements are as follows:

• Android Studio 4.1 or higher.
• Android SDK API Level 24 or higher.
• A mobile device that runs Android 4.1 or higher.

• Implemented SDK quickstart for Video Calling.

To implement FaceUnity features in your app, open the SDK quickstart for Video Calling project you created previously.

1. Setup the sample project

   Agora provides the FaceUnity sample project on Github to help you get started with FaceUnity. To set up and run the sample project, do the following:

   1. Clone the Github repository. In Terminal, execute the following command:

      
      

      _1

      git clone https://github.com/AgoraIO-Community/AgoraMarketPlace.git

      
      

   2. Complete the setup. Follow the steps in the README.

2. Get the FaceUnity extension

   To get the FaceUnity extension license, filters, and resource files, take the following steps:

   1. Contact Agora for activation and provide your package name. For example, yourCompany.yourProject.faceunity.

      You receive the authpack certificate file from Agora that corresponds to your package name. You use this file to integrate FaceUnity into your app.

   2. Download the FaceUnity AR Filter package for your platform from the Extensions Marketplace.

   3. Download the FaceUnity resource package.

This section describes the call sequence you implement to use FaceUnity features in your app.

1. Add the extension to your project

   To add the FaceUnity extension to to your project, take the following steps:

   1. Unzip the FaceUnity AR Filter package, then save all .aar files to the /app/libs folder in your Android project.

   2. Save the certificate file authpack.java to the path corresponding to the package name you provided when you obtained the certificate.

      For example, if the package name is io.agora.rte.extension.faceunity.example, the certificate file should be saved to /app/src/main/java/io/agora/rte/extension/faceunity/example).

   3. Save the model and prop files you need from the resource package to the /app/src/main/assets folder in your project.

      For details of the files provided in the resource pack, see Resource package structure.

   4. In app/build.gradle, add the following line under dependencies:

      
      

      _1

      implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])

      
      

   5. Add the following libraries in the list of import statements in your Android activity:

      
      

      _8

      import io.agora.rtc2.Constants;

      _8

      import io.agora.rtc2.IMediaExtensionObserver;

      _8

      import io.agora.rtc2.IRtcEngineEventHandler;

      _8

      import io.agora.rtc2.RtcEngine;

      _8

      import io.agora.rtc2.RtcEngineConfig;

      _8

      import io.agora.rtc2.video.VideoCanvas;

      _8

      _8

      import io.agora.rte.extension.faceunity.ExtensionManager;

      
      

2. Enable the extension

   When you initialize RtcEngine, call enableExtension before other APIs, including enableVideo and joinChannel.

   
   

   _6

   RtcEngineConfig config = new RtcEngineConfig();

   _6

   // load the plugin

   _6

   config.addExtension("AgoraFaceUnityExtension");

   _6

   mRtcEngine = RtcEngine.create(config);

   _6

   // enable plugin

   _6

   mRtcEngine.enableExtension("FaceUnity", "Effect", enabled);

   
   

3. Initialize the extension

   After receiving the onExtensionStarted callback, call setExtensionProperty, and pass the corresponding key and value pair:

   1. Load the extension.

      Set the key as fuSetup, and the value as the pointer to the certificate file.

   2. Load the AI model.

      Set the key as fuLoadAIModelFromPackage, and the value to the path of the AI capability model file ai_xxx.bundle and the AI capability type.

      
      

      _33

      private void initExtension() {

      _33

      _33

      // initialization

      _33

      try {

      _33

      JSONObject jsonObject = new JSONObject();

      _33

      JSONArray jsonArray = new JSONArray();

      _33

      for (byte it : authpack.A()) {

      _33

      jsonArray.put(it);

      _33

      }

      _33

      jsonObject.put("authdata", jsonArray);

      _33

      setExtensionProperty("fuSetup", jsonObject.toString());

      _33

      } catch (JSONException e) {

      _33

      Log.e(TAG, e.toString());

      _33

      }

      _33

      _33

      _33

      // load the AI model

      _33

      File modelDir = new File(getExternalFilesDir("assets"),

      _33

      "face_unity/model/ai_face_processor.bundle");

      _33

      try {

      _33

      JSONObject jsonObject = new JSONObject();

      _33

      jsonObject.put("data", modelDir.getAbsolutePath());

      _33

      jsonObject.put("type", 1 << 8);

      _33

      setExtensionProperty("fuLoadAIModelFromPackage", jsonObject.toString());

      _33

      } catch (JSONException e) {

      _33

      Log.e(TAG, e.toString());

      _33

      }

      _33

      }

      _33

      _33

      // you may call setExtensionProperty multiple times

      _33

      private void setExtensionProperty(String key, String property) {

      _33

      mRtcEngine.setExtensionProperty("FaceUnity", "Effect", key, property);

      _33

      }

      
      

4. Configure beauty effects and body recognition

   You can implement the following functions:

   • Load props and adjust beautification intensity
   • Recognize and track human faces, gestures, and bodies

   To set the effects, call setExtensionProperty and pass the corresponding keys and values as needed. For a full list of keys and values, see the FaceUnity key-values.

5. Release the resources

   When you stop using FaceUnity do the following:

   1. Call setExtensionProperty and pass in the fuDestroyLibData key.
   2. After receiving the fuDestroyLibData callback, call destroy to destroy the Agora Engine.

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.

This section lists the FaceUnity APIs you can use with Agora SDK.

The key corresponds to the name of the FaceUnity API, and the value corresponds to the parameters of the API. In this section, if the value is the same as the parameters of the FaceUnity API, the link leads to the FaceUnity documentation. If the value is different from the parameters of the FaceUnity API, the link leads to a section on this page.

Method keys refer to the keys you pass in when calling the setExtensionProperty or setExtensionPropertyWithVendor method of the Video SDK.

Method keys	Description
fuSetup	Initialize the extension and authenticate the user. Must be executed before other keys.
fuLoadAIModelFromPackage	Preload AI capabilities.
fuReleaseAIModel	Free up resources occupied by AI capabilities.

Method keys	Description
fuCreateItemFromPackage	Loads the prop package.
fuLoadTongueModel	Loads tongue detection data.
fuItemSetParam	Modifies or sets the value of a variable in the prop package.

Method keys	Description
fuDestroyItem	Destroys a specified item.
fuDestroyAllItems	Destroys all loaded items and releases all occupied resources.
fuOnDeviceLost	Resets the system's GL state. Use this key when the OpenGL context is released/destroyed by external resources.
fuDestroyLibData	Frees up the memory allocated to the face tracking module after calling fuSetup.

Method keys	Description
fuBindItems	Binds resource items to a target item.
fuUnbindItems	Unbinds the resource items from a target item.
fuIsTracking	Sets whether to get the number of faces being tracked.
fuSetMaxFaces	Sets the maximum number of tarcked faces.
fuSetDefaultRotationMode	Sets the default human face orientation.

Method keys	Description
fuFaceProcessorSetMinFaceRatio	Sets the distance of face detection.
fuSetTrackFaceAIType	Sets the fuTrackFace algorithm type.
fuSetFaceProcessorFov	Sets the fov (equivalent to focal length) of the FaceProcessor algorithm module.
fuHumanProcessorReset	Resets the state of the HumanProcessor algorithm module.
fuHumanProcessorSetMaxHumans	Sets the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorGetNumResults	Sets whether to get the number of bodies tracked by the HumanProcessor algorithm module.
fuHumanProcessorSetFov	Sets the fov (equivalent to focal length) used by the HumanProcessor algorithm module to track 3D key points on human bodies.
fuHandDetectorGetResultNumHands	Sets whether to get the number of gestures tracked by the HandGesture algorithm module. Note that ai_gesture.bundle needs to be loaded.

Value parameters	Description
authdata	The path to the certificate file.

Value parameters	Description
data	String. The path of the AI capability model file ai_xxx.bundle. Such model files are located in the assets/AI_Model directory of the resource package.
type	Int. The AI capability type corresponding to the bundle file. Possible values are listed in enum FUAITYPE.

Value parameters	Description
data	String. The path to the prop package you want to load. A prop package usually has a suffix *.bundle .

Value parameters	Description
data	String. The path of tongue model data tongue.bundle.

Value parameters	Description
obj_handle	String. The path of the prop package passed in when calling fuCreateItemFromPackage.
name	String. The name of the variable to set in the prop package.
value	Object. The variable value to be set.

For details on the variable names and values in the prop package, refer to the FaceUnity documentation.

Value parameters	Description
item	String. The path of the prop package passed in when calling fuCreateItemFromPackage.

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array. The paths to the resource items you want to bind.

Value parameters	Description
obj_handle	String. The path of the target item.
p_items	String Array, the paths to the resource items you want to unbind.

Value parameters	Description
enable	Bool. Whether to get the number of faces being tracked. If set to true, you can receive the fuIsTracking callback.

Value parameters	Description
enable	Bool. Whether to get the number of human bodies tracked by the HumanProcessor algorithm module. If set to true, you can receive the fuHumanProcessorGetNumResults callback.

Value parameters	Description
enable	Bool. Whether to get the number of gestures tracked by the HandGesture algorithm module. If set to true, you can receive the fuHandDetectorGetResultNumHands callback.

​

Callback keys refer to the keys returned in the onEvent callback of the Agora SDK.

Callback keys	Description
fuIsTracking	Returns the number of faces being tracked.
fuHumanProcessorGetNumResults	Returns the number of human bodies tracked by the HumanProcessor algorithm module.
fuHandDetectorGetResultNumHands	Returns the number of gestures tracked by the HandGesture algorithm module.
fuDestroyLibData	Reports that the memory allocated to the face tracking module after calling fuSetup is released.

​

Value parameters	Description
faces	Int. The number of faces being tracked.

​

Value parameters	Description
people	Int. The number of bodies being tracked.

​

Value parameters	Description
hands	Int. The number of gestures being tracked.

​

Its value contains no parameters.

• setExtensionProperty

• addExtension in the RtcEngineConfig class

• enableExtension in the RtcEngineConfig class

• onEvent

This section describes the error codes returned when API calls fail, including the following two types:

• The error code at the SDK layer: Check the return value of the corresponding API.
• Error code at the extension layer: View it through the agorasdk.log file. For example, fuSetup ret: -1 means that an error occurred when calling setExtensionProperty/setExtensionPropertyWithVendor and passing in the key as fuSetup; the error code is -1.

API	Error Code	Description
enableExtension	-3 (SDK layer)	The SDK cannot find the corresponding extension dynamic library. The possible reasons are the following: The extension dynamic library is not correctly imported into the project or the passed extension name is incorrect.
setExtensionProperty	-1 (extension layer)	When the key is fuSetup: Repeated calls, fuSetup only needs to be called once. When the key is fuLoadAIModelFromPackage or fuCreateItemFromPackage: The passed value is wrong.
	-2 (SDK layer)	The timing of calling enableExtension is wrong.
	-2 (extension layer)	Parameter error. Possible reasons are: The input key or value is empty or the value cannot be parsed according to the JSON rules. Please refer to the specification in the document to pass the value.
	-20 (extension layer)	The incoming key is not supported yet. When the key is fuLoadAIModelFromPackage: The AI capability has already been loaded, and there is no need to load it again. When the key is fuDestroyItem: The item has been destroyed, and there is no need to destroy it again.