3D Spatial Audio

3D Spatial Audio brings theater-like effects to Voice Calling, making it seem as if the sound originates from all around the user. Voice SDK provides spatial audio effects that give users an immersive audio experience in scenarios such as e-sports competitions, and online conferences.

You can configure the following spatial audio effects:

• Spatial audio effects for users:

  By setting the local and remote user's spatial positions, you create an environment that enables users to experience changes in the distance, position, and orientation of other users in real time. This includes:

  • Audio range: Enable users in the same group and within the range to hear each other.

  • Voice blur: Enable users to hear sounds outside of the selected range and distance, or sounds of specific objects, as blurred whispers.

  • Air attenuation: Create a more natural and realistic audio experience, making it easier for people to have natural conversations. The audio engine models how sound behaves as it travels through the air and adjusts the volume based on the distance of the audio source from the listener.

  • Local sound source playback: Customize the background sound and other audio effects.

• Spatial audio effects for media player:

  Update the spatial position of the media player to add a sense of space to media resources such as background sounds and musical accompaniment. Agora provides local cartesian coordinate system calculation solution for the media player. This solution calculates the relative positions of the local user and the media player through Voice SDK. You update the spatial coordinates of the local user and the media player, respectively, so that the local user can hear the spatial audio effect of the media player.

The following figure shows the workflow you need to integrate spatial audio into your app.

In order to follow this procedure you must have:

• Implemented the SDK quickstart project for Voice Calling.

• Flutter 2.0.0 or higher

• Dart 2.15.1 or higher

• Android Studio, IntelliJ, VS Code, or any other IDE that supports Flutter, see Set up an editor.

• If your target platform is iOS:

  • Xcode on macOS (latest version recommended)
  • A physical iOS device
  • iOS version 12.0 or later

• If your target platform is Android:

  • Android Studio on macOS or Windows (latest version recommended)
  • An Android emulator or a physical Android device.

• If you are developing a desktop application for Windows, macOS or Linux, make sure your development device meets the Flutter desktop development requirements.

• An Agora account and project.

• A computer with Internet access.

  Ensure that no firewall is blocking your network communication.

• To ensure a true spatial experience, Agora recommends using an audio device that supports true binaural playback.

To create the environment necessary to implement spatial audio into your app, open the SDK quickstart project for Voice Calling you created previously.

This section shows how to use the Video SDK to implement spatial audio into your app, step-by-step.

In a real-word application, you report your local spatial position to a server in your environment and retrieve positions of remote users in the channel from your server. In this simple example, you use a slider to change the spatial position of a remote user.

To add the Slider to the UI, in /lib/main.dart, add the following code to the build method after ListView(...children: [:

const Text('Move the slider to update spatial position'),

Slider(

min: 0,

max: 50,

value: distance,

onChanged: _isJoined ? (value) => {onDistanceChanged(value)} : null,

),

To implement spatial audio features in your app, take the following steps:

1. Declare the variables you need

   You create an instance of LocalSpatialAudioEngine to configure spatial audio parameters and set self and remote user positions. In /lib/main.dart, add the following declarations to the _MyAppState class:

   late LocalSpatialAudioEngine localSpatial;
   

   double distance = 0; // Used to manage spatial position
   

2. Create and configure the spatial audio engine

   To set up an instance of LocalSpatialAudioEngine at startup, take the following steps:

   1. When a user launches the app, you create an instance of LocalSpatialAudioEngine, configure it and update the user's self position. To do this, add the following method to the _MyAppState class:

      void configureSpatialAudioEngine() async {
      

      // Enable spatial audio
      

      await agoraEngine.enableSpatialAudio(true);
      

      
      

      // Get the spatial audio engine
      

      localSpatial = agoraEngine.getLocalSpatialAudioEngine();
      

      
      

      // Initialize the spatial audio engine
      

      localSpatial.initialize();
      

      
      

      // Set the audio reception range of the local user in meters
      

      localSpatial.setAudioRecvRange(50);
      

      
      

      // Set the length of unit distance in meters
      

      localSpatial.setDistanceUnit(1);
      

      
      

      // Define the position of the local user
      

      var pos = [0.0, 0.0,0.0];
      

      var axisForward = [1.0, 0.0,0.0];
      

      var axisRight = [0.0, 1.0,0.0];
      

      var axisUp = [0.0, 0.0,1.0];
      

      
      

      // Set the position of the local user
      

      localSpatial.updateSelfPosition(
      

      position: pos, // The coordinates in the world coordinate system.
      

      axisForward: axisForward, // The unit vector of the x axis
      

      axisRight: axisRight, // The unit vector of the y axis
      

      axisUp: axisUp // The unit vector of the z axis
      

      );
      

      }
      

   2. To execute this method at startup, add the following line to setupVideoSDKEngine(); after await agoraEngine.enableVideo();:

      configureSpatialAudioEngine();
      

3. Set the spatial position of a remote user

   To update the spatial position of a remote user, define the RemoteVoicePositionInfo and call updateRemotePosition. Add the following method to the _MyAppState class:

   void updateRemotePosition(double distance) {
   

   
   

   // Define the remote user's spatial position
   

   RemoteVoicePositionInfo positionInfo = RemoteVoicePositionInfo(
   

   position: [distance, 0.0, 0.0],
   

   forward: [distance, 0.0, 0.0],
   

   );
   

   
   

   // Update the spatial position of a remote user
   

   localSpatial.updateRemotePosition(
   

   uid: _remoteUid ?? 0, posInfo: positionInfo);
   

   }
   

4. Change the spatial position when a user moves the slider

   To update the spatial position of a remote user, you handle the onChanged callback of the slider. In _MyAppState class, add the following method:

   onDistanceChanged(double newValue) {
   

   setState(() {
   

   distance = newValue;
   

   updateRemotePosition(distance);
   

   });
   

   }
   

To ensure that you have implemented 3D Spatial Audio features into your app:

1. Generate a temporary token in Agora Console .

2. In your browser, navigate to the Agora web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.

3. In your IDE, open /lib/main.dart and update appId, channelName and token with the values from Agora Console.

4. Connect a test device to your development device.

5. In your IDE, click Run app. A moment later, you see the project installed on your device.

   If this is the first time you run your app, grant camera and microphone permissions.

6. Press Join to connect to the same channel as your web demo.

7. Test spatial audio effects for remote users

   1. Put on earphones connected to the Flutter test device.

   2. Speak into the microphone connected to the web demo app. Note the quality of audio playing through your earphones.

   3. Move the slider to the right. Your app updates the position of the remote user in the spatial audio engine by increasing the distance.

   4. Speak into the microphone again and listen to the audio through your earphones. You feel that the remote user has moved away from you.

8. Test spatial audio effects for media player

   1. To setup spatial audio position of your media player, add Media Playing to your app.

   2. Replace the call to localSpatial.updateRemotePosition in updateRemotePosition with the following:

      // Update spatial position of the media player
      

      localSpatial.updatePlayerPositionInfo(
      

      playerId: _mediaPlayerController.getMediaPlayerId(),
      

      positionInfo: positionInfo
      

      );
      

   3. Play the media file in your app.

   4. Connect your earphones to the web demo and note the quality of audio playing.

   5. Move the slider to the right. Your app updates the position of the media player in the spatial audio engine.

      Note the change in quality of media playing through your earphones. You feel that the position of the media player has moved away from you.

This section completes the information on this page, or points you to documentation that explains other aspects about this product.

• LocalSpatialAudioEngine
• updateSelfPosition
• updateRemotePosition
• removeRemotePosition
• clearRemotePositions
• updatePlayerPositionInfo