Raw video and audio processing

In some scenarios, raw audio and video captured through the camera and microphone must be processed to achieve the desired functionality or to enhance the user experience. Video SDK enables you to pre-process and post-process the captured audio and video data for implementation of custom playback effects.

Understand the tech

You can use the raw data processing functionality in Video SDK to process the feed according to your particular scenario. This feature enables you to pre-process the captured signal before sending it to the encoder, or to post-process the decoded signal before playback. To implement processing of raw video and audio data in your app, take the following steps.

Set up video and audio frame observers.
Register video and audio frame observers before joining a channel.
Set the format of audio frames captured by each callback.
Implement callbacks in the frame observers to process raw video and audio data.
Unregister the frame observers before you leave a channel.

The figure below shows the workflow you need to implement to process raw video and audio data in your app.

Process raw audio and video

Prerequisites

To follow this procedure you must have implemented the SDK quickstart project for Video Calling.

Project setup

To create the environment necessary to integrate processing of raw audio and video data in your app, open the SDK quickstart Video Calling project you created previously.

Implement raw data processing

When a user captures or receives video and audio data, the data is available to the app for processing before it is played. This section shows how to retrieve this data and process it, step-by-step.

Implement the user interface

To enable or disable processing of captured raw video data, add a button to the user interface. In /app/res/layout/activity_main.xml add the following lines before </RelativeLayout>:

<Button
    android:id="@+id/ZoomButton"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_below="@+id/JoinButton"
    android:layout_alignEnd="@id/LeaveButton"
    android:layout_alignStart="@id/JoinButton"
    android:onClick="setZoom"
    android:text="Zoom In" />
Copy

Handle the system logic

This sections describes the steps required to use the relevant libraries, declare the necessary variables, and set up access to the UI elements.

Import the required Android and Agora libraries

To integrate Video SDK frame observer libraries into your app and access the button object, add the following statements after the last import statement in /app/java/com.example.<projectname>/MainActivity.

import io.agora.rtc2.video.IVideoFrameObserver; import io.agora.rtc2.IAudioFrameObserver; import io.agora.base.VideoFrame; import java.nio.ByteBuffer; import android.widget.Button;
Define a variable to manage video processing

In /app/java/com.example.<projectname>/MainActivity, add the following declaration to the MainActivity class:

private boolean isZoomed = false;

Implement processing of raw video and audio data

To register and use video and audio frame observers in your app, take the following steps:

Set up the audio frame observer

IAudioFrameObserver gives you access to each audio frame after it is captured or access to each audio frame before it is played back. To setup the IAudioFrameObserver, add the following lines to the MainActivity class after variable declarations:

private final IAudioFrameObserver iAudioFrameObserver = new IAudioFrameObserver() { @Override public boolean onRecordAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { // Gets the captured audio frame. // Add code here to process the recorded audio. return false; } @Override public boolean onPlaybackAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { // Gets the audio frame for playback. // Add code here to process the playback audio. return false; } @Override public boolean onMixedAudioFrame(String channelId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { // Retrieves the mixed captured and playback audio frame. return false; } @Override public boolean onEarMonitoringAudioFrame(int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { return false; } @Override public boolean onPlaybackAudioFrameBeforeMixing(String channelId, int userId, int type, int samplesPerChannel, int bytesPerSample, int channels, int samplesPerSec, ByteBuffer buffer, long renderTimeMs, int avsync_type) { // Retrieves the audio frame of a specified user before mixing. return false; } @Override public int getObservedAudioFramePosition() { return 0; } @Override public AudioParams getRecordAudioParams() { return null; } @Override public AudioParams getPlaybackAudioParams() { return null; } @Override public AudioParams getMixedAudioParams() { return null; } @Override public AudioParams getEarMonitoringAudioParams() { return null; } };
Set up the video frame observer

IVideoFrameObserver gives you access to each local video frame after it is captured and access to each remote video frame before it is played back. In this example, your modify the captured video frame buffer to crop and scale the frame and play a zoomed-in version of the video. To set up IVideoFrameObserver, add the following lines to the MainActivity class after the variable declarations:

private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() { @Override public boolean onCaptureVideoFrame(VideoFrame videoFrame) { if (isZoomed) { VideoFrame.Buffer buffer = videoFrame.getBuffer(); int w = buffer.getWidth(); int h = buffer.getHeight(); int cropX = (w - 320)/2, cropY = (h - 240)/2, cropWidth = 320, cropHeight = 240, scaleWidth = 320, scaleHeight = 240; buffer = buffer.cropAndScale(cropX, cropY, cropWidth, cropHeight, scaleWidth, scaleHeight); videoFrame.replaceBuffer(buffer, 270, videoFrame.getTimestampNs()); } return true; } @Override public boolean onPreEncodeVideoFrame(VideoFrame videoFrame) { return false; } @Override public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) { return false; } @Override public boolean onPreEncodeScreenVideoFrame(VideoFrame videoFrame) { return false; } @Override public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) { return false; } @Override public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) { return false; } @Override public int getVideoFrameProcessMode() { // The process mode of the video frame. 0 means read-only, and 1 means read-and-write. return 1; } @Override public int getVideoFormatPreference() { return 1; } @Override public boolean getRotationApplied() { return false; } @Override public boolean getMirrorApplied() { return false; } @Override public int getObservedFramePosition() { return 0; } };

Note that you must set the return value in getVideoFrameProcessMode to 1 in order for your raw data changes to take effect.
Register the video and audio frame observers

To receive callbacks declared in IVideoFrameObserver and IAudioFrameObserver, you must register the video and audio frame observers with the Agora Engine before joining a channel. To specify the format of audio frames captured by each IAudioFrameObserver callback, use the setRecordingAudioFrameParameters, setMixedAudioFrameParameters and setPlaybackAudioFrameParameters methods. To do this, add the following lines after if (checkSelfPermission()) { in the joinChannel method:

agoraEngine.registerVideoFrameObserver(iVideoFrameObserver); agoraEngine.registerAudioFrameObserver(iAudioFrameObserver); // Set the format of the captured raw audio data. int SAMPLE_RATE = 16000, SAMPLE_NUM_OF_CHANNEL = 1, SAMPLES_PER_CALL = 1024; agoraEngine.setRecordingAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL); agoraEngine.setPlaybackAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, Constants.RAW_AUDIO_FRAME_OP_MODE_READ_WRITE,SAMPLES_PER_CALL); agoraEngine.setMixedAudioFrameParameters(SAMPLE_RATE, SAMPLE_NUM_OF_CHANNEL, SAMPLES_PER_CALL);
Unregister the video and audio observers when you leave a channel

When you leave a channel, you unregister the frame observers by calling the register frame observer method again with a null argument. To do this, add the following lines to the leaveChannel(View view) method before agoraEngine.leaveChannel();:

agoraEngine.registerVideoFrameObserver(null); agoraEngine.registerAudioFrameObserver(null);
Start and stop video processing

When a user presses the button, enable or disable video processing. To do this, add the following method to the MainActivity class:

public void setZoom (View view){ isZoomed = !isZoomed; Button button = (Button) view; if (isZoomed) button.setText("Zoom Out"); else button.setText("Zoom In"); }

Test your implementation

To ensure that you have implemented raw data processing into your app:

Generate a temporary token in Agora Console.
In your browser, navigate to the Agora web demo and update App ID, Channel, and Token with the values for your temporary token, then click Join.
In Android Studio, open app/java/com.example.<projectname>/MainActivity, and update appId, channelName and token with the values for your temporary token.
Connect a physical Android device to your development device.
In Android Studio, click Run app. A moment later you see the project installed on your device.

If this is the first time you run the project, grant microphone and camera access to your app.
Press Join to see the video feed from the web app.
Test processing of raw video data.

Press Zoom In. You see that the local video captured by your device camera is cropped and scaled to give a zoom-in effect. The processed video is displayed both locally and remotely. Pressing the button again stops processing of raw video data and restores the original video.
Test processing of raw audio data.

Edit the iAudioFrameObserver definition by adding code that processes the raw audio data you receive in the following callbacks:
- onRecordAudioFrame: Gets the captured audio frame data
- onPlaybackAudioFrame: Gets the audio frame for playback

Reference

This section contains information that completes the information in this page, or points you to documentation that explains other aspects to this product.