Agora SEI information

By default, Agora adds the encoding information of the current video to the transcoded H264/H265 SEI (Supplemental Enhancement Information) during Media Push. The encoding information is a JSON string. The following is the sample code:

_32

{

_32

"canvas": {

_32

"w": 640,

_32

"h": 360,

_32

"bgnd": "#000000"

_32

},

_32

"regions": [

_32

{

_32

"uid": 1,

_32

"alpha": 1.0,

_32

"zorder": 1,

_32

"volume": 50,

_32

"x": 0,

_32

"y": 0,

_32

"w": 320,

_32

"h": 360

_32

},

_32

{

_32

"uid": 2,

_32

"alpha": 1.0,

_32

"zOrder": 1,

_32

"volume": 89,

_32

"x": 320,

_32

"y": 0,

_32

"w": 320,

_32

"h": 360

_32

}

_32

],

_32

"ver": "20190611",

_32

"ts": 1535385600000,

_32

"app_data": ""

_32

}

The definition of the parameters:

Parameter	Description
canvas	The canvas information. It contains the following properties: w: Width (pixel) of the canvas. It corresponds to the width member in the LiveTranscoding class. h: Height (pixel) of the canvas. It corresponds to the height member in the LiveTranscoding class. bgnd: The background color (RGB) of the canvas, represented by a hexademical string. It corresponds to the backgoundColor member in the LiveTranscoding class.
regions	The layout information of the host. It corresponds to the transcodingUsers member in the LiveTranscoding class. It contains the following properties: suid: (Optional)The string user account of the host in this region. This parameter applies to scenarios where string user accounts are used to identify the host. uid: UID of the host in this region. It corresponds to the uid member in the TranscodingUser class. alpha: The transparency of the video frame of the host. The value range is [0.0, 1.0]. It corresponds to the alpha member in TranscodingUser . zorder: The layout position of the video frame of the host. The value range is [0, 100]. It corresponds to the zOrder member in TranscodingUser. volume: The volume (dB) of the host. The value range is [0, 100]. x: The horizontal position of the video frame of the broadacaster from the top left corner of the Media Push. It corresponds to the x member in TranscodingUser. y: The vertical position of the video frame of the host from the top left corner of the Media Push. It corresponds to the y member in TranscodingUser. w: Width (pixel) of the video frame of the host. It corresponds to the width member in TranscodingUser. h: Height (pixel) of the video frame of the host. It corresponds to the height member in TranscodingUser.
ver	The version of the SEI protocol. The current version is 20190611.
ts	Timestamp (ms) of the current encoding information.
app_data	Extra user-defined information. It corresponds to the transcodingExtraInfo member in the LiveTranscoding class.

The SEI information is as follows:

_12

0000 0664bd7b 22617070 5f646174 61223a22 .d.{"app_data":"

_12

0010 222c2263 616e7661 73223a7b 2262676e ","canvas":{"bgn

_12

0020 64223a22 23666666 66666622 2c226822 d":"#ffffff","h"

_12

0030 3a363430 2c227722 3a333630 7d2c2272 :640,"w":360},"r

_12

0040 6567696f 6e73223a 5b7b2261 6c706861 egions":[{"alpha

_12

0050 223a3235 352c2268 223a3634 302c2275 ":255,"h":640,"u

_12

0060 6964223a 33313031 32373137 39312c22 id":3101271791,"

_12

0070 766f6c75 6d65223a 32382c22 77223a33 volume":28,"w":3

_12

0080 36302c22 78223a30 2c227922 3a302c22 60,"x":0,"y":0,"

_12

0090 7a6f7264 6572223a 317d5d2c 22747322 zorder":1}],"ts"

_12

00a0 3a313533 37393630 32333537 38332c22 :1537960235783,"

_12

00b0 76657222 3a223230 31383038 3238227d ver":"20190611"}

In which:

• 06: The SEI frame.
• 64: The SEI frame type defined by the user. Here we define it as 100.
• bd: The length of the SEI frame. The following are some sample calculations rendered in decimal and hexadecimal:
  • If the length is 922, because 922 can be divided by 255 (0xff) three times and the remainder is 157 (0x9d), then bd is ffffff9d.
  • If the length is 572, because 572 can be divided by 255 (0xff) two times and the remainder is 62 (0x3e), then bd is ffff3e.
  • If the length is 234, because 234 divided by 255 (0xff) gives 0 and the remainder is 234 (0xea), then bd is ea.
• Other digits: Content of the SEI frame.

Q: If I use the SEI frame in Media Push, is it necessary to use the signaling system to send the layout information? Do I use either signaling or SEI? A: Adding the H264/H265 SEI information to the video stream during Media Push is different from sending the uplink data. Only the app_data parameter of the SEI information relates to the uplink data.

Q: Does the signaling system send the layout information? Why does the SEI frame contain the layout information? A: Agora puts the layout information in the SEI frame because the audience members in the LIVE_BROADCASTING profile also need the layout information of the current stream to set the window region of the hosts.