WebRTC SFU: the complete Guide.

This content originally appeared on DEV Community and was authored by alakkadshaw

What is WebRTC SFU (Selective Forwarding Unit)?

Selective Forwarding Unit or SFU

SFU is an server based architecture that is used in webrtc to conduct multi party video calling, audio calling and data transfer.

The SFU acts an intermediary between devices that recives media streams from multiple devices and then decides which streams must be send to which devices.

What is WebRTC?

WebRTC is an open source project, using which you can enable real time audio, video and data transfer between devices

WebRTC runs in web browsers as well as in apps, WebRTC is supported by almost all modern browsers and is an standard for video, audio communication

How does SFU or Selective Forwarding Unit work

here is a step by step guide on how does an SFU work

Step 1 Client connections and media capture

1. Participant connection

Each participant first establishes a connection with the SFU. This process involves a signalling server where session descriptions are exchanged.

the session description contains important information, which is used to establish a session.

2. Media stream capture

On the client side the media streams are captured from the participant devices, like audio from microphone, video from camera etc. In the next step these streams are sent to the SFU

Step 2: Stream Transmission to SFU

1. Encoding and sending the stream: The client device encodes the media stream using codecs for video and audio and sends it to the SFU

This is done using the RTP protocol or the SRTP protocol if you need encryption

2. Receipt by SFU

The SFU or the Selective forwarding unit recieves the individual streams from each connected device and maintains a seprate inbound stream for each device.

Step 3 Stream Processing at SFU

1. No Mixing

In SFU as compated to MCU all the streams are seprate from each other and there is no mixing

2. Selective Forwarding

The SFU recieves a seprate stream from each device that is connected to the SFU and then the SFU maintains a seprate stream for each participant

Step 4: Stream Distribution by SFU

1. Adapting:

Before forwarding the stream to uesrs the SFU performs certain adaptations to the stream, such as chossing different quality video codecs and selecting between different simulcast streams.

These are based on the receiving client CPU and Bandwidth capabilities

2*. Forwarding Streams*

The SFU forwards the stream which each client is asking for.

The client devices could receive multiple  of streams from the SFU or even a single stream of video and audio data

1. RTP or SRTP Packets

The media is sent in RTP or SRTP packets, and because different client devices might have different cpu and bandwidth limitations.

The SFU optimizes the streams packets to flow such that the stream flows smoothly and the video quality is smooth on the devices

Step 5 Client Reception and Display

1. Decoding: Each client recieves a RTP streams from the SFU, the client then decodes the stream and renders it on the screen

2. Synchronization: Audio and video synchronization is important and it is handled at the client level.The client uses the timestamps and the sequence numbers from the RTP stream to sync video and audio playback smoothly

SFU Vs p2p vs MCU

In this section let us consider the distinctions between SFU that is Selective forwarding unit, Peer to Peer or P2p and MCU that is Multipoint Control Unit

1.  Selective Forwarding Unit SFU

selective forwarding: The SFU recieves multiple streams and selectively forwards the stream to devices without altering stream content

Server Based: The SFU relies on a server but the server is used for routing the steams and not for anything else.

Advantages:

Scalability: the SFU is highly scalable and it reduces the load on single client devices by managing which stream is sent to any client device

Low processing requirement: Because the SFU just forwards the stream and does not do any processing on the stream, thus reducing the latency and the cpu processing requirement

Stream flexibility:

The client devices can recive multiple streams or just one stream or no stream at all, it all depends on which stream the client device is asking for and needs to display it on the screen

Disadvantages:

Bandwidth Intensive: The SFU are bandwidth intensive on the server side because each stream is recieved by the server and the SFU should have the CPU and the bandwidth to handle all the streams

2. Peer-to-Peer P2P

In P2p architecture each participant connects to another participant directly without the need for an intermediate server

but often a STUN server or a TURN server is required for NAT traversal and to relay the streams around NAT

if you are looking for a turn server you can consider the Metered TURN server service provider

Metered is a Global TURN server provider with TURN server all around the world, that connects to the user nearest to the TURN server for minimum latency and best performance

Disadvantages:

Each participant device must handle multiple streams if there are multiple devices in a meeting.

Which is CPU and bandwidth intensive on all the client devices. For this you can consider the Metered SFU

Multipoint Control Unit

The MCU mixes all incoming streams and creates a single stream or few streams out of a many streams and then sends it to all the clients

The MCU requires a lot of CPU computation and resources on the server, it also creates latency when transmitting streams but it has one advantage, it can greatly reduce the computation and bandwidth requirement on client devices

When in the past the mobile devies were pretty slow and the network infrastructe was also bad, in those days MCU made a lot of sence

But these days with the advance in technology it perhaps provides a bad experience with increased latency and slow video delivery.

Advantages of SFU in WebRTC

Here are some of the advantages of SFU in WebRTC

1. Scalability

the most important advantage of SFU in WebRTC is scalability. You can also scale P2P video calling but it is difficult to create a mesh and scale it

• Reduced Client Bandwidth

When you are video calling with multiple client devices in peer 2 peer format, each client must send their stream to all the other clients and vice versa.

Because of these there is exponentially greater requirement of bandwidth as well as CPU for each client

Some client that do not have capacity for very high bandwidth and cpu compute struggle with the video calling

2. Improved Quality Management

The SFU can make intelligent decisions regarding which streams to send to which clients thus saving imporving quality of video

• Stream Adaptation

The SFU can adapt the stream according to the bandwidth and CPU capacity of the recieving devices.

If the client device has lower bandwidth then the SFU can send low quality streams to that perticular device so that the video runs smoothly without buffering

3. Reduced Processing Load on Clients

The SFU handles the majority of the data processsing that is required for stream management, the client devices themsevles are freed from all the heavy lifting

This is helpful for mobilde devices and devices with low processing power that is CPU power

• Less CPU intensive

Client devices do not face a need for encoding and decoding the multiples streams that are comming to the SFU for processing instead the SFU does this for them

Thus these devices drink less battery run cooler and require less cpu time

• User Experience

The reduction in CPU processing leads to smoother interface, cooler devices and overall better user experience

4. Flexibility in Media handling

SFU offer greater flexibility in how you want to handle the media steams and how you want the video to be displayed on the participants client devices

• Individual Stream Control

The client deives or the meeting participants can choose which streams they want to download and also how they want to display the stream on their screens

They also have the ability to download different quality of strems or make it automatic such that the SFU chooses the best stream based on the client device bandwidth capacity and CPU capability

• Support for advanced featuers

SFU or the Selective Forwarding Unit supports a variety of advanced features like

simulcasting ( ability to send multiple qualites of a single video stream), scalable video codec  and many more  

5. Enhanced Flexibility and Security

All the streams passing through the SFU are end to end encrypted meaning no one not even the SFU know what streams are passing through it

The encryption protocols such as DTLS and SRTP are used for end to end encryption

the SFU can log the interactions but cannot intercept the streams for compliance purposes as well.

Metered TURN servers

1. API: TURN server management with powerful API. You can do things like Add/ Remove credentials via the API, Retrieve Per User / Credentials and User metrics via the API, Enable/ Disable credentials via the API, Retrive Usage data by date via the API.

2. Global Geo-Location targeting: Automatically directs traffic to the nearest servers, for lowest possible latency and highest quality performance. less than 50 ms latency anywhere around the world

3. Servers in all the Regions of the world: Toronto, Miami, San Francisco, Amsterdam, London, Frankfurt, Bangalore, Singapore,Sydney, Seoul, Dallas, New York

4. Low Latency: less than 50 ms latency, anywhere across the world.

5. Cost-Effective: pay-as-you-go pricing with bandwidth and volume discounts available.

6. Easy Administration: Get usage logs, emails when accounts reach threshold limits, billing records and email and phone support.

7. Standards Compliant: Conforms to RFCs 5389, 5769, 5780, 5766, 6062, 6156, 5245, 5768, 6336, 6544, 5928 over UDP, TCP, TLS, and DTLS.

8. Multi‑Tenancy: Create multiple credentials and separate the usage by customer, or different apps. Get Usage logs, billing records and threshold alerts.

9. Enterprise Reliability: 99.999% Uptime with SLA.

10. Enterprise Scale: With no limit on concurrent traffic or total traffic. Metered TURN Servers provide Enterprise Scalability

11. 5 GB/mo Free: Get 5 GB every month free TURN server usage with the Free Plan

12. Runs on port 80 and 443

13. Support TURNS + SSL to allow connections through deep packet inspection firewalls.

14. Supports both TCP and UDP

15. Free Unlimited STUN

This content originally appeared on DEV Community and was authored by alakkadshaw