Screen Sharing with WebRTC and LiveKit: Finding the Best Encoder

Contents#

• Introduction
• Test Setup
• Experiments
• Summary of Results
• Suggestions
• Conclusion

Introduction#

Choosing the right video encoder for video sharing can be a daunting task, especially if you aren't familiar with video compression technologies. This post compares the four main encoders, H264, VP8, VP9, and AV1, in a WebRTC stack with LiveKit (LiveKit is an open source WebRTC stack for real-time audio/video applications) for screen sharing. In particular, we focus on

• End-to-End Latency
• CPU Usage
• Video Quality
• Frame Rate
• Bandwidth Requirements

You'll get benchmark data, which helps you choose the best encoder for your screen sharing app.

You can find the code used for the experiments here, which contains both the screen sharer and the viewer.

Note that we're using our own LiveKit fork, which we modified to export the libwebrtc DesktopCapturer module for easier cross-platform screen capturing.

Summary: For low-latency, high frame rate, use VP8 on macOS or H264 on Windows. For best quality at low bandwidth, use VP9 or AV1 if CPU allows. See tables and charts below for details.

Test Setup#

We followed the testing setup from our previous post, with the addition of running tests on Windows 11 too.

To simulate real-world scenarios, we used two different network conditions: one with limited upload bandwidth and one with adequate upload bandwidth.

The profiles follow:

The experiments were conducted using the following environment combinations, where only the sharer's connection was modified while the controller's connection remained constant.

• 1080p_4up: Sharer using the WiFi_testing profile, and streaming at 1080p resolution.
• 1080p_33up: Sharer using the WiFi profile, streaming at 1080p resolution.
• 1440p_33up: Sharer using the WiFi profile, streaming at 1440p resolution.

To throttle the upload bandwidth, we used clumsy on Windows and network-link conditioner on macOS.

The following table shows the specifications of the machines used for testing:

We performed two types of tests:

• One with simulcast enabled.
• One with simulcast turned off.

Simulcast allows the sender to encode and send multiple resolutions of the same video stream. This supports viewers with different bandwidth and device capabilities.

We split the tests into two scenarios to reflect different real-world use cases. If your app focuses on low latency and runs on stable connections (for example, Hopp), you can turn off simulcast to speed up encoding. But if your app likely has many users in a call, enabling simulcast is recommended.

Note that on Windows, a different display was used for sharing, resulting in a lower resolution than on macOS, specifically the height was smaller. In the following plots, you might notice lower latency and CPU usage values for the Windows tests, especially for the 1080p experiments. This difference results from the smaller screen size.

Experiments#

Every experiment ran for 400 seconds, with sampling starting after the first 500 frames. We also sampled every 150 frames. For each experiment, we set the max frame rate to 30 fps.

We used heuristics to select the bitrate for each codec and resolution. To ensure comparable video quality across encoders, we double-checked our choices using webrtc-vmaf by LiveKit. We ran the Video Multi-Method Assessment Fusion (VMAF) script on the recordings from our experiments. VMAF provides a video quality metric developed by Netflix.

The following plot shows the VMAF score for each encoder and resolution, using the selected bitrates. Each bar displays the corresponding bitrate. The VMAF scores for all encoders are close, ranging from 96 to 97.

VMAF Score Comparison#

Note that when simulcast wasn't enabled, VP8 and H264 used a lower targetBitrate than the max bitrate set in LiveKit.

VP8#

VP8 is an open and royalty-free video codec developed by Google. It's widely supported in browsers and WebRTC applications, making it a common choice for real-time communication. VP8 offers good performance and compatibility, but generally provides lower compression efficiency and video quality compared to newer codecs like VP9 and AV1, especially at lower bitrates. It's less complex than other codecs, which can lead to lower CPU usage and latency.

In the following plots, latency, inbound bandwidth, outbound bandwidth, and frame rate are shown for each scenario, both for macOS and Windows, with simulcast enabled and turned off.

The plots show that both latency and outbound bandwidth increase when simulcast is enabled. On Windows, which uses a less powerful CPU, the reduction in frame rate with simulcast enabled is more pronounced.

We also see that outbound bandwidth is significantly higher with simulcast enabled. Initially, we thought this was due to multiple streams, but after checking the encoder's targetBitrate, we found that the target bitrate is lower when simulcast is turned off. For example, for 1440p, the target bitrate was 6mbps without simulcast and 10mbps with simulcast, which matches the max bitrate value set in LiveKit.

The table below shows average CPU usage (%) for each scenario.

Encoder CPU usage is higher with simulcast enabled across all scenarios and machines. Decoder CPU usage and bandwidth remain unchanged, as simulcast primarily affects the sender.

VP9#

VP9 is an open and royalty-free video codec developed by Google as the successor to VP8. It offers better compression efficiency and video quality than VP8 and H264, especially at lower bitrates, making it well-suited for high-resolution screen sharing. VP9 is widely supported in modern browsers and WebRTC, but it's more computationally intensive, which can lead to higher CPU usage during encoding and decoding compared to VP8.

In the following plots, latency, inbound bandwidth, outbound bandwidth, and frame rate are shown for each scenario, both for macOS and Windows, with simulcast enabled and turned off.

Generally, the plots follow the same patterns as VP8. One noticeable difference: for both macOS and Windows, the frame rate is lower with simulcast enabled across all scenarios, which in some cases leads to lower inbound bandwidth.

CPU usage for the decoder is higher when simulcast isn't used in all scenarios. For Windows, encoder CPU usage is higher when simulcast is enabled, while for macOS there doesn't seem to be a notable difference

AV1#

AV1 is a modern, open, and royalty-free video codec developed by the Alliance for Open Media. It delivers better compression efficiency and video quality than VP9 and H264, especially at lower bitrates, making it ideal for high-resolution and bandwidth-constrained scenarios. But AV1 encoding is much more computationally intensive, which can result in higher CPU usage and slower encoding times. Support for AV1 is growing in browsers and hardware, but may not be as universally available as VP8 or VP9 yet.

In the following plots, latency, inbound bandwidth, outbound bandwidth, and frame rate are shown for each scenario, both for macOS and Windows, with simulcast enabled and turned off.

In the plots, we can see that AV1 shows no difference in any metric when simulcast is enabled or turned off. This is expected, as simulcast isn't yet implemented in librwebrtc's AV1 encoder.

As expected, CPU usage is the same with simulcast enabled or turned off.

H264#

H264 (also known as Advanced Video Coding or AVC) is one of the most widely used video codecs and is supported across virtually all browsers, devices, and video platforms. It offers a good balance between compression efficiency, video quality, and computational complexity. While H264 doesn't achieve the same compression ratios as VP9 or AV1 at lower bitrates, its broad compatibility and low CPU requirements make it a reliable choice for real-time screen sharing, especially when hardware acceleration is available.

The following plots include H264 data for all scenarios on Windows, and for the 1080p scenario only on macOS.

This is because LiveKit allows only the 42e01f profile for H264 codecs with sdpFmtpLine available, see here. This profile uses the 3.1 level, which has an upper limit of 1280×720, lower than the resolutions we tested.

As a result, the stream performed poorly, with many lags and low frame rates. For this reason, we excluded macOS data for H264. If you want to use lower resolutions, you can try H264 on macOS.

In the following plots, latency, inbound bandwidth, outbound bandwidth, and frame rate are shown for each scenario, for Windows, with simulcast enabled and turned off.

We can see that H264 follows the same patterns as VP8 for outbound bandwidth. Everything else is similar with simulcast enabled or turned off.

On the decoder side, CPU usage is low, which makes sense as H264 is hardware-accelerated on macOS. For the encoder, CPU usage is higher with simulcast enabled than with simulcast turned off.

Summary of Results#

Latency#

The following figures show end-to-end latency for each encoder per scenario with one figure for macOS and one for Windows.

1080p 4up#

1080p 33up#

1440p 33up#

For the weak internet connection, all encoders perform similarly on macOS, with VP8 being slightly better than the others. On Windows, H264 is the fastest, with a significant lead over the others.

For 1080p and a better connection, VP8 and AV1 have similar latency on macOS, with VP9 being worse. The same is true for Windows, with the addition that H264 outperforms the others by a large margin. Note that the latency for H264 on macOS is slightly lower than VP8 but with notably lower frame rate.

For 1440p, latency is similar for all encoders on macOS, with VP8 still being the fastest. On Windows, H264 remains the fastest, followed by VP8. AV1 and VP9 have similar, higher latency, which is expected since the Windows machine has a weaker CPU and the newer encoders are more demanding.

Bandwidth#

In this section we compare the bandwidth requirements of each encoder per testing scenario.

Inbound#

The following figures show inbound bandwidth for each encoder per scenario. The same values can be used to compare the outbound bandwidth when simulcast is turned off.

1080p 4up#

1080p 33up#

1440p 33up#

Outbound Simulcast Enabled#

The following table shows the average outbound bandwidth (Mbps) for each encoder and scenario, with simulcast enabled. AV1 isn't included in the table, as simulcast isn't yet implemented.

In poor network conditions, the limiting factor is the sharer's upload bandwidth. The higher the resolution, the greater the gap between newer and older encoders.

The inbound bandwidth improvement of AV1 is noticeable.

With simulcast enabled and the encoder using the full available bandwidth, H264 and VP8 perform worse than VP9.

Frame Rate#

In this section we compare the frame rate the decoder receives for each codec per testing scenario.

1080p 4up#

1080p 33up#

1440p 33up#

We can see that on macOS, the frame rate drops when simulcast is enabled for all encoders, except for VP8. Also, on H264 the frame rate doesn't change much.

CPU Usage#

The following tables summarize average encoder CPU usage (%) for each codec, scenario, and simulcast setting.

macOS Encoder CPU Usage#

Windows Encoder CPU Usage#

Note: H264 data is only available for Windows for both resolutions due to resolution limitations on macOS.*

As expected, VP8 performs better on macOS, and H264 performs better on Windows.

Suggestions#

Note: These suggestions are based on our experiments with static content and screen sharing (such as pair programming). Results may differ for dynamic video or camera feeds.

• For low latency and high frame rate: Use VP8 on macOS or H264 on Windows. Both encoders perform well in these scenarios. The downside is that they require more bandwidth than the newer encoders.
• For high video quality and low bandwidth: Use VP9 or AV1. Both encoders perform well in these scenarios, but they require more CPU resources.
• For many users in a call: Enable simulcast. The latency and bandwidth requirements increase, but the video quality is better for all users.

Conclusion#

In this post, we compared H264, VP8, VP9, and AV1 in a WebRTC stack with LiveKit for screen sharing. We focused on end-to-end latency, CPU usage, video quality, frame rate, and bandwidth requirements.

Our experiments focused on screen sharing of static content. If you’d like us to test other scenarios, let us know!

In future posts, we do deep dives into each encoder and try to tune them for low latency and high quality.

As always, if you have questions or suggestions, reach out on X/Twitter or email me at iason@gethopp.app.