Streaming Protocol and Codec Decision Matrix

Owned by Leo Tian, created
2023-11-13
2 min read

 Relevant data

Bandwidth requirements of various video streaming protocols at 1 FPS (e.g. RTSP, RTMP, WebRTC, etc.), including video codecs (AV1, etc.).

Include the option where we just send images individually (i.e. through a raw socket), which includes research on various image codecs (e.g. JPEG, PNG, JPEG XL, AVIF, etc.).

 Options Considered

There are some other methods such as MPEG-DASHS that were previously on this table that are no longer considered due to extremely high latency or lack of demonstratable functionality to be found.

 

Real-Time Messaging Protocol (RTMP)

Real-Time Streaming Protocol (RTSP)

WebRTC

Secure Reliable Transport (SRT)

TCP / UDP

HTTP

Sending images over socket

 

Real-Time Messaging Protocol (RTMP)

Real-Time Streaming Protocol (RTSP)

WebRTC

Secure Reliable Transport (SRT)

TCP / UDP

HTTP

Sending images over socket

Classification

Legacy Protocol

Legacy Protocol

Modern Streaming Protocol

Modern Streaming Protocol

 

 

 

Description

Works on top of TCP, maintains persistent connections and allows low-latency communication.

Main task is not streaming, rather it is a control protocol that works in conjunction with RTP and RTCP, which often also works on TCP.

Supported by most modern applications. Primarily works over UDP, TCP may be supported

Uses the UDP transmission protocol. However, has extra layers of optimization and security

Same category because they are treated very similarly

UDP: lower latency
TCP: more reliable delivery

Streaming over HTTP

Not as common due to HTTP theoretically having higher latency, but according to the video it performs similarly

Self-explanatory

Probably not a good approach, considering streaming protocols are just a optimized and specialized version of this

Video Codecs

H.264, VP8

H.265, H.264, VP9, VP8

H.264, VP8, VP9

Codec-agnostic

H.264 and H.265

H.264 and H.265

 

Latency

5 - 30s (LL-RTMP can be around 2-3s)

2-5s

< 0.5s

< 3s

few hundred ms

few seconds, but probably heavily depends on context since the example below has it working

Higher latency and less continuity than using a streaming protocol

Implementation & Difficulty

Out of date for most platforms, latency could be a problem. Seems RTSP is favoured in most cases

Example: https://www.youtube.com/watch?v=EoB20ooWPJw

 

Probably more structurally complex than TCP/UDP/HTTP

Example: https://www.youtube.com/watch?v=yCfv2GKhkXs

Seems a lot more complicated, probably not specialized for drones

Theoretically would be good. However, SRT is not standardized and no demonstrated implementation is found

Example: https://www.youtube.com/watch?v=EoB20ooWPJw

Example: https://www.youtube.com/watch?v=EoB20ooWPJw

 

Prevalence
(Determines how much support we realistically receive)

Not used very often

Seems like a good choice, there are a lot of sources using RTSP

Some sources on it, not as much

Next to none

Quite a common approach

A few sources

There is 0 information on this

Sources