1 Overview
2 Dataset
3 Profile run categories
4 Measuring performance
- 4.1 Time
- 4.2 Space
- 4.3 Quality
5 Getting started

Overview

As images are being sent over a connection with limited bandwidth, it is important to be able to compress (AKA encode) these images to minimize data usage. However, better compression is slower, and which is mutually exclusive to the goal of minimizing Autonomy’s control loop latency. Additionally, some compression is lossy, which degrades the quality of the final image after it is decompressed (AKA decoded).

Therefore, there is a trade off between time and space (as always :)).

Profiling: Testing performance.

Dataset

The dataset remains consistent between all runs. The dataset is a series of images taken at 30 FPS over exactly 10 seconds (300 frames), numbered consecutively starting from 0: https://uofwaterloo-my.sharepoint.com/personal/uwarg_uwaterloo_ca/_layouts/15/onedrive.aspx?ga=1&id=%2Fpersonal%2Fuwarg%5Fuwaterloo%5Fca%2FDocuments%2FSubteam%20Folders%2FAutonomy%2FEncode%20Test%20Dataset%202024

To test the performance with different simulated FPS, skip images. For example:

IMAGE_COUNT = 300
IMAGE_FILES_FPS = 30  # Will never change

simulated_fps = 10  # As though the camera is at 10 FPS in this run
# Loops over every 30 / 10 = 3rd image
for i in range(0, 300, IMAGE_FILES_FPS / simulated_fps):
    # Load image i
    ...

    # Do something with it
    ...

Profile run categories

Encoding:

AVIF
- Quality (integer 0-63 inclusive) (start with 20)
- Effort (integer 0-9 inclusive) (start with 0)
JPEG
- Quality (integer 0-100 inclusive)
H.264 (do NOT save as a video file)
- Chroma subsampling
  - 4:4:4
  - 4:2:2
- Others?

Simulated FPS:

1 FPS
5 FPS
10 FPS
30 FPS

Hardware:

Raspberry Pi 4B

Each of these categories is set multiplied. That means all permutations are tested. For example:

AVIF, quality 20, effort 0, 1 FPS
AVIF, quality 25, effort 0, 1 FPS
AVIF, quality 30, effort 0, 1 FPS
…
AVIF, quality 20, effort 1, 1 FPS
…
AVIF, quality 20, effort 0, 5 FPS
…
JPEG, quality 0, 1 FPS
JPEG, quality 10, 1 FPS
JPEG, quality 20, 1 FPS
…
JPEG, quality 100, 30 FPS
H.264, chroma subsampling 4:4:4, 1 FPS
H.264, chroma subsampling 4:4:4, 5 FPS
…
H.264, chroma subsampling 4:2:2, 1 FPS
…
H.264, chroma subsampling 4:2:2, 30 FPS

Run each at least 5 times and record each individual result.

Measuring performance

Disable the Python garbage collector when measuing performance. This is dangerous, so make sure that the garbage collector is disabled for as short a period as possible.

import gc

gc.disable()
# Start measuring

# Done measuring
gc.enable()

Time

Time taken by encoding each individual image is independent of all other code.

# Load image
...

# Encode image
time_before = time.time_ns()
encoded_data = encode(image)
time_after = time.time_ns()

total_time += time_after - time_before

# Other (e.g. saving, decoding)

Space

Measure the space of each encoded image.

Quality

Decode and save the final images as PNG files. After running the test, note visible differences. Example criteria:

Is it noticeably different?
Quality of edges (e.g. between landing pad and ground, H portion and rest of landing pad)

Getting started

Clone and setup the virtual environment: GitHub - UWARG/profile-encode: Profile usage of time and space of various image encoders

Get each step working before moving to the next step:

Encode part working with any non default setting
Decoding and saving
Time profiling
Space profiling

Profile encode speed and space