Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

Overview

Training is done run on the WARG desktop on the Windows partition. A default Ultralytics model is loaded (e.g. nano) with no initial weights. Training configurations are described here: https://docs.ultralytics.com/usage/cfg/

...

https://github.com/UWARG/model-training

Software

...

Setup

Move or copy the 3 directories of the dataset so that it is in the dataset directory:

  • C:\Users\WARG\Ultralytics\datasets\[test, train, val]

Make sure that any old datasets are out of this directory or have their test, train, val directories renamed (e.g. test_landing_pad, train-old, val0)! Hiding them in a directory underneath dataset is not sufficient (e.g. ...\datasets\landing_pad\test might still be erroneously used).

Activate the environment: Python Repository

Navigate into the training directory and run training:

Code Block
cd training
python -m training

Training will take a few hours.

If training is interrupted, change the model load path:

  • MODEL_RESUME_PATH = C:\Users\WARG\Ultralytics\runs\[latest training number]\last.pt

  • data=MODEL_RESUME_PATH

Setup

Follow the instructions: Python Repository

Install packages: pip install -r training/requirements.txt

  • Do not use the other requirements files.

Navigate into the training directory and run training:

...

Follow the instructions to clone the repository and activate the environment: Autonomy Workflow Software

Install packages:

Code Block
pip install -r requirements.txt

Run initial training to create the configuration files:

Code Block
python -m training

If it reaches the dataset checking phase, press ctrl-c to stop the program. It will probably fail before that point with a file not found error.

...

Code Block
datasets_dir: C:\Users\WARG\Ultralytics\datasets
weights_dir: C:\Users\WARG\Ultralytics\weights
runs_dir: C:\Users\WARG\Ultralytics\runs

Or more appropriate directories if desired.

Use other directories if desired.

Note

On Windows, if there is an error along the lines of "path/to/fbgem.dll" or one of its dependencies is missing when importing pytorch, you can check if it’s missing. Usually it is not missing, but its dependency libomp that’s missing. You can install this as part of the Microsoft Visual C++ redistributable. If that still does not work, you can directly download it from here and put it in C:\Windows\System32.

Usage

Move or copy the 3 directories of the dataset so that it is in the dataset directory:

Code Block
C:\Users\WARG\Ultralytics\datasets\[test, train, val]

Make sure that any old datasets are out of this directory or have their test, train, val directories renamed (e.g. test_landing_pad , train-old , val0 )! Hiding them in a directory underneath dataset is not sufficient (e.g. ...\datasets\landing_pad\test might still be erroneously used).

Navigate into the repository and activate the environment: Autonomy Workflow Software

Run training:

Code Block
python -m training

Training will take a few hours.

If training is interrupted, change the model load path in training.py :

Code Block
languagepy
MODEL_RESUME_PATH = C:\Users\WARG\Ultralytics\runs\[latest training number]\last.pt

...

    model.train(
        data=MODEL_RESUME_PATH,
        ...,
    )

Where [latest training number] is the number of the checkpoint.

Hardware

Each epoch takes approximately 5 minutes to complete on an NVIDIA GeForce RTX 2060 with 6GB VRAM: https://www.techpowerup.com/gpu-specs/geforce-rtx-2060.c3310

  • There is only enough VRAM for nano and small models, not larger ones.

  • PyTorch version 2.4.1 is being used with CUDA 12.4, which has minimum driver requirements. Please ensure that a suitable driver is installed (there should be a high enough driver version for the RTX 2060, you can find them here if needed. (We are not going to use the old Jetson TX2i, but the RPi5 on the drone)

WARG desktop details: WARG Desktop

More detailed CUDA compatibility information: CUDA and PytorchPyTorch