/
Python Style Convention

Python Style Convention

Owned by R D (Deactivated)
Last updated: 2024-03-24
15 min read

Overview

Code is read and used much more often than it is written. It is better to write good code slowly than bad code quickly.

The goal of style conventions is to increase code readability.

This style convention is applied all new Autonomy Python code (you do not need to worry about old code or libraries). If you’re uncertain, ask the Autonomy project managers and/or leads. In general, the style is PEP8 with a soft 100 character line limit.

Going against convention

System and installed modules do not follow this style convention, including their naming. It is acceptable to use module calls with their default names.

Otherwise, if you write code that goes against this convention:

  • Is going against the convention actually necessary? Why is it bad to follow the convention in this code?

  • If the answer to the above is yes, then write a code comment justifying the reason why

    • A PR comment is NOT an acceptable substitute

# Explanation why code does not follow convention and the justification [code that does not follow convention]

Linters and formatters

Autonomy uses linters and formatters which cover the majority of the style convention, but not all. You are responsible for resolving all linting and formatting issues before your code can be merged.

If there is a conflict between this style convention and the linters and formatters, follow the linters and formatters. Contact the Autonomy leads so that the conflict can be resolved.

More detail: Linters and formatters

Pylance

Pylance is the default Visual Studio Code Python checker by Microsoft. It is very useful for linting.

Disabling Pylance

You can disable Pylance errors by adding this to the end of the line: # type: ignore . You do not need to add a comment explaining the disable.

The only time Pylance should be disabled is in tests and for library calls:

  • Fixtures: yield returns a generator containing the type, not the type itself

    • Generators are a form of lazy evaluation (i.e. the actual object is created at the last possible moment)

  • Accessing a private member or method

  • Pylance is not recognizing a library access

@pytest.fixture() def generate_3() -> int # type: ignore """ Create number 3. """ yield 3 # type: ignore
module_name._ClassName__accessing_privates_for_a_unit_test = 3 # type: ignore
cv2.imshow("Image", image) # type: ignore

Pylint

Pylint is a linter installed by pip as part of Autonomy’s repositories. It can also be installed as a VSCode extension: https://marketplace.visualstudio.com/items?itemName=ms-python.pylint

Pylint covers:

  • Naming convention

  • Unused variables and imports

  • Docstring existence enforcement

  • Pythonic style enforcement (e.g. loop iteration)

  • Exception type handling enforcement

  • Function call correctness

Disabling Pylint

If a lint issue cannot be resolved by restructuring code, Pylint can be disabled. All Pylint disables MUST be accompanied by a comment explaining why this is necessary. The disabled section must be as short as possible.

Pylint can be disabled for a single line or for an entire block.

# Explanation for disabling the linter # pylint: disable-next=[problem code] line that will be disabled this line will not be disabled # Explanation for disabling the linter # pylint: disable=[problem code] stuff # pylint: enable=[problem code]

[problem code] is the descriptive version (e.g. use import-error instead of E0401 ).

Flake8

Flake8 is another linter used by Autonomy.

While Flake8 is a full linter similar to Pylint, Autonomy only uses Flake8 to cover:

  • Usage of type annotations of function and method signatures

    • Flake8 enforces using -> None for functions that do not return anything

# Not accepted by Flake8 def thingy(yo): def thingy(yo: int): def thingy(yo) -> None: # Accepted def thingy(yo: int) -> None:

Flake8 does not cover type annotation of variable names.

# Both are accepted cars = [] cars: "list[Car]" = []

Disabling Flake8

Disabling Flake8 is not supported. However, there are some workarounds involving default objects if they are unknown or it is not feasible to get the type (e.g. somewhere within a library). This MUST be accompanied by a comment explaining why this is necessary.

If the type is known but it is multiple types, instead use: |

# thing can be type ClassA or ClassB or ClassC def multiple_input(thing: "ClassA | ClassB | ClassC") -> None: # anything is unknown, so use type object by default def unknown_input(anything: object) -> None:

Black

Black is a formatter that modifies the Python source code to follow the Black style guide.

Black covers:

  • Line spacing

  • Spaces within lines

  • Line length and multiline formatting

While Black modifies the code, the semantics (i.e. functionality) of the code does not change.

Disabling Black

Black can be disabled. Black is disabled for code that requires formatting to be retained (e.g. matrices). All Black disables MUST be accompanied by a comment explaining why this is necessary. The disabled section must be as short as possible.

Black can be disabled for a single line or for an entire block.

# Explanation for disabling the formatter line that will be disabled # fmt: skip this line will not be disabled # Explanation for disabling the formatter # fmt: off stuff # fmt: on

Formatting

Naming

Format:

  • variable_names in snake_case

  • function_names() in snake_case

    • Function and method names do not start with test_ unless they are specifically part of a UNIT test

  • ClassNames in PascalCase (UpperCamelCase)

    • Class names do not start with Test unless they are specifically part of a UNIT test

  • CONSTANT_NAMES in UPPER_SNAKE_CASE

  • file_names.extension in snake_case

    • File names do not start with test_ unless they are specifically a unit or integration test file

  • Private functions and class members (methods and attributes) are prefixed with 2 underscores:

    • def __my_private_func():

    • __my_private_var = ...

Names are full length and not shortened. This is to increase readability:

  • Prefer error_message instead of err_msg

Numbered variables with underscore surround the number with underscores, unless the letters before and/or after are specifically part of the number:

  • my_1_var

  • HEIGHT_3m

  • g2_star

Initialisms and acronyms:

  • PascalCase: 1st letter capitalized followed by lowercase:

    • NTFS becomes Ntfs

    • RAM becomes Ram

  • snake_case: All lowercase:

    • NTFS becomes ntfs

    • RAM becomes ram

All empty containers are type annotated so that it is clear what they eventually hold:

  • numbers: "list[float]" = []

  • names_to_values: "dict[str, int] = {}

Spacing

TODO Figure out which of these Black covers

Lines:

  • Indents are 4 spaces long

  • Empty lines do not have any space characters

  • No spaces at the end of a line

Spaces:

  • No extra spaces around parantheses:

    • Prefer my_function(my_var) instead of my_function (my_var) or my_function( my_var )

  • Operators:

    • 1 space around mathematical (e.g. + , / ), boolean (e.g. ^ ), and assignment (e.g. = , += ) operators

      • Excluding named arguments: np.sum(arr, axis=1)

    • No spaces around the slicing operator: my_list[0:2] , my_numpy_array[:, 1:3]

  • 1 space after comma: my_function(var1, var2, var3)

  • 1 space after : in type annotations: my_function(param: int) -> ...

  • No spaces around default parameters: my_function(param: float=0.5)

Miscellaneous

  • Only use parantheses when necessary:

    • Prefer while my_num < 3: instead of while (my_num < 3):

  • All strings are enclosed with double quotation marks: print("Hello world!")

    • Prefer f-strings when variables are used: text = f"Height: {height}, width: {width}"

      • Multiline strings are concatenated: text = "Title\n" + "Header\n" + "Body\n"

    • The default encoding is UTF-8: open(filename, encoding="utf-8")

File structure

File contents are arranged in the following sections:

""" File docstring. """ import section GLOBAL_CONSTANTS_SECTION = 1 top_level_call_section() def top_level_functions_section(): """ Function docstring. """ pass class Section: """ Class docstring. """ def class_method(self): """ Method docstring. """ pass # No additional sections # Last blank line below, then end of file

Docstring

A docstring uses 3 consecutive marks, each on its own line. Documentation uses proper grammar, capitalizes the 1st letter (unless it is a name that specifically starts lowercase (e.g. variable name)), and ends with a period.

Use Canadian spelling.

""" This is a docstring example. height: Height of the human in metres. """

metre is the correct Canadian spelling. Same with centre, licence, organization, and manoeuvre.

Imports

Imports are used to bring in code from other libraries into the file.

Separation with previous section

1 empty line between system imports and file docstring.

""" Example. """ import os

Rules

Import order:

  1. System modules: Anything that comes with Python by default

  2. Installed modules: Anything that requires pip install

  3. Local modules: Anything written by/for WARG (including submodules)

Import groups are separated by 1 empty line, with module names in alphabetical order within the group along with the following rules:

  • Sort using the name immediately after the import or from keyword

    • Do not use the keyword to sort

  • Dot (. ) is above a/A

  • Shorter is above longer (e.g. hello above hello_world )

import os import sys from math import pi import numpy as np from . import local_thing from .modules import common_thing from .modules import destroyer from .modules.far_far import away from .modules.near import here

No wildcards (asterisk/star) in from imports. Explicitly state what needs to be imported. Prefer importing the module namespace instead of the object name directly.

from pathlib import * # NO from pathlib import Path # Discouraged import pathlib # Preferred

Do not call private functions or members from an imported module.

import my_module my_module.__some_private_function() # NO my_module.ClassName.__private_member # NO

Global constants

Global constants are used for various settings that are immutable when the code is running.

Global variables are NOT allowed.

Separation with previous section

2 empty lines between global constants and the previous section, except file docstring.

from . import magic MAGIC_NUMBER = 5

If there are no sections other than file docstring, 1 empty line between global constants and file docstring.

""" Example. """ MAGIC_NUMBER = 4

Rules

Global constant groups are separated by 1 empty line. Global constants within each group are ordered by earliest usage of the variable name in the code.

# Warrior FIGHTER = 8 PALADIN = 4 RANGER = 3 # Priest CLERIC = 1 DRUID = 2 # Rogue THIEF = 5 BARD = 6 # Wizard MAGE = 9 SPECIALIST = 10 def rogue_group_magic() -> int: """ Sums the rogue group's magic values. """ thief_magic = THIEF bard_magic = BARD * 10 return thief_magic + bard_magic

Global constants may involve operations and function calls, as long as it is single assignment.

ML_DEVICE = 0 if torch.cuda.is_available() else "cpu"

Top level calls

Top level calls are rare. They are generally used to indicate certain things about the file itself. For example, skipping all unit tests in the file.

Top level calls include linter and formatter setting comments.

Separation with previous section

2 empty lines between top level calls and the previous section, except file docstring.

MAGIC_NUMBER = 3 # This file does not contain unit tests pytest.skip("Integration test", allow_module_level=True)

If there are no sections other than file docstring, 1 empty line between top level calls and file docstring.

""" Example. """ # This file does not contain unit tests pytest.skip("Integration test", allow_module_level=True)

Rules

Related top level calls are grouped. Every group has a comment explaining why the call(s) is necessary.

# This file does not contain unit tests pytest.skip("Integration test", allow_module_level=True)

Top level call groups are separated by 1 empty line.

# This file does not contain unit tests pytest.skip("Integration test", allow_module_level=True) # Clueless i_dunno()

Functions

Functions are useful as helpers to call from other functions and methods.

Separation with previous section

2 empty lines between functions and the previous section, except file docstring.

from . import magic def do_something(): """ Does nothing. """ pass

If there are no sections other than file docstring, 1 empty line between functions and file docstring.

""" Example. """ def do_something(): """ Does nothing. """ pass

Rules

Functions are separated by 2 empty lines.

def do_something(): """ Does nothing. """ pass def do_something_else(): """ Also does nothing. """ pass

Functions and methods have additional general rules below.

Classes

Separation with previous section

2 empty lines between classes and the previous section, except file docstring.

MAGIC_NUMBER = 2 class CatBox: """ A box containing a cat. """

If there are no sections other than file docstring, 1 empty line between top level functions and file docstring.

""" Example. """ class CatBox: """ A box containing a cat. """

2 empty lines between classes.

class Cat: """ Internet animal. """ def __init__(self, name: str): """ name: Name of the cat. """ self.name = name class CatBox: """ A box containing a cat. """

Rules

Classes have additional general rules below.

There are no sections past the classes section.

General

No dead code

Dead code is code that is not used or is unreachable. Dead code is not allowed.

# Unused imports is dead code import numpy as np # np is never called # Code that has been commented out is dead for i in range(0, 5) my_arr.append(i * 2) # print(i) # Commented out # Unused variables is dead code def my_function_0(input_1: int, input_2: str) -> float: """ Example. # Function parameter input_1 is unused """ factor = 3 # Variable factor is unused return float(input_2) # Code that is unreachable is dead def do_thing(i: int) -> int: return i + 3 # Code below is unreachable i *= -1 raise Exception

How does dead code occur?

Dead code occurs for a variety of reasons, but mainly the following:

  • Hoarding: Developers do not like writing code twice, so they keep it around as a reference and just in case it will be used again.

  • Laziness: When modifying code, developers want to change as little as possible. This is especially apparent in function and method signatures, which may no longer require some of the parameters.

How do I avoid dead code?

Delete old code, including unused parameters. Code is usually under version control (e.g. Git), so in the case that recovery or reference is required, it is still available. However, such cases are extremely rare. Old code generally has little to no value (e.g. inefficient, incompatible, deprecated by a library, does not conform to standards, hard to understand).

…you do use Git for your own code, right? If you don’t, you can use GitHub or the UW GitLab: https://git.uwaterloo.ca .

What about all the old WARG code for the previous competition?

We keep the previous year’s competition code as a starting point and reference for the current year’s competition. Any code before the previous year is removed.

Functions and methods

A method is a function that operates as part of a class object.

Functions are arranged in the following sections:

def name(parameters: Type) -> ReturnType: """ Docstring """ # Code blocks section

Parameters and return

All function parameters have their type annotated as well as the return. This is enforced with Flake8.

def my_function_1(input_1: int, input_2: str) -> float: """ Example. """ return float(str(input_1) + input_2) def my_function_2(error_message: str): """ Displays the error message. error_message: The message. """ print(error_message)

Type annotations of containers (e.g. list , dict , tuple ) are surrounded by marks.

def my_function_3(names: "list[str]", mapping: "dict[str, tuple[float, float]]") -> "list[tuple[float, float]]"

Docstring

Separation with previous section:

  • 0 empty lines between function docstring and function declaration

The function docstring contains a short description of what the function does, its parameters, and its return:

  • Parameters: Each parameter name is listed, followed by a short description of what the parameter represents, including units if any

    • Parameters may be grouped, separated by commas

  • Return: Return: followed by a short description of what the return represents

def custom_fibonacci(initial_1: int, initial_2: int, iterations: int) -> "list[int]": """ Fibonacci sequence but with custom initial numbers rather than 1, 1. initial_1, initial_2: Initial values of the sequence. iterations: Number of values past the initial values. Negative is considered as 0. Return: The values in the sequence. """

Code blocks

Code blocks have additional general rules below.

Separation with previous section:

  • 0 empty lines between function code blocks and function docstring

Classes

Classes are arranged in the following sections:

class ClassName(OptionalBaseClass): """ Class docstring. """ __create_key = object() __CLASS_CONSTANTS_SECTION = 1 class SubclassHelper: """ Usually simple like enumerations. """ ... def class_methods_section(self): """ Method docstring. """ pass

Docstring

The class docstring is a short description of what the class is for and what it does. The class docstring follows the docstring rules.

Separation with previous section:

  • 0 empty lines between class docstring and class declaration

Create key

The class create key is an optional class constant used to enforce a private class constructor.

  • Separation with previous section:

    • 0 empty lines between class create key and class docstring

Constants

Class constants are private constants specific to the class.

Separation with previous section:

  • 1 empty line between class constants and the previous section, except class docstring

  • If there are no sections other than class docstring, 0 empty lines between class constants and class docstring

class ClassName(OptionalBaseClass): """ Class docstring. """ __CLASS_CONSTANTS_SECTION = 1

1 empty line between groups of class constants.

Subclasses

Helper classes can be nested in the class. These are usually very simple. If a class constant requires the helper class, then it should be moved out as a separate class.

Separation with previous section:

  • 1 empty line between subclasses and the previous section, except class docstring

  • If there are no sections other than class docstring, 0 empty lines between subclasses and class docstring

Methods

Class methods are functions that operate as part of the class object.

Separation with previous section:

  • 1 empty line between class methods and the previous section, except class docstring

  • If there are no sections other than class docstring, 0 empty lines between class methods and class docstring

class ClassName(OptionalBaseClass): """ Class docstring. """ def class_methods_section(self): """ Method docstring. """ pass

Methods always have self as the 1st parameter, unless otherwise indicated.

Constructor:

The class constructor initializes the class into an object, including its attributes.

  • Method name is: __init__()

  • All class attributes (i.e. self.var_name ) are created in the constructor and nowhere else

    • If a variable is not used until later, initialize it to a default value (e.g. 0 , [] ) or None

      • Prefer default value instead of None , if possible

class CatBox: """ A box containing a cat. """ def __init__(cat_name: "str"): """ cat_name: Name of the cat in the box. """ self.cat_name = cat_name

Private constructor:

The class constructor always succeeds (unless it throws an exception), so all checks must be done prior to calling the constructor.

The create() method checks the input:

  • Decorated with @classmethod

  • Method name is: create()

    • If there are multiple methods, suffix the method name appropriately

  • 1st parameter is cls to allow access to class constants

  • Returns:

    • True, [ClassName] on success

    • False, None on failure

The create() method is arranged in the following manner:

def create(cls, parameters: Type) -> "tuple[bool, ClassName | None]": """ Description that would have been in the constructor. """ # Checks return True, ClassName(cls.__create_key, parameters)

The constructor is modified to have class_private_create_key as the 2nd parameter. This is a workaround to prevent the constructor from being called outside the class without accessing a private member.

def __init__(self, class_private_create_key: object, parameters: Type): """ Private constructor, use create() method. """ assert class_private_create_key is ClassName.__create_key, "Use create() method" self.parameters = parameters

Example:

class BayesChain: """ Bayesian probability chain. """ __create_key = object() def create(cls, probability: float) -> "tuple[bool, BayesChain | None": """ Sets initial probability of the chain. probability: Initial probability in % (between 0.0 and 100.0 inclusive). """ if probability < 0.0: return False, None if probability > 100.0: return False, None return True, BayesChain(cls.__create_key, probability) def __init__(self, class_private_create_key, probability): """ Private constructor, use create() method. """ assert class_private_create_key is BayesChain.__create_key, "Use create() method" self.probability = probability

Static methods:

Static methods are class methods that do not require the class object. They can be thought of as independent functions that happen to be in the class namespace.

Static methods are decorated with @staticmethod and do not have self or cls as the 1st parameter.

@staticmethod def subtraction(minuend: float, subtrahend: float) -> float: """ Subtracts 2 numbers. minuend: Number to be subtracted from. subtrahend: Number to subtract. Return: Difference. """ return minuend - subtrahend

Code blocks

Code blocks are separated by 1 empty line.

TODO More detail?

initial = 3 maximum = initial * 2 print(maximum) prefix_text = "i is: " for i in range(0, 10): print(text + str(i)) if i > maximum: break print("Not yet reached") print("Done!")

Conditionals

Conditionals have the value to be compared on the left and the value to be compared against on the right.

maximum = 3 count = 0 while count < maximum: if count > maximum: break count += 1 # NOT THIS if 5 > other_variable: return other_variable

Checks

Prefer checking for failure and exit early, instead of putting large amounts of code within conditional blocks.

is_good = do_something() if not is_good: return do_something_else() do_another_thing() # NOT THIS is_good = do_something() if is_good: do_something_else() do_another_thing()

Assertions to quiet linter complaints about types are only allowed if they are impossible to fail.

result, value = possibly_failing_function() if not result: continue # Get Pylance to stop complaining assert value is not None do_a_thing(value)

Comments

Comments are used to summarize and/or provide reasons for a specific implementation.

  • Especially if you are going against convention or doing something unusual, add a comment explaining why

File paths

Use pathlib for file paths: pathlib — Object-oriented filesystem paths

This library handles file paths in an OS agnostic manner. It also has some other niceties (e.g. checking path/file existence, creating paths).

FILENAME_FOR_WEB_SCRAPERS = "robots.txt" PATH_TO_WEB_SCRAPERS_FILE = pathlib.Path("resources", "files", "information", FILENAME_FOR_WEB_SCRAPERS) # In Windows, this path is: .\resources\files\information\robots.txt # In Linux, this path is: resources/files/information/robots.txt

Pythonic

Follow a Pythonic style of coding if possible.

file_names = [ "a", "b", "c", "d", ] for file_name in file_names: print(file_name) # This will print `a`, then `b`, then `c`, etc.
# 3x2 array a = np.array( [ [1, 1], [2, 3], [5, 8], ] ) a_sum_columns = a.sum(axis=0) # array([8, 12]) a_sum_rows = a.sum(axis=1) # array([2, 5, 13]) a_sum_all0 = a.sum() # 20 a_sum_all1 = a.sum(axis=None) # 20 # @ is the matrix multiplication operator a @ a_sum_columns # array([20, 52, 136]) # Get column 1 a[:, 1] # array([1, 3, 8])
my_keys = ["C", "V", "P"] my_values = [3, 5, 0] dict(zip(my_keys, my_values)) # "dict[str, int]": {"C": 3, "V": 5, "P": 0}

No raised exceptions

Exceptions are expensive to detect and catch, so they are not allowed.

Exceptions (ha):

  • Unit and integration test assertions

  • Assertions after checks that are impossible to fail

  • Private constructors' create key assertions

def test_add_positive(adder: Add): """ Tests the adder with positive numbers. """ addend_1 = 4 addend_2 = 205454164987651646513 addend_3 = 0.000000000009 expected_sum = 205454164987651646517.000000000009 result, actual_sum = adder.add(addend_1, addend_2, addend_3) assert result assert actual_sum == expected_sum

Instead, use the result, value pattern, which returns:

  • True, [value] on success

  • False, None on failure (including checks)

def integer_divide(dividend: int, divisor: int) -> "tuple[bool, int | None": """ Performs an integer divide. dividend: Integer to be divided. divisor: Integer to divide by. Returns: Quotient. """ if divisor == 0: return False, None return dividend // divisor def get_sample_rate_period(max_frequency: int) -> "tuple[bool, int | None": """ Get the required sample rate period based on the maximum frequency of the signal. max_frequency: Maximum frequency in Hz. Returns: Period in milliseconds. """ if max_frequency < 0: return False, None return integer_divide(2 * 1000, max_frequency)

Multiline

Multiline is automatically formatted by Black.

Related content